Data Model
Pandas is built around data structures called Series and DataFrames. Data for these collections can be imported from various file formats such as comma-separated values, JSON, Parquet, SQL database tables or queries, and Microsoft Excel. [7]
A Series is a 1-dimensional data structure built on top of NumPy's array. [8] : 97 Unlike in NumPy, each data point has an associated label. The collection of these labels is called an index. [4] : 112 Series can be used arithmetically, as in the statement series_3 = series_1 + series_2: this will align data points with corresponding index values in series_1 and series_2, then add them together to produce new values in series_3. [4] : 114 A DataFrame is a 2-dimensional data structure of rows and columns, similar to a spreadsheet, and analogous to a Python dictionary mapping column names (keys) to Series (values), with each Series sharing an index. [4] : 115 DataFrames can be concatenated together or "merged" on columns or indices in a manner similar to joins in SQL. [4] : 177–182 Pandas implements a subset of relational algebra, and supports one-to-one, many-to-one, and many-to-many joins. [8] : 147–148 Pandas also supports the less common Panel and Panel4D, which are 3-dimensional and 4-dimension data structures respectively. [8] : 141
Users can transform or summarize data by applying arbitrary functions. [4] : 132 Since Pandas is built on top of NumPy, all NumPy functions work on Series and DataFrames as well. [8] : 115 Pandas also includes built-in operations for arithmetic, string manipulation, and summary statistics such as mean, median, and standard deviation. [4] : 139, 211 These built-in functions are designed to handle missing data, usually represented by the floating-point value NaN. [4] : 142–143
Subsets of data can be selected by column name, index, or Boolean expressions. For example, df[df['col1'] > 5] will return all rows in the DataFrame df for which the value of the column col1 exceeds 5. [4] : 126–128 Data can be grouped together by a column value, as in df['col1'].groupby(df['col2']), or by a function which is applied to the index. For example, df.groupby(lambda i: i % 2) groups data by whether the index is even. [4] : 253–259
Pandas includes support for time series, such as the ability to interpolate values [4] : 316–317 and filter using a range of timestamps (e.g. data['1/1/2023':'2/2/2023'] will return all dates between January 1st and February 2nd). [4] : 295 Pandas represents missing time series data using a special NaT (Not a Timestamp) object, instead of the NaN value it uses elsewhere. [4] : 292
Indices
By default, a Pandas index is a series of integers ascending from 0, similar to the indices of Python arrays. However, indices can use any NumPy data type, including floating point, timestamps, or strings. [4] : 112
Pandas' syntax for mapping index values to relevant data is the same syntax Python uses to map dictionary keys to values. For example, if s is a Series, s['a'] will return the data point at index a. Unlike dictionary keys, index values are not guaranteed to be unique. If a Series uses the index value a for multiple data points, then s['a'] will instead return a new Series containing all matching values. [4] : 136 A DataFrame's column names are stored and implemented identically to an index. As such, a DataFrame can be thought of as having two indices: one column-based and one row-based. Because column names are stored as an index, there are also not required to be unique. [8] : 103–105
If data is a Series, then data['a'] returns all values with the index value of a. However, if data is a DataFrame, then data['a'] returns all values in the column(s) named a. To avoid this ambiguity, Pandas supports the syntax data.loc['a'] as an alternative way to filter using the index. Pandas also supports the syntax data.iloc[n], which always takes an integer n and returns the nth value, counting from 0. This allows a user to act as though the index is an array-like sequence of integers, regardless of how it's actually defined. [8] : 110–113
Pandas supports hierarchical indices with multiple values per data point. An index with this structure, called a "MultiIndex", allows a single DataFrame to represent multiple dimensions, similar to a pivot table in Microsoft Excel. [4] : 147–148 Each level of a MultiIndex can be given a unique name. [8] : 133 In practice, data with more than 2 dimensions is often represented using DataFrames with hierarchical indices, instead of the higher-dimension Panel and Panel4D data structures [8] : 128
