ModelRefs / Pandas Data Analysis — Tutorial
Pandas Data Analysis — Tutorial
Load, clean, explore, and transform tabular data — the daily work of every ML practitioner. Covers DataFrames: the core data structure.
Overview
Load, clean, explore, and transform tabular data — the daily work of every ML practitioner
Level: Beginner. Estimated reading time: 30 minutes.
DataFrames: the core data structure
A Pandas DataFrame is a table — rows are observations, columns are features. Every ML project starts here: loading raw data and understanding its shape.
Key operations you'll use constantly: - df.shape — (rows, cols) - df.head(5) — first 5 rows - df.info() — column types and null counts - df.describe() — mean, std, min, max for numeric columns - df.value_counts() — frequency of each unique value - df.isnull().sum() — count missing values per column
Loading data: pd.read_csv("file.csv"), pd.read_parquet(), pd.read_sql(). Most real datasets arrive as CSVs.
Filtering, grouping, and aggregating
Filtering rows: use boolean masks. df[df["age"] > 30] returns rows where age > 30. Chain conditions: df[(df["age"] > 30) & (df["country"] == "US")].
GroupBy: split the data by category, apply a function, combine results. df.groupby("category")["revenue"].sum() totals revenue per category. This is the pandas equivalent of SQL's GROUP BY.
Merging: pd.merge(df1, df2, on="user_id", how="left") is a SQL LEFT JOIN. Joining tables on a shared key is fundamental for feature engineering.
Applying functions: df["column"].apply(my_function) runs any Python function on every value in a column — useful for custom feature extraction.
Handling missing data
Missing values (NaN) are a fact of real-world data. Your options:
Drop: df.dropna() removes any row with a missing value. df.dropna(subset=["critical_col"]) only drops rows where a specific column is null.
Fill: df.fillna(0) replaces all NaN with 0. df["age"].fillna(df["age"].median()) fills with the median — safer than mean for skewed distributions.
Interpolate: df.interpolate() fills gaps in time series using linear interpolation.
For ML, you almost always need to handle missing values before fitting a model — most sklearn estimators will throw an error on NaN inputs. The choice of strategy (drop vs fill) matters: always check whether data is missing at random or whether the missingness itself is informative.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Pandas Data Analysis — Tutorial.