ModelRefs / Data Preprocessing — Tutorial
Data Preprocessing — Tutorial
Missing values, encodings, scaling, and the sklearn Pipeline — turning raw data into model-ready features. Covers The preprocessing checklist.
Overview
Missing values, encodings, scaling, and the sklearn Pipeline — turning raw data into model-ready features
Level: Intermediate. Estimated reading time: 30 minutes.
The preprocessing checklist
Raw data is almost never model-ready. Before fitting any model, work through this checklist:
1. Missing values — does your model handle NaN? (sklearn estimators throw errors; XGBoost handles them natively) 2. Data types — are all columns the right type? Dates stored as strings, booleans stored as "yes/no" strings 3. Cardinality — categorical columns with 10k unique values need special handling 4. Scale — do features span very different ranges? Distance-based models and neural networks require scaling 5. Leakage — does any feature contain information from the future or from the target? A common source of inflated scores 6. Train/test split — always split before any preprocessing that learns from data (scaling, encoding), then fit on train only and transform both sets
Encoding categorical variables
Categorical variables must be converted to numbers. Three main approaches:
Label encoding: assign each category an integer (0, 1, 2…). Only valid for ordinal data (low/medium/high). For nominal categories (colour, country), integer encoding implies an ordering that doesn't exist.
One-hot encoding: create a binary column per category. "country" with 3 values → 3 columns. Use pd.get_dummies() or OneHotEncoder. Watch out for the "dummy variable trap" — always drop one category to avoid perfect multicollinearity.
Target encoding: replace each category with the mean target value of rows in that category. Powerful but leaky if not done inside cross-validation folds. Use category_encoders.TargetEncoder with appropriate smoothing.
High-cardinality categories (zip codes, user IDs): use embeddings (neural networks) or frequency encoding.
sklearn Pipelines: the right way to preprocess
The most common preprocessing mistake is fitting the scaler on the entire dataset before splitting. This leaks test set statistics into training. sklearn Pipelines fix this automatically.
A Pipeline chains transformers and a final estimator. When you call pipe.fit(X_train, y_train), it fits each transformer on X_train, transforms X_train, then fits the next step on the output. When you call pipe.predict(X_test), it transforms X_test using the already-fitted transformers (no re-fitting) and passes the result to the estimator.
ColumnTransformer applies different transformations to different columns simultaneously: numeric columns get scaling, categorical columns get one-hot encoding, all in one step. This is the production-grade preprocessing pattern.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Data Preprocessing — Tutorial.