ModelRefs / Feature Engineering — Tutorial
Feature Engineering — Tutorial
Transform raw data into informative features that ML models can learn from. Covers Raw data is never model-ready, Encoding categorical variables.
Overview
Transform raw data into informative features that ML models can learn from
Level: Intermediate. Estimated reading time: 35 minutes.
Raw data is never model-ready
Feature engineering bridges the gap between raw data and a learning algorithm. Models can't process text strings, date objects, or categorical labels directly — they need numbers. Beyond format conversion, good features amplify the signal in your data.
The rule of thumb: garbage in, garbage out. A logistic regression with excellent features beats a neural network with poor features. Feature engineering is where most of the business value is created, and it requires domain knowledge.
Encoding categorical variables
One-hot encoding: creates a binary column for each category. Good for nominal (unordered) variables with few categories. Beware the dummy variable trap — use drop='first' to remove multicollinearity.
Ordinal encoding: assigns integers in order (small=0, medium=1, large=2). Use only for truly ordered categories.
Target encoding: replaces a category with the mean target value for that category — powerful for high-cardinality features (hundreds of categories) but prone to leakage. Use with cross-fitting.
Label encoding: converts labels to integers. Use only for the target variable, not features, unless the model is a tree-based method.
Scaling — critical for distance-based models
StandardScaler: removes mean, divides by std. Result has mean=0, std=1. Required for: logistic regression, SVM, KNN, PCA, neural networks.
MinMaxScaler: scales to [0, 1]. Use when you need bounded output.
RobustScaler: scales using median and IQR — resistant to outliers.
Decision trees and random forests are scale-invariant (splits are rank-based). For every other algorithm, scale your features. Always fit the scaler on train data only; transform both train and test.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Feature Engineering — Tutorial.