ModelRefs / Regularization — Tutorial

Regularization — Tutorial

L1, L2, dropout, and early stopping — the tools that keep your model from memorising the training set. Covers Why models overfit, L1 and L2 regularisation.

Overview

L1, L2, dropout, and early stopping — the tools that keep your model from memorising the training set

Level: Advanced. Estimated reading time: 25 minutes.

Why models overfit

Overfitting happens when a model learns the training data too well — including its noise — and fails to generalise. Signs: training accuracy near 100%, validation accuracy much lower; loss curves that diverge after some point.

Root cause: the model has more capacity (parameters) than the effective size of the training signal justifies. A 100M-parameter network trained on 100 examples will memorise those examples rather than learn the underlying pattern.

Three solutions: (1) get more data — the best cure when possible, (2) reduce model capacity, (3) regularisation — add a penalty or noise that prevents the model from using its full capacity.

L1 and L2 regularisation

L2 (weight decay / Ridge): add λ·Σwᵢ² to the loss. This penalises large weights, pushing them toward zero but rarely exactly to zero. The gradient adds 2λwᵢ to each weight update. In PyTorch: weight_decay parameter in the optimiser.

L1 (Lasso): add λ·Σ|wᵢ| to the loss. This pushes many weights exactly to zero — producing sparse models. Useful for feature selection. In PyTorch: not built into the optimiser, manually add l1_loss to the model loss.

Elastic Net: combine L1 and L2. sklearn's ElasticNet(alpha=..., l1_ratio=...) controls the mix.

L2 is the default in most settings. For neural networks, "weight decay" in AdamW implements L2. For linear models (regression, logistic), try L1 when you suspect many features are irrelevant.

Dropout and early stopping

Dropout (Srivastava et al., 2014): during training, randomly set each neuron's output to zero with probability p (typically 0.1–0.5). At test time, all neurons are active but weights are scaled by (1−p). This forces the network to learn redundant representations — no single neuron can be relied on.

Dropout is most effective for large fully-connected layers. Use p=0.1–0.2 for conv layers, 0.3–0.5 for dense layers. Set model.eval() at test time — this disables dropout automatically in PyTorch.

Early stopping: monitor validation loss during training and stop when it starts increasing. Save the checkpoint with the best validation loss. Effectively limits the training budget and prevents extended overfitting. In practice, train with a learning rate schedule and save the best checkpoint rather than literally stopping.

Batch normalisation incidentally acts as a regulariser by adding noise (the batch statistics vary). It often reduces the need for dropout.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Regularization — Tutorial.