ModelRefs / Optimization: SGD, Momentum, and Adam — Tutorial
Optimization: SGD, Momentum, and Adam — Tutorial
How gradient descent variants actually work — and why Adam is the default for almost everything. Covers Vanilla SGD and its problems, Momentum and RMSProp.
Overview
How gradient descent variants actually work — and why Adam is the default for almost everything
Level: Advanced. Estimated reading time: 30 minutes.
Vanilla SGD and its problems
Stochastic Gradient Descent updates weights using the gradient of a single (or small batch of) training example(s) rather than the full dataset. The update rule: w ← w − α·∇L.
Problems with vanilla SGD: 1. High variance: gradients from individual examples are noisy. Loss oscillates rather than steadily decreasing. 2. Same learning rate for all parameters: some weights need large updates (sparse features), others need small ones. 3. Stuck in saddle points: in high-dimensional spaces, saddle points (zero gradient, not a minimum) are common. SGD can stall. 4. Ravines: if the loss surface is much steeper in one direction than another, SGD oscillates across the steep direction while making slow progress along the other.
Batch size matters: smaller batches give noisier gradients (more oscillation, can escape local minima) but train faster per epoch. Typical range: 32–512 for vision, 16–256 for language.
Momentum and RMSProp
Momentum: maintain a running average of past gradients (velocity v) and update in that direction: v ← β·v + (1−β)·∇L w ← w − α·v
With β=0.9, momentum accumulates gradients in consistent directions (accelerating through ravines) and cancels oscillating gradients. Like a heavy ball rolling downhill.
RMSProp (Hinton, 2012): adapt the learning rate per parameter by dividing by the running average of squared gradients: s ← β·s + (1−β)·∇L² w ← w − α·∇L / (√s + ε)
Parameters with large recent gradients get smaller updates. This solves the different-scale problem.
Adam: combining momentum and RMSProp
Adam (Adaptive Moment Estimation, Kingma & Ba 2014) combines momentum (first moment) and RMSProp (second moment) with bias correction:
m ← β₁·m + (1−β₁)·∇L # momentum v ← β₂·v + (1−β₂)·∇L² # RMSProp m̂ = m / (1−β₁ᵗ) # bias correction v̂ = v / (1−β₂ᵗ) w ← w − α·m̂ / (√v̂ + ε)
Defaults: β₁=0.9, β₂=0.999, ε=1e-8, α=1e-3. These work out-of-the-box for most tasks.
Variants: AdamW adds weight decay separately from the gradient update (fixes a known bug in Adam's regularisation). Lion (2023) is sign-based and uses less memory. For transformers, AdamW with a linear warmup + cosine decay schedule is standard.
When to use SGD: large-scale vision training where you have time to tune momentum. Use Adam by default for everything else.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Optimization: SGD, Momentum, and Adam — Tutorial.