Gradient Descent: What Your Intuition Gets Wrong

The "ball rolling down a hill" analogy is everywhere. It's also subtly wrong in ways that matter. Loss landscapes for neural networks are not convex bowls. They have saddle points, flat regions, and narrow ravines — and the optimiser you choose determines whether you escape them or get stuck.

The standard mental model: we have a loss surface, gradient descent finds the downhill direction, and we step in that direction until we reach the minimum. Simple, intuitive, incomplete.

Why the bowl analogy fails:

For neural networks, the loss surface is non-convex and exists in millions of dimensions. Visualisations showing a smooth bowl with a single minimum are generated for two-parameter toy models. Real loss surfaces have: saddle points (gradient = 0 but not a minimum), plateaus (gradient ≈ 0, not a minimum), sharp ravines (very different curvature in different directions), and many local minima.

In high dimensions, local minima are surprisingly rare — saddle points are far more common. The real enemy isn't getting stuck in a local minimum; it's getting stuck near a saddle point where gradients are near zero.

SGD vs momentum vs Adam:

SGD: gradient = mean over mini-batch, step = -learning_rate * gradient. Unbiased but high variance. Sensitive to learning rate. Struggles in ravines (oscillates across, moves slowly along).

Momentum: keeps an exponential moving average of past gradients (velocity). Helps escape saddle points. Smooths oscillations in ravines. The β parameter (typically 0.9) controls how much history to retain.

RMSProp: divides the gradient by a running average of squared gradients. Adapts learning rate per parameter. Parameters with consistently large gradients get smaller updates. Useful for sparse features.

Adam: combines momentum and RMSProp. First moment (mean) + second moment (uncentered variance). Bias correction for the first few steps. Most commonly used, but has known convergence issues and can generalise worse than SGD with careful tuning.

The learning rate is the most important hyperparameter:

Too high: loss diverges (explodes) or oscillates without converging. Too low: convergence is slow and may stall near saddle points. Warmup: start small, ramp up to target, then decay. Standard for transformer training. Cosine annealing: smoothly reduces learning rate following a cosine curve. Often improves final accuracy.

Batch size interacts with learning rate:

Larger batch → more accurate gradient estimate → can use larger learning rate → fewer updates to converge. The linear scaling rule: if you multiply batch size by k, multiply learning rate by k (with warmup). This breaks down at very large batch sizes.

What practitioners actually get wrong:

Not checking that training loss is actually decreasing in the first 100 steps. Using Adam's default parameters (lr=0.001) for fine-tuning LLMs without adjusting. Using the same learning rate for all parameter groups. Not visualising the gradient norms over training — norm collapse or explosion is diagnostic. Not logging the loss curve at all.