Momentum

Velocity accumulation, Nesterov look-ahead, and escaping oscillation in ravines.

Gradient descent treats every step as independent — it computes the gradient at the current position, takes a step, then forgets everything that happened before. On a smoothly curved loss surface this works, but real deep network landscapes have ravines: narrow valleys where the curvature across the valley is much higher than the curvature along it. In a ravine, gradient descent oscillates. Each step overshoots across the narrow dimension while barely advancing along the valley floor.

The gradient alternates direction across the ravine at every step, so the optimizer zigzags instead of running forward.

The solution is to give the optimizer memory. Momentum maintains a velocity vector that accumulates the history of past gradients, governed by a momentum coefficient β (a hyperparameter, 0 ≤ β < 1, that sets how much of the previous velocity carries forward): v ← βv + ∇L(θ), then θ ← θ − αv. Gradients that consistently point the same direction accumulate into large velocity in that direction. Gradients that alternate direction — the oscillation across the ravine — partially cancel each other in the velocity and die out. The optimizer now runs along the valley floor rather than bouncing off the walls. A further refinement, Nesterov momentum, computes the gradient not at the current position but at the position you will be after applying the current velocity. This look-ahead avoids applying a correction that is already about to be corrected — it removes a systematic lag between where the optimizer is and where the gradient says to go.

Key points

• Gradient descent without momentum fails in ravine-shaped landscapes because it discards all trajectory information. Each step is computed fresh from the local gradient. In a ravine with high curvature across the width and low curvature along the length, the across-ravine gradient is large and the along-ravine gradient is small — so each step takes a large oscillating step across the ravine and a tiny step forward. The optimizer zigzags.
• Momentum fixes this by accumulating velocity: v ← βv + ∇L(θ); θ ← θ − αv. The velocity is an exponential moving average of past gradients. Across the ravine, gradients alternate sign (+large, -large, +large) — they cancel in the moving average, reducing velocity in that direction. Along the ravine, gradients consistently point forward — they accumulate in the moving average, growing velocity in that direction. Oscillation dampens; forward progress accelerates.
• The effective step size with momentum is amplified in steady-state. When gradients consistently point the same direction, velocity converges to ∇L/(1−β). With β=0.9 and α=0.01, the effective learning rate is 0.01/0.1 = 0.1 — 10x larger than α alone. This is why adding momentum requires reducing the base learning rate: the same α now produces much larger effective steps.
• Nesterov momentum computes the gradient at the look-ahead position θ − αβv rather than at θ. Vanilla momentum applies the current velocity, then corrects based on the gradient at the new position — it overshoots, then corrects. Nesterov sees where it is going before committing to the full step, allowing it to brake early. The classical O(1/t²) accelerated-convergence guarantee (Nesterov, 1983) is proven for a momentum coefficient that increases toward 1 across iterations (e.g. β_k=(k−1)/(k+2)) — not for the fixed β=0.9 used throughout this module. The constant-β "Nesterov momentum" used in deep learning does not carry that formal guarantee, though it typically converges faster in practice near minima than plain momentum.
• Momentum helps escape saddle points. At a saddle point the gradient is exactly zero, so gradient descent stalls: θ ← θ − α·0 = θ. Momentum continues: θ ← θ − α·(βv + 0) = θ − αβv. The accumulated velocity from the approach to the saddle carries the optimizer through the zero-gradient region. Once past the saddle point, the gradient is nonzero again and normal optimization resumes. Note this θ − αβv is a different role than the Nesterov look-ahead point above, even though the algebra matches: here it is the actual (plain-momentum) parameter update, and it only collapses to this form because ∇L(θ)=0 at the saddle; Nesterov's θ − αβv is never itself the final update, only the point at which the next gradient is evaluated before the real step is taken.
• The β hyperparameter sets the effective memory window. β=0.9 gives a window of 1/(1−0.9) = 10 gradient steps — enough to smooth mini-batch noise without making the optimizer sluggish to landscape changes. β=0.99 gives a window of 100 steps, useful for very noisy gradient signals but slow to respond when the loss landscape changes character during training.

Recap

• Plain GD has amnesia: every step is computed fresh from the local gradient with no memory of where it came from, so it zigzags back and forth across ravine walls and stalls dead at saddle points where the gradient is zero.
• Momentum keeps a velocity — a running average of past gradients: v ← βv + ∇L(θ), then θ ← θ − αv. The optimizer carries momentum through the landscape instead of restarting from scratch each step.
• Oscillations cancel, consistent directions accumulate: across a ravine the gradient flips sign each step so the velocity contributions cancel and the zig-zag is damped; along the gentle floor the gradient is consistent so velocity builds and the optimizer speeds up.
• It amplifies the effective learning rate: at steady state velocity → ∇L/(1−β), so β=0.9 makes steps roughly 10× larger than plain GD — remember to *reduce* the base α to compensate or you'll overshoot.
• Nesterov momentum looks ahead — it evaluates the gradient at θ − αβv (where the velocity is about to carry it) and brakes *before* overshooting, typically converging faster in practice near minima than plain momentum (the classical O(1/t²) guarantee needs an increasing-β schedule, not the fixed β used here).
• Momentum escapes saddle points and plateaus: the accumulated velocity carries it straight through the flat zero-gradient region where plain GD freezes, turning a stall into a slowdown.
• β sets the effective memory window: β=0.9 averages ~1/(1−0.9) = 10 steps (smooths mini-batch noise, still responsive), β=0.99 ~100 steps (handles very noisy gradients but is slow to react when the landscape changes character).

Check your understanding