Gradient Descent Fundamentals

The update rule, learning rate, and why the gradient changes at every step.

The last module ended on a warning: saddle points and flat plateaus, not local minima, are the real obstacles waiting in a deep network's loss landscape. Gradient descent is the algorithm that actually has to fight through them, starting now.

You are training a neural network with 500,000 parameters, and you need the settings that make its loss smallest. How hard could that be? Try to brute-force it — just 10 possible values per parameter — and you face $10^{500000}$ combinations to check. For scale, the observable universe holds about $10^{80}$ atoms. Brute force is not slow; it is flat-out impossible. You need a completely different idea, and it comes from one simple picture.

Imagine you are standing somewhere on a vast, foggy hillside, trying to reach the lowest point in the valley. You cannot see anything, but you can feel the slope of the ground under your feet. So you do the obvious thing: feel which way is downhill, take a small step that way, and repeat. Step by step, you descend. That is gradient descent, and it is the entire idea.

The update rule

Notice the slope is not the same everywhere on that hillside: near the rim of the valley the ground tilts sharply, so a big downhill push is available; near the floor it is almost flat, so barely any push is left. That changing steepness — the direction and strength of "downhill" at your current spot — has a precise name: the gradient. At any point, the gradient points in the direction of steepest *uphill*; flip its sign and you have the steepest way *down*. So each step is:

new weights = old weights − (step size) × gradient

That step size has a name too: the learning rate, and getting it right is most of the game.

Concretely: say one weight currently sits at 2.0, the gradient there is 3.0, and the learning rate is 0.1. The update is new weight = 2.0 − (0.1 × 3.0) = 2.0 − 0.3 = 1.7 — a small nudge downhill, not a leap to the bottom.

Compute the gradient, step a little in the downhill direction, repeat — fifty thousand times, a million times, until the loss stops dropping.

The learning rate: too big, too small, just right

Set the learning rate too small and you inch downhill in tiny timid steps — you will get there, but it might take days of compute. Set it too large and each step overshoots the valley floor, landing partway up the far side; do that repeatedly and the loss bounces around or even flies off to infinity. Somewhere in between is "just right," and the picture above shows all three.

Why not just measure the slope once and follow it straight to the bottom? Because the slope is *local*. The moment you take a step, the ground under your feet has changed — a valley curves, so the downhill direction rotates as you move. Follow your very first reading in a straight line and you would sail right past the floor and up the opposite wall. That is why gradient descent recomputes the gradient at every single step: it is fresh, local information, good only for where you are standing right now.

Why one learning rate is never quite right (going deeper)

Here is the deep frustration with plain gradient descent. Real loss landscapes are rarely tidy round bowls; they are often long, narrow ravines — very steep across the ravine, very gentle along its floor. A step size small enough to be safe on the steep walls is far too small to make progress along the gentle floor. One learning rate simply cannot serve two wildly different steepnesses at once. This mismatch even has a name: the condition number — the ratio of the steepest curvature to the gentlest, the largest Hessian eigenvalue divided by the smallest. A high condition number means a long, narrow ravine like this one, and it is the reason nearly every fancier optimizer exists (momentum, Adam, learning-rate schedules): each one is a trick to take bigger steps in the flat directions and smaller steps in the steep ones.

The gold-standard fix would be Newton's method, which looks at the *curvature* in every direction (through a giant matrix called the Hessian) and sizes each step perfectly on its own. It works beautifully — and it is hopeless at scale: for 500,000 parameters that curvature matrix has 250 billion entries (a terabyte just to store), and inverting an n×n matrix costs O(n³) — hopelessly far too much compute to attempt. So in practice everyone uses gradient descent and its cheap approximations, which borrow a little of Newton's curvature wisdom without paying the full price.

One last honest note: on the simple, bowl-shaped losses of something like logistic regression, gradient descent is guaranteed to reach the bottom. On the wildly bumpy landscapes of deep networks there is no such guarantee — the best you can promise is that it will roll to *some* flat spot. In practice that turns out to be fine, because in very high dimensions the truly bad traps (points that curve upward in every direction at once) become vanishingly rare; the real enemies are the ravines and the long flat plateaus, not getting stuck in a little pit.

How much data per step: full-batch, stochastic, mini-batch

We glossed over *what* the gradient is computed on. Full-batch gradient descent uses the entire dataset for every step — the gradient is exact but each step is expensive, and for millions of examples you take painfully few steps. Stochastic gradient descent (SGD) goes to the other extreme: one example per step — cheap and fast, but the gradient is a noisy estimate that jitters around the true direction. Mini-batch SGD is the practical middle (batches of 32–512): enough examples to average out most of the noise, few enough to take many steps per pass. One full pass over the data is an epoch; each mini-batch update is a step. And the noise isn't purely bad — the jitter of small batches helps the model skip past sharp, brittle minima toward flatter, better-generalising ones. Batch size is a real knob, not just a memory setting.

Where the gradient actually comes from: backprop

Gradient descent *uses* a gradient; backpropagation is how you *get* it efficiently. Backprop is just the chain rule applied layer by layer, computing the gradient of the loss with respect to every weight in a single backward pass — turning what would be a separate derivative computation per parameter into one sweep. Keep the two ideas distinct: backprop computes the gradients, the optimizer decides how to step with them. They're partners, not the same thing.

The optimizer family, in one map

Plain gradient descent's one-step-size weakness spawned a family, each borrowing a little of Newton's curvature wisdom cheaply (each has its own lesson). Momentum keeps a running average (velocity) of past gradients, so consistent directions build speed while oscillations across a ravine cancel out — it damps the zig-zag. AdaGrad/RMSProp give each parameter its *own* effective learning rate based on how large its recent gradients have been, taking bigger steps in flat directions and smaller in steep ones. Adam combines both: a momentum term (first moment, m, controlled by β₁) and a per-parameter scaling (second moment, v, controlled by β₂), plus bias correction for the early steps and an ε for numerical safety — and AdamW fixes how weight decay interacts with that scaling. Adam is the default for most deep learning; SGD-with-momentum still wins in some vision settings.

Learning-rate schedules

One fixed rate is rarely best across a whole run, so you *schedule* it. Warmup starts tiny and ramps up over the first few hundred steps (a big early step on a fresh, unstable model can blow up). Then you decay — step decay (drop by a factor at milestones), cosine decay (smoothly anneal to near zero), or reduce-on-plateau (cut the rate whenever validation loss stalls). The point: take large steps early to cover ground, small steps late to settle precisely. Schedules interact with early stopping (halt when validation stops improving) — together they're how modern training both moves fast and lands cleanly.

When do you actually stop?

"Until the loss stops dropping" needs to be made concrete. Common convergence criteria: validation loss stops improving for *N* checks (early stopping with a patience window — the most common in deep learning), the gradient norm falls below a threshold (you're at a flat spot), the loss improvement per step drops under a tolerance, or you simply hit a max epochs / compute budget. In practice validation-based early stopping is what you use, because it stops at best *generalisation*, not just lowest training loss.

The gradient pathologies to recognise

A few characteristic failure modes, each with a tell and a fix. Vanishing gradients: gradients shrink toward zero in early layers, which barely learn (fix: ReLU-family activations, residual connections, normalisation, good init). Exploding gradients: gradients blow up, loss goes NaN (fix: gradient clipping, better init). Saddle points and plateaus: large flat regions where the gradient is near zero but you're not at a minimum — momentum and adaptive methods power through them. Poor initialisation: weights scaled wrong make activations saturate or explode from step one. Normalisation layers (BatchNorm/LayerNorm) smooth the landscape and make all of this more forgiving. Most of these have dedicated lessons — the point here is to recognise the symptom from the loss curve.

Key points

• Use gradient descent when you have a differentiable loss and many parameters — but not when a closed-form answer exists. For plain linear regression, the exact formula (XᵀX)⁻¹Xᵀy is cheaper and more accurate — do not iterate when you can just solve. Gradient descent earns its keep once there are too many parameters or no closed form: logistic regression on 10,000 features, or any neural network. Rough rule: if solving the exact equations would cost more than running enough gradient steps to converge, iterate.
• The trap: treating the learning rate as one fixed number when the landscape needs different step sizes in different directions. If your loss drops fast for a while and then crawls along a long plateau, that is usually not a data problem — it is the ravine problem: the flat directions are starving while the steep ones already converged. Quick diagnosis: plot the loss on a log scale. A straight line means steady fractional progress (the learning rate is fine); a curve that flattens out means you have hit a high-mismatch region, where momentum or an adaptive optimizer will help.
• The diagnostic: healthy training shows the loss falling steadily — roughly a straight line on a log scale. Oscillating loss means the learning rate is too big. A fast drop then an early plateau means it is too small, or you are in a ravine. A loss that never moves at all usually means a bug — check your gradient against a numerical estimate: nudge one weight up and down by a tiny amount, see how the loss changes, and compare. If that finite-difference slope disagrees with your computed gradient, the gradient code is wrong.
• Know the batch spectrum and that backprop supplies the gradients the optimizer steps with. Full-batch gives an exact but expensive gradient; SGD (one example) is cheap and noisy; mini-batch (32–512) is the practical middle, and its noise actually helps escape sharp minima toward flatter ones — one pass is an epoch, one batch update a step. Backprop (the chain rule, one backward pass) computes those gradients efficiently; the optimizer decides how to use them. The optimizer family — momentum (velocity of past gradients), RMSProp/AdaGrad (per-parameter rates), Adam/AdamW (both, with bias correction and β₁/β₂) — each cheaply borrows curvature to fix plain GD's single-step-size weakness.
• Schedule the learning rate, stop on validation, and recognise the pathologies. Use warmup (ramp up to avoid early blow-ups) then decay (step, cosine, or reduce-on-plateau) — big steps early, small steps late. Stop via validation-based early stopping with a patience window (best generalisation), or gradient-norm/loss-tolerance/max-budget criteria. Read pathologies off the loss curve: vanishing gradients (early layers stall → ReLU/residuals/normalisation/init), exploding gradients (NaN → clipping/init), saddle points and plateaus (flat, near-zero gradient → momentum/adaptive methods power through), poor initialisation (saturated activations from step one).

Recap

• Brute-forcing the parameters is impossible: 500k parameters × just 10 values each = $10^{500000}$ combinations — dwarfing the ~$10^{80}$ atoms in the observable universe. You need a completely different idea, and it's local descent.
• Update rule: new weights = old weights − (learning rate) × gradient. The gradient points steepest *uphill*, so you flip its sign to go down, take a small step, and repeat tens of thousands of times until the loss stops dropping.
• Learning rate is most of the game: too small and you crawl (days of compute); too large and each step overshoots the valley floor, so the loss bounces or flies off to infinity. Somewhere between is "just right."
• Recompute the gradient at *every* step — it's purely *local* information. The valley curves, so the instant you move the downhill direction rotates; follow your first reading in a straight line and you'd sail past the floor and up the far wall.
• Ravine / condition-number problem: real losses are long narrow ravines, steep across and gentle along the floor. One step size safe on the steep walls is far too small for the flat floor — a single learning rate can't serve both, which is why momentum, Adam, and LR schedules exist.
• Newton's method is the ideal fix but hopeless at scale: it sizes each step by the curvature (the Hessian) but for 500k params that matrix has ~250 billion entries (~1TB) and costs O(n³) to invert — so everyone uses gradient descent and its cheap curvature approximations instead.
• Batch spectrum: full-batch (exact gradient, painfully slow) → SGD one example (cheap, noisy) → mini-batch 32–512 (the practical default, and its noise helps escape sharp minima). Backprop *supplies* the gradient via the chain rule; the optimizer *decides how to step* with it — keep the two distinct.

Check your understanding