Learning Rate Schedules

Warmup, cosine annealing, cyclic LR, and why the schedule shape changes what you find.

You are training a ResNet from scratch. You pick a learning rate and run 90 epochs. If you picked too high — say $α = 0.1$ when $0.01$ is appropriate — loss oscillates from epoch 1 and the model never converges. If you picked too low — $α = 0.0001$ — loss decreases smoothly but stops at 72% accuracy when the same architecture should reach 76%. Congratulations, you found a mediocre minimum and permanently settled there. No single fixed learning rate gives you both the early progress you need and the fine-grained convergence required to reach the best basin. This is not a tuning problem. It is a structural mismatch between one constant value and a landscape that requires different step sizes at different phases of training.

The solution is to make the learning rate change over time. But how? The crudest version is step decay: drop $α$ by a factor of 10 at epoch 30 and epoch 60. This is the classic ResNet schedule and it works. The problem is the suddenness. Wherever the optimizer happens to be at epoch 30, that basin is now where it will stay — a sharp drop removes the energy needed to escape. If the optimizer landed in a slightly sharp basin at epoch 29, it is now trapped there.

Before we even get to decay strategies, there is a problem at the very start. Early in training, gradient directions are unreliable: weights are far from any useful configuration, batch statistics are noisy, and Adam's second moment estimate $v_t$ has not stabilized from zero. Applying the full learning rate at step 1 means taking large steps in arbitrary directions. Warmup — linearly ramping $α$ from near-zero to the target value over the first 1%–5% of training steps — gives gradient estimates time to accumulate before large steps are applied. For transformers, skipping warmup causes early embedding corruption that is nearly impossible to recover from.

NOT this. Most people think warmup is an Adam-specific trick to work around bias correction. Actually, warmup solves a different problem: gradient direction reliability. Even with perfect bias correction, the gradient direction at step 1 is computed on one mini-batch of a randomly-initialized model — it is essentially noise. Warmup says "do not trust this yet, take small steps until the signal stabilizes." Bias correction fixes the magnitude of early moments; warmup is about not acting aggressively on unreliable directions.

After the stable phase, cosine annealing replaces step decay's abrupt drop with a smooth curve: $α(t) = α_{min} + 0.5(α_{max} - α_{min})(1 + cos(πt/T))$. The gradual decrease means the optimizer keeps exploring broadly early and narrows its search gradually rather than stopping abruptly. Empirically, cosine annealing finds flatter basins than step decay, delivering 0.5%–2% better test accuracy on standard benchmarks. The mechanism: in the high-$α$ phase, the optimizer can still occasionally escape mediocre basins. As $α$ decreases continuously, exploration narrows and the optimizer settles into the flattest basin it has found.

OneCycleLR (Smith, 2018) goes further: ramp $α$ up from $α_{min}$ to a peak 5–10x higher than a typical constant rate over 30% of steps, then cosine decay down over the remaining 70%. The high-$α$ peak phase is aggressive exploration. The long decay phase is fine-grained convergence. This "super-convergence" has achieved matching accuracy in 10–20x fewer epochs on some tasks. The canonical transformer schedule — linear warmup, cosine decay to near zero — is structurally identical: aggressive early phase, extended fine-grained final phase.

Key points

• Use linear warmup for 1%–5% of total steps, then cosine decay to near zero — this is the production-proven schedule for most deep learning. For ResNets: warmup over 5 epochs, then cosine decay. For transformers: warmup over 4% of training tokens, then cosine or linear decay. Peak learning rate: $1e$-$4$ to $3e$-$4$ for transformers, $0.1$ for ResNet+SGD. Getting the warmup length wrong by 2x costs less than getting it completely absent — absent warmup on transformers causes divergence in the first few hundred steps with near-certainty.
• The most common production trap: using a fixed learning rate because "it converged." A model that "converged" on a fixed LR has actually found a basin and stopped exploring. It may have found a mediocre basin early and gotten stuck. Symptom: training loss stabilizes but is 5%–10% above published benchmarks for your architecture. Fix: add cosine decay — if loss continues improving after adding the schedule, you were prematurely converged. If loss does not improve further, you have actually found a good basin and the schedule is just confirming it. The schedule costs nothing to add.
• Diagnostic: loss that decreases fast, then plateaus 20+ epochs before the end = the schedule dropped $α$ too early or too sharply. With step decay: plateau after a step drop means the jump was too large — the optimizer lost the ability to escape the current basin but is not yet in a flat enough one to stay. Fix: use a smaller step factor (0.5 instead of 0.1) or switch to cosine annealing. With cosine: plateau in the middle of the schedule means the peak $α$ was too low — the early phase did not explore broadly enough. Fix: increase peak $α$ by 3x and rerun. Log $α$ alongside loss at each step to see exactly when plateaus align with schedule changes.

Recap

• No single fixed LR is best across a whole run: too high and the loss oscillates from epoch 1; too low and the optimizer locks into a mediocre minimum. You want big steps early to cover ground, small steps late to settle — hence a *schedule*.
• Warmup ramps α from near-zero over the first ~1–5% of steps: early gradients are unreliable in *direction*, not just biased in magnitude, so a big early step on a fresh unstable model can blow it up.
• Warmup fixes direction reliability, not just Adam's bias: even with perfect bias correction the step-1 gradient is essentially noise, so warmup is still needed — the two are separate problems.
• Step decay locks the basin abruptly: dropping the rate 10× at a milestone freezes the optimizer wherever it happens to be at that moment — cosine decay anneals smoothly to near zero instead, avoiding a premature lock-in.
• Cosine annealing tends to find flatter basins than step decay, worth roughly 0.5–2% test accuracy on benchmarks — the smooth glide-down lets the optimizer keep settling rather than snapping into place.
• OneCycleLR: ramp up to a 5–10× peak over the first ~30% of steps, then cosine-decay the rest — aggressive exploration up front, fine convergence at the end.
• Canonical Transformer schedule = linear warmup + cosine decay to near zero, worth 5–10% accuracy over a naive fixed rate; it also pairs with early stopping so training both moves fast and lands cleanly.

Check your understanding