Online Learning & Model Staleness
Continuous training, concept drift, staleness management, shadow deployment
A news recommender retrains once a week. A big crypto story breaks Tuesday and interest spikes within hours. The weekly model doesn't surface crypto until Sunday — five days after the peak. An online model, updating on every click in real time, is surfacing crypto within minutes. For trending content, that five-day lag is real lost engagement.
That's the case for online learning. But it's a *narrow* case, so start with a measurement, not a preference.
First ask: is batch actually too slow?
Compare a daily-retrained model against an online model on the metric you care about. If daily is close enough, take the simpler system — because online learning gives up three things a good model wants. It's harder to *debug* (the model changes constantly — which update broke it?), harder to *audit* (there's no fixed version to evaluate), and harder to *roll back* (revert to what?). Reach for it only when retraining latency demonstrably hurts the business metric.
If it's warranted, pick the right algorithm — and mind deep models
FTRL (Follow-The-Regularized-Leader) is Google's workhorse for large-scale online *linear* models (ads serving): it keeps a running per-feature sum of past gradients and uses that sum, weight by weight, to decide whether to zero the weight out entirely (L1) or just shrink it (L2) — efficient because it never has to store or replay individual past examples, only the running sums. Online SGD updates weights per example or mini-batch, fine for linear models and shallow nets. But deep models hit catastrophic forgetting: trained on a stream, they overwrite what they haven't seen lately. Feed a recommender two weeks of crypto-heavy traffic and it starts recommending *only* crypto, having forgotten sports, politics, and finance. Fixes: *experience replay* (mix a buffer of diverse old examples into each update) and *elastic weight consolidation* (protect the weights that mattered for older patterns).
Rolling out the online model needs its own staged process — you don't flip all traffic to a model that updates every second.
Two stages, in order. First, shadow mode, typically 1-2 weeks: the challenger runs alongside the current production model on live traffic, its predictions get logged, but it never actually serves a response to a user. This is where you catch the challenger silently degrading (or improving) before it can hurt anyone, and 1-2 weeks is enough to see it across a full weekly traffic cycle. Second, once shadow mode looks clean, canary release: start serving the challenger's real predictions to a small slice of traffic — 1% — and only widen that slice (5%, 25%, 50%, 100%) after each step clears fixed metric gates (the business metric you're optimizing, plus guardrails like latency and error rate). Any gate failure at any step rolls the slice back to 0%, not just pauses it.
The sneakiest failure: feedback loops
The model's own predictions shape user behavior, that behavior becomes training data, and the data reinforces the predictions. A model that leans slightly toward crypto drives crypto clicks, which show up as a strong crypto signal, which makes it lean harder — a filter bubble compounding at the speed of online updates. Break it with *exploration*: spend a small fixed slice of traffic, say 5%, on picks other than the model's top prediction (ε-greedy), or sample each candidate's predicted click-through rate from its own uncertainty range rather than always serving the single highest point estimate, so an under-shown category with wide uncertainty still gets a fair shot at winning that slice (Thompson sampling). Either way, categories the model hasn't reinforced yet keep getting real traffic instead of being starved out.
Not all "the world changed" is the same problem — three kinds of shift, three different fixes.
Split the joint distribution P(X, Y) into inputs and label: covariate shift is P(X) changing while P(Y|X) — the actual input-to-label relationship — stays the same (e.g. your fraud model sees more mobile traffic this quarter, but a given transaction pattern is exactly as fraudulent as before). Because the relationship itself hasn't moved, this can often be corrected without retraining, by *importance weighting*: reweight training examples to look more like the new input distribution. Label shift is P(Y) changing on its own (e.g. fraud becomes rarer overall, say during a platform-wide crackdown) while P(Y|X) again stays fixed; this can often be fixed with *threshold recalibration* alone — the model's scores are still valid, just the operating point needs to move with the new base rate. Concept drift is the one that actually forces retraining: P(Y|X) itself changes — the same transaction pattern that used to be safe now genuinely is fraud, because fraudsters adapted their behavior. No amount of reweighting or threshold-tuning fixes a relationship that has itself changed; you need new labeled data reflecting the new relationship.
The bottom line: "more current is always better" is wrong, because recency is only one virtue — auditability, debuggability, and one-click rollback are others, and batch retraining has all of them (rollback is just a registry promotion). For most systems, periodic batch retraining is the more reliable choice. Online learning wins only when the metric clearly suffers from retraining lag *and* the measured gain justifies the extra complexity.
Key points
- Use FTRL or online SGD only when the retraining-to-deployment cycle is measurably too slow for your use case — establish this empirically before committing to the infrastructure. Compare a daily-retrained model against an online model on a key business metric. If the gap is small, choose the simpler system. The engineering cost of online learning — continuous deployment pipeline, feedback loop monitoring, catastrophic forgetting mitigations, audit trail for a model that changes every second — is only worth paying when the measurement shows a gap that matters to the business.
- Trap: feedback loops in online learning create filter bubbles and popularity bias that compound over time. The model's predictions influence user behavior, which becomes training data, which reinforces the predictions. A small initial bias amplifies on every update cycle. Add exploration (ε-greedy, Thompson sampling) to the online model to break the feedback loop — without exploration, the model collapses toward the already-popular and the already-predicted, destroying the diversity that makes recommendations valuable.
- Diagnostic: track concept drift metrics alongside the online model's performance — if drift is low but performance is degrading, the feedback loop is the problem; if drift is high and performance is degrading, the model is not adapting fast enough. These two failure modes look similar from the outside (degrading performance) but require opposite interventions. Feedback loop: add exploration, reduce learning rate, increase replay buffer diversity. Insufficient adaptation: increase learning rate, reduce replay buffer weight, consider triggered full retraining on large drift events.
Online learning solves a real problem — retraining latency — but creates three new ones: feedback loops, catastrophic forgetting, and a continuously changing model that cannot be audited, debugged, or rolled back the way a batch-trained model can; use it only when the measurement shows the tradeoff is worth it.
Recap
- Case for online: weekly news recommender lags a crypto spike by 5 days; an online model surfaces it in minutes.
- But start with a measurement, not a preference: is daily-retrained batch actually too slow on the metric you care about?
- Online gives up three things: harder to debug (which update broke it?), audit (no fixed version), roll back (revert to what?).
- Algorithms: FTRL for large sparse linear (ads), online SGD for linear/shallow. Deep models hit catastrophic forgetting.
- Catastrophic forgetting: 2 weeks of crypto traffic → recommends only crypto. Fix with experience replay + elastic weight consolidation.
- Rollout is staged too: shadow mode (1-2 weeks, log predictions, don't serve) then canary (1% → 100% with metric gates at each step, rolled back to 0% on any gate failure).
- Feedback loops: predictions shape behavior → behavior becomes training data → bias compounds. Break with exploration (ε-greedy, Thompson).
- Three kinds of shift, three fixes: covariate shift (P(X) changes, P(Y|X) stable) → importance weighting; label shift (P(Y) changes) → threshold recalibration; concept drift (P(Y|X) itself changes) → retraining required.
- "More current is always better" is wrong: batch has auditability, debuggability, one-click rollback. Use online only when the measured gain justifies it.
Check your understanding
Q1. Your fraud detection model's precision starts dropping 3 weeks after deployment with no code changes, and feature distributions are stable. Select the two correct diagnoses/actions.
- A) This is most likely real concept drift — fraudsters have adapted and the relationship P(fraud|features) has changed
- B) Stable feature distributions guarantee stable model performance by definition; the precision drop must be a monitoring dashboard artifact
- C) Investigate false positive clusters for new fraud patterns and retrain with recency-weighted recent data on a shortened retraining cycle
- D) Stable features with dropping precision means the model was undertrained from the start; increase training epochs by 50% and redeploy the same architecture
Q2. You are deploying a new recommendation model. Walk through the shadow mode and canary release process.
- A) Deploy directly to 50% traffic using a fixed traffic-splitting cookie, monitor for exactly 24 hours, then promote to 100% automatically if metrics stay stable
- B) Shadow mode (1-2 weeks): deploy alongside production, log both models' predictions without serving the challenger; then canary from 1% to 100% with metric gates at each step
- C) Shadow mode is only needed for models with a significantly different transformer architecture; for same-architecture models sharing a tokenizer, go directly to canary at 10%
- D) Skip shadow mode entirely and run the canary for a fixed 3 days at 50% traffic split on a single availability zone — a longer canary period is inherently more informative than any shadow-mode comparison
Q3. A streaming recommendation model is trained online (each user interaction updates the model weights immediately). After 2 weeks, you notice the model systematically recommends items from only 3 categories. What has happened?
- A) The model has correctly identified, via a Thompson-sampling bandit layer, the 3 most popular categories and is now optimally optimizing purely for short-term engagement
- B) Catastrophic forgetting from recency bias — 2 weeks of category skew has overwritten other categories through repeated SGD updates; fix with a replay buffer and elastic weight consolidation
- C) Three-category collapse is a well-known random-seed initialization artifact specific to two-tower recommendation models trained with Xavier initialization; simply reinitialize the embedding layer and retrain from scratch
- D) Online learning, by construction, cannot ever cause category collapse; the real issue must be in the feature pipeline producing systematically biased category one-hot encodings
Q4. Define covariate shift, label shift, and concept drift precisely. For each, describe whether retraining is required and what other interventions are available.
- A) All three types of drift, per the standard MLOps taxonomy used across most large production teams today, require immediate full retraining on at least 30 days of fresh incoming data — distinguishing between them is considered a purely academic exercise
- B) Covariate shift (P(X) changes, P(Y|X) stable): importance weighting can correct it without retraining. Label shift (P(Y) changes): threshold recalibration may suffice. Concept drift (P(Y|X) changes): retraining is required
- C) Covariate shift always requires full retraining on a GPU cluster; label shift and concept drift can both be fixed with simple threshold recalibration alone, no retraining needed
- D) Only concept drift requires any intervention at all; covariate shift and label shift both self-correct automatically as the model receives more streaming data over time
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →