Offline vs Online Evaluation
Why NDCG/recall diverge from CTR/retention — the offline-online gap and A/B tests
You can compute a beautiful NDCG@10 on last month's logs, ship the model, and watch engagement *drop*. This isn't a bug in your metric — it's the structural gap between what offline evaluation can measure and what the live system actually does. Understanding *why* the two diverge is what separates a metric-chaser from someone who can be trusted to ship.
Offline metrics score a ranking against logged relevance; each measures something narrower than "good recommendations." Recall@k — did the relevant items make the shortlist (retrieval's metric). NDCG@k — are relevant items ordered near the top, position-discounted (ranking's metric). MAP — mean average precision across the ranked list. AUC — probability a random relevant item outranks a random irrelevant one. All are computed on *historical* logs, and that's the catch.
Why offline and online diverge — four structural reasons
- Logs are biased: offline "relevance" is what the *old* model exposed, so a new model that surfaces different-but-good items is *penalized* offline for disagreeing with the old model's choices. - Counterfactual blindness: offline metrics can only score items that were logged; a genuinely better item the old system never showed has no label, so its win is invisible. - Metric ≠ objective: NDCG on clicks rewards clickbait; the business cares about dwell, retention, revenue, and harm — offline click-NDCG can rise while long-term value falls. - No system effects: offline eval can't see diversity, freshness, feedback loops, or how the recommendation changes future behavior.
Online evaluation measures the objective directly, via A/B tests. Split traffic, ship the candidate to one arm, and measure CTR, dwell, session length, retention, revenue — the things offline metrics only proxy. The discipline: offline metrics are a cheap *filter* (kill obviously worse models before they touch users), never the *decision* (the A/B test decides). Watch for the traps — novelty effects (a new model gets a temporary bump), delayed metrics (retention takes weeks), guardrail metrics (don't win CTR by raising a harm metric), and sample-ratio mismatch (SRM: the observed control/treatment traffic split doesn't match the intended ratio — e.g. 48/52 instead of 50/50 — a sign the randomization or logging pipeline itself is broken, which invalidates trust in the whole test's readout, not just one metric). The offline-online gap is not eliminated; it's *managed*: use offline to filter, online to decide, and treat a persistent gap as a signal that your offline proxy or your logs are broken.
Key points
- Offline metrics are cheap proxies computed on biased historical logs; online metrics measure the objective on live traffic. Recall@k / NDCG@k / MAP / AUC each capture a slice of ranking quality; CTR / dwell / retention / revenue capture what the business actually wants. The former filter; the latter decide.
- The gap is structural, not accidental — four causes. (1) Logs are biased toward the old model, penalizing new-but-good rankings for disagreeing; (2) counterfactual blindness — unlogged better items have no label; (3) metric ≠ objective (click-NDCG rewards clickbait while retention falls); (4) offline can't see diversity, feedback loops, or downstream behavior change.
- A/B testing is the decision procedure, with its own traps. Novelty effects, delayed metrics (retention takes weeks), guardrail metrics (don't win CTR by raising a harm metric), and sample-ratio mismatch all corrupt naive readouts. Ramp gradually and pre-register the north-star + guardrails.
- Manage the gap, don't pretend it's closed. Use offline to filter obviously worse models, online to decide, and treat a *persistent* offline-online divergence as evidence your offline proxy or your logging pipeline is broken.
Offline metrics (recall@k, NDCG@k, MAP, AUC) are cheap proxies on biased historical logs; online metrics (CTR, dwell, retention, revenue) measure the real objective on live traffic. They diverge structurally — biased logs, counterfactual blindness, metric≠objective, and unseen system effects — so the discipline is offline-to-filter, online-A/B-to-decide, treating a persistent gap as a broken-proxy alarm rather than noise.
Recap
- Offline = cheap proxies on biased logs; online = the real objective on live traffic. Recall@k (did relevant items make the shortlist), NDCG@k (ordered near the top, position-discounted), MAP, AUC (P(relevant outranks irrelevant)) — all computed on *historical* logs. CTR/dwell/retention/revenue are what the business actually wants.
- The gap is structural — four causes: (1) logs are biased toward the old model, so a new-but-good ranking is penalized for disagreeing; (2) counterfactual blindness — unlogged better items have no label; (3) metric ≠ objective (click-NDCG rewards clickbait while retention falls); (4) no system effects (diversity, feedback loops, behavior change).
- A/B testing is the decision procedure, with traps: novelty effects (temporary bump), delayed metrics (retention takes weeks), guardrail metrics (don't win CTR by raising a harm metric), sample-ratio mismatch. Ramp gradually; pre-register north-star + guardrails.
- Discipline: offline to filter, online to decide. Offline kills obviously worse models cheaply; the A/B test makes the call. A *persistent* offline-online gap is a signal your offline proxy or logging pipeline is broken — not noise to average away.
Check your understanding
Q1. A new ranker has clearly higher NDCG@10 offline than the incumbent, but in an A/B test its engagement is *lower*. Which explanation is most consistent with how offline evaluation works?
- A) The A/B test is underpowered given typical RecSys effect sizes (~1-2% CTR deltas); extend it to a 4-week run and NDCG's offline verdict will eventually be confirmed by live traffic.
- B) Offline NDCG is computed against logged relevance from the *incumbent*, so agreeing with its past choices scores higher offline without being better for users; trust the A/B test.
- C) NDCG is simply the wrong offline metric for this comparison; recompute both models' offline scores using MAP@10 instead and the two will agree with the A/B result.
- D) The new ranker overfit the offline evaluation set during hyperparameter search; add dropout and early stopping and both offline and online numbers will converge.
Q2. Why can't offline evaluation reliably credit a model that recommends genuinely great items the old system *never showed*?
- A) It can — modern offline harnesses simulate a synthetic user-response distribution for unshown items directly from the new model's own predicted scores, which is a statistically valid substitute for a real label.
- B) Counterfactual blindness: an item the old system never exposed has no logged interaction, so its winning recommendation is invisible to offline scoring; only an online test or a propensity-based counterfactual estimator can credit it.
- C) Offline metrics assign unshown items a default relevance score of exactly zero, and this convention is provably unbiased under the standard NDCG formulation.
- D) The ANN index physically excludes any item that has never received a click from its searchable graph, so unshown items cannot even be retrieved for offline scoring.
Q3. Your candidate ranker wins CTR in an A/B test. Before shipping, select the *two* most important additional checks.
- A) Guardrail metrics — confirm the CTR win didn't come at the cost of a harm metric (reports, "see fewer") or of dwell/retention, since click-optimizing models often win CTR via clickbait.
- B) Rule out a novelty effect (a temporary CTR bump that fades once the model stops being new) and a sample-ratio mismatch (a broken randomization/logging pipeline that invalidates trust in the whole test's readout, not just a fading bump).
- C) Confirm the candidate's p99 inference latency is strictly below the incumbent's, since a slower model is disqualified from replacing it in production regardless of CTR.
- D) Confirm the ranker's output embedding dimension exactly matches the retrieval stage's 64-dimensional space, since a mismatch would leave the downstream dot product undefined.
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →