Offline vs Online Evaluation

Proxy metrics, A/B gap, how to close the offline-online divide

A recommendation team retrains their model. The offline holdout NDCG jumps from 0.79 to 0.82 — a clear win. They ship it in an A/B test. CTR drops 3%. The number that was supposed to predict success went up; the real-world number went down. What happened?

Here is the catch. The holdout set was built from interaction logs collected six months ago. So the model was tuned to predict what users *would have clicked*, in a six-month-old catalog, through a six-month-old interface. Deployed today it recommends slightly stale content, misses what is trending now, and chases engagement patterns that have since moved on. The test set was a snapshot of the past. The offline metric was faithfully measuring the wrong world.

Why offline and online disagree

Offline metrics measure how well a model predicts *past* behaviour. Online metrics measure how real users respond *now*. Three structural gaps drive them apart.

Distribution shift. The historical test set is not live traffic — preferences, the catalog, and the context all drift over time. A model tuned on a holdout built from last winter's logs still ranks now-discontinued items near the top and has never seen anything added to the catalog since, so its offline score reflects a version of the product that no longer exists.

Proxy labels. A click is not satisfaction. Someone who clicked a clickbait headline and bounced immediately was not helped, but offline NDCG counts it as a win. Clicks are also position-biased: an item shown at rank 1 gets clicked far more than an equally good item buried at rank 8, simply because it was seen first — so a raw click log conflates relevance with how prominently the old model chose to display the item. You are measuring a stand-in for the thing you actually care about.

Feedback loops. Once deployed, the model *shapes* the data it will later be trained on. Show recommendation A, users interact with A, those interactions become tomorrow's training set. The model changes the future distribution just by being live.

How much do they even correlate?

Nowhere near perfectly. For search and recommendation, the correlation between an offline gain and the matching online gain is usually only about 0.3 to 0.7. A 5% offline NDCG bump might buy a 2% CTR lift, which might buy a 1% revenue lift — the signal fades at every hop. So measure this correlation *on your own system*: over your last ten A/B tests, plot the offline delta against the realised online delta. If they barely track each other (correlation below about 0.5), then redesigning your evaluation matters far more than tuning the model yet again.

The habit that keeps you honest

A strong offline result is a *hypothesis*, not a verdict. Models can overfit the evaluation set itself — a kind of leakage, where information about the evaluation set bleeds into what the model has effectively learned — lifting offline NDCG while making the real experience worse — for instance by memorising which items were in the historical judgment pool. So treat every offline win as "promising, let's test it," never as "done." Two safeguards make offline numbers trustworthy: split your data by *time* (train on the past, test on the most recent window, never a random shuffle across all dates), and shadow-deploy the new model on live traffic before the A/B test to catch distribution shift before it costs you anything.

The deepest reason they diverge: causal vs observational

Here's the framing that ties it together. An A/B test randomises users to treatment or control, so the difference in outcomes is a clean causal estimate of the model's effect — randomisation cancels out confounders. Offline logs are observational: they were generated by the *old* model's policy, so they're policy-biased — you only ever observed outcomes for the items the old system chose to show. Evaluating a new policy on data collected under a different policy is comparing apples to a biased sample of oranges. That's why offline evaluation of a policy that behaves differently from the logging policy is fundamentally hard, and why the A/B test is the gold standard: it's the only step that actually manipulates the variable and measures the effect.

Primary metric versus guardrails

Never judge an experiment on one number. Define a primary metric (the thing you're trying to move, e.g. CTR) plus a set of guardrail metrics you refuse to harm — retention, complaint rate, latency, diversity, revenue quality, unsubscribes. The classic failure: CTR goes up 3% (ship it!) while 90-day retention quietly drops and revenue-per-session falls because the model learned to bait clicks. A win on the primary metric that breaks a guardrail is not a win. List the guardrails *before* the test so you can't rationalise afterward.

Experiment design: power, MDE, duration, peeking

A test can mislead by being badly designed. Statistical power and minimum detectable effect (MDE) set how big a sample you need — underpowered tests exaggerate the effect size of the "wins" that happen to reach significance (winner's curse). Duration must cover full weekly cycles (weekday/weekend differ) and outlast novelty effects (users click a new thing just because it's new, then stop) and seasonality. Peeking — checking the p-value repeatedly and stopping when it crosses 0.05 — inflates false positives massively; use a fixed horizon or sequential-testing corrections. And running many metrics or many variants is multiple testing: correct for it or you'll "find" significance by chance.

Fixing offline logs: IPS and its sharp edges

You can partly de-bias offline log evaluation with inverse propensity scoring — weight each logged outcome by 1/P(the old policy showed this item), which corrects the exposure bias. But IPS is fragile: it has high variance (a tiny propensity makes 1/p explode), needs accurate propensity estimates (often unknown and hard to model), and requires overlap/support (the new policy can only be evaluated where the old policy had some chance of showing the same items). Variants like self-normalised IPS (SNIPS) and doubly-robust estimators reduce the variance, but none fully rescue you when propensities are extreme — which is exactly when you fall back to an actual A/B test.

Interleaving and bandits: where each fits

Interleaving blends two rankers' results into one list and sees which side gets the clicks — it cancels between-user variance and needs far fewer users, but it measures *relative ranking preference*, not absolute business impact, and gets tricky with personalisation, ads, hard constraints, or session-level outcomes. Bandits adaptively route more traffic to the better arm, minimising regret — but that adaptivity makes the allocation endogenous, so you can't read off a clean causal effect the way a fixed-split A/B test gives you, and delayed feedback further complicates them. Use interleaving for cheap ranking comparisons, bandits when minimising regret matters more than a clean estimate, and A/B tests when you need the rigorous causal number.

Key points

• Shadow-deploy before A/B test — run the new model in parallel, log its predictions, evaluate on fresh offline data with matching timestamps. This catches distribution shift before it costs traffic. For the recommendation model: in shadow mode, the new model receives the same production requests as the live model and generates predictions, but only the live model's predictions are served. Log the new model's recommendations and the live outcomes. Evaluate the new model's offline NDCG on data from the past 2 weeks — not data from 6 months ago. If shadow-mode NDCG is lower than training-set NDCG, the model has distribution shift. Investigate before running any A/B test. Shadow mode also validates serving infrastructure under real load before any user is exposed to the new model.
• Trap: using past user interactions as ground truth for future predictions without temporal splitting. Users who clicked item X in January might have completely different preferences in April. Time-stamp your train/test split. For the recommendation model: train on months 1–9, evaluate on month 10. Evaluate on month 10 data only, using only items and users present in month 10. The alternative — a random 80/20 split across all months — lets the model train on August data to predict March clicks. August preferences contaminate the March predictions. Temporal splitting is non-negotiable for any system where item relevance changes over time, which is most recommendation and search systems.
• Diagnostic: measure the Spearman rank correlation between offline metric deltas and online metric deltas across your last 10 model launches. If the correlation is < 0.5, your offline metric is a poor proxy — redesign the offline evaluation before investing in further model development. For the recommendation team: collect the log of all model launches from the past year. For each launch, record the offline NDCG delta (new vs previous model) and the realized A/B CTR delta. Compute Spearman ρ. If ρ < 0.5, offline NDCG is not reliably predictive of CTR. Consider redesigning the offline evaluation: use more recent evaluation data, add temporal evaluation windows, add session-level signals beyond clicks, or use debiased click labels via Inverse Propensity Scoring.
• An A/B test is causal; offline logs are observational and policy-biased — and one primary metric is never enough. Randomisation makes the A/B difference a clean causal effect; offline logs only recorded outcomes for what the old policy chose to show, so evaluating a different policy on them is biased. Always pair a primary metric with guardrails (retention, complaints, latency, diversity, revenue quality) defined before the test — CTR up while retention drops is not a win. IPS can partly de-bias offline logs but has high variance under small propensities, needs accurate propensities and overlap, and even SNIPS/doubly-robust don't fully rescue extreme cases.
• Design the experiment properly and pick the right online tool. Set sample size from power and MDE (underpowered tests exaggerate winners), run across full weekly cycles past novelty and seasonality, and don't peek — repeated significance checks inflate false positives, so fix the horizon or use sequential corrections, and adjust for multiple metrics/variants. Interleaving is a cheap, low-variance ranking comparison but measures relative preference, not absolute impact; bandits minimise regret but their adaptive, endogenous allocation blocks a clean causal estimate. Use A/B tests when you need the rigorous causal number.

Recap

• Offline measures how well a model predicts the *past*; online measures how real users respond *now*: a holdout built from six-month-old logs measures the wrong world, which is why offline NDCG can jump 0.79→0.82 while the shipped model's CTR drops 3%.
• Three structural gaps drive them apart: distribution shift (preferences, catalog, context drift), proxy labels (a click on clickbait that bounced isn't satisfaction, but NDCG counts it a win), and feedback loops (once live, the model shapes the very data it's next trained on).
• The offline→online correlation is weak — usually ρ ≈ 0.3–0.7 — and the signal fades at every hop (5% NDCG → maybe 2% CTR → maybe 1% revenue). Measure it on *your own* system across the last ten A/B tests; if it's below ~0.5, redesigning evaluation beats tuning the model again.
• An A/B test is causal; offline logs are observational and policy-biased: randomisation cancels confounders, so the A/B difference is a clean causal effect. Offline logs only recorded outcomes for what the *old* policy chose to show, so evaluating a different policy on them compares apples to a biased sample of oranges — the A/B is the gold standard.
• Pair a primary metric with guardrails defined *before* the test: CTR up 3% while 90-day retention drops and revenue-per-session falls is not a win. List the guardrails (retention, latency, complaints, diversity, revenue quality) up front so you can't rationalise afterward.
• Design the experiment properly: size it from power and MDE (underpowered tests exaggerate the wins that reach significance — winner's curse), run across full weekly cycles past novelty and seasonality, don't peek (repeated significance checks inflate false positives — fix the horizon or use sequential corrections), and correct for multiple metrics/variants.
• Know the online-eval tools: IPS de-biases offline logs (weight by 1/P(old policy showed it)) but has high variance under tiny propensities and needs overlap — SNIPS/doubly-robust help but don't rescue extreme cases; interleaving is a cheap, low-variance *relative* ranking comparison (not absolute impact); bandits minimise regret but their adaptive allocation is endogenous, blocking a clean causal estimate.

Check your understanding