Evaluation in Production

Shadow mode, staged rollout, guardrail metrics, significance at scale

An A/B test runs for 2 days. The treatment shows +18% CTR. The team ships it. Two weeks later, revenue is down 4%. What happened?

Novelty. People click new things just because they are new. Once the shine wears off, engagement settles back — and here, below baseline. The 2-day test ran entirely inside the novelty window, so it measured novelty, not the model. The team optimised for a signal that was never going to last.

Production evaluation is the craft of measuring a model's real impact in ways that survive traps like this. The guiding rule: every ship / no-ship call should ultimately rest on a *controlled experiment*. Everything else — offline metrics, shadow mode, proxy-metric checks — exists to cheaply filter candidates so you only spend real A/B tests on the few worth it.

Shadow mode first

Before any high-stakes launch, run the new model in shadow mode: it sees the same live traffic and makes predictions, but only the current champion's predictions actually reach users. This proves the serving path works under real load, catches weird output distributions, and surfaces big disagreements between the two models — all before a single user is affected. It catches the infrastructure bugs that offline testing simply cannot.

Three A/B rules people keep breaking

1. Run for at least two full business cycles — a minimum of two weeks, four if the product is strongly seasonal. A 2-day win has almost certainly not outlived novelty.

2. Fix the primary metric and required sample size *before* you start. Glancing at results midway and stopping the moment p < 0.05 wrecks the statistics — with daily peeking at the 0.05 level, your real false-positive rate climbs to roughly 30% by day 20. If you must peek, use a sequential test built for it.

3. Set guardrail metrics up front — secondary numbers that are not allowed to get worse even if the main metric improves. A recommender that lifts CTR by pushing sensational content while tanking session length is worse than nothing; guardrails catch that before it ships.

"Significant" is not the same as "worth it."

A small p-value only says the effect is unlikely to be pure noise. It does not tell you the effect will *last*, that it is *big enough to matter*, or that the experiment was *run properly*. At ten million daily users, a 0.01% CTR bump hits p < 0.0001 in a single day — and that is about a thousand extra clicks. Is a whole new model to retrain, deploy, and maintain worth a thousand clicks a day? "Is this effect real?" and "Is this effect worth acting on?" are different questions, and at scale the first is almost always yes. So a trustworthy test needs all of it: random assignment (a user always in the same arm), enough time, a pre-registered metric and sample size, and guardrails watched throughout.

Check the lift holds across segments

A significant, practically-meaningful average lift can still be an artifact of one subgroup carrying the whole result. If the aggregate lift comes almost entirely from one platform, one region, or one power-user cohort while every other segment is flat or negative, you have found a group whose behaviour moves the average, not a broadly better model -- and it can vanish, or reverse, in segments you did not check. Before shipping, break the primary metric out by the segments that matter for your product (platform, geography, new vs. returning users) and confirm the lift holds directionally in each one, not just in the pooled number.

Check the split before you read any metric: SRM

The very first sanity check on an A/B test is sample ratio mismatch. You assigned 50/50, so the two arms should have ~equal traffic — if you observe 55/45, something is broken (a bug in assignment, a crash that drops one arm's users, bot traffic hitting one side, a logging gap). And a broken split means every downstream metric is untrustworthy, because the arms are no longer comparable populations. Run a chi-squared test on the observed counts *before* looking at the treatment effect; if SRM fails (roughly, the split deviates more than chance allows), stop and debug the pipeline — do not interpret the result.

Power, MDE, and the winner's curse

Size the test *before* running it. Statistical power is the chance of detecting a real effect of a given size; the minimum detectable effect (MDE) is the smallest lift the test can reliably catch, and it's set by the baseline metric's variance and your sample size. An underpowered test isn't just "might miss a win" — worse, the wins it *does* report are exaggerated (the winner's curse): only unusually large noisy estimates cross the significance line, so significant effects from small samples are biased upward. Compute the required sample size from a target MDE and power (usually 80%) up front, and be skeptical of large effects from small tests.

Faster tests with variance reduction: CUPED

You can reach significance sooner without more traffic by *reducing variance*. CUPED (controlled experiment using pre-experiment data) subtracts off each user's pre-experiment behaviour — a heavy pre-period spender is expected to spend heavily regardless of treatment, so removing that predictable component shrinks the metric's variance and tightens the confidence interval. Same effect, smaller error bars, shorter test. Any strong pre-experiment covariate works; CUPED is the standard packaging.

Don't jump straight to 50/50: staged ramp-up

High-stakes launches roll out in stages, each a safety gate: shadow (no user impact) → 1% canary (catch crashes and catastrophic regressions cheaply) → 5% → 10% → 25% → 50% → full, with automated rollback wired to guardrails at every step. The point is to limit blast radius: a bug that would harm 50% of users is caught at 1%. Never take a brand-new model straight to half of traffic.

When one user's treatment leaks to another: interference

Standard A/B analysis assumes SUTVA — one user's outcome depends only on their *own* assignment. Whole classes of systems violate this. In a marketplace, giving treatment sellers a boost takes impressions *away* from control sellers, so the control group is contaminated and the measured lift is inflated. Social networks (a treated user posts more, changing their control-group friends' feeds) and shared-resource systems (a treated user consumes limited inventory) have the same problem. The fixes are cluster-level randomisation (randomise by market/region/social-cluster instead of by user) or switchback designs — plain user-level A/B tests give biased answers under interference.

Short-term wins, long-term harm: delayed metrics

The metric that moves inside the test window is often not the one you care about. CTR responds in minutes; retention, revenue quality, refunds, churn, and lifetime value unfold over weeks or months. A model that boosts short-term clicks by pushing clickbait can simultaneously *reduce* long-term retention — and a two-week test may never see it. Guardrails plus a long-lived holdout group (a fraction of users permanently excluded from all changes) are how you catch effects that only appear after the experiment ends. A holdout group also beats running a *series* of individual A/B tests for measuring a model's real long-term value: each individual test only measures one change's effect over its own short window, and changes that each look like a win in isolation can still cancel out or net negative in aggregate -- one feature's engagement gain cannibalising another's. Comparing the holdout's cumulative trajectory against everyone else over months is how you catch that a year of individually-shipped "wins" actually added up to nothing, or worse.

Overlapping experiments: layers and mutual exclusion

At scale dozens of tests run at once, and they can interact — test A changes the ranking that test B is also modifying, so the effects aren't additive. Two defenses: mutual exclusion (tests touching the same surface can't share users) and experiment layers (an orthogonalisation scheme where each user is in one experiment per layer, so tests in different layers are randomised independently). Use factorial design deliberately when you *want* to measure an interaction, but only after confirming independence. Document which tests ran simultaneously so surprising results can be traced.

Key points

• Always run A/B tests for at least 2 full business cycles before making a decision — novelty effects typically last 3–7 days and can fully reverse a 15% positive signal. For the recommendation test: the +18% CTR was measured on days 1–2. On days 3–14, CTR in the treatment arm declines steadily as novelty fades. By day 14, CTR in treatment is 2% below control -- the same underlying reversal that shows up as the -4% revenue drop the team discovers two weeks after shipping in the summary above. If the test had run for 2 weeks, the team would have seen the reversal before shipping. The diagnostic: plot daily CTR for treatment vs control separately. If treatment CTR is declining toward control CTR over the first 7 days, novelty is explaining the effect. If treatment CTR is stable and above control at day 14, the effect is real.
• Trap: peeking at results before the planned end date inflates Type I error. Running a sequential test that checks significance daily at α = 0.05 gives a true false positive rate of ~30% by day 20. Use sequential testing methods (mSPRT, always-valid inference) if you need to peek. For the recommendation test: the team checks results every morning instead of waiting for a pre-set stopping point. By day 2, p crosses 0.05 -- this is the same +18% CTR result described in the summary above -- so they stop the test right there and ship. The problem: every day of peeking at α = 0.05 is an independent opportunity to cross the significance threshold by noise alone. With 20 days of peeking, the probability of at least one false positive reaches ~30%. The fix: use an always-valid sequential test framework (Spotify's SPRT, Microsoft's ExP). These frameworks compute valid p-values at any stopping point by construction, letting you stop early when the result is clear without inflating false positive rate.
• Diagnostic: if your positive A/B test result reverses after shipping, check novelty curves — plot daily CTR separately for users in their first 7 days of exposure vs later. If day-1 CTR is much higher than day-7, you measured novelty. For the recommendation model: after shipping and observing the -4% revenue decline, segment users by days since first exposure to the new model (here "baseline" means each cohort's own pre-launch CTR, not the control arm). New-to-model users (day 1–3): CTR 22% above their own baseline. Users 4–7 days in: CTR 8% above baseline. Users 8+ days in: CTR 3% below baseline. Averaged across the whole treatment population this is the same decay already seen in the aggregate numbers above -- a strong day 1-2 lift (the +18% CTR from the summary) settling toward, then below, control by day 14. The novelty curve is unmistakable. The model was not better — it was new. The operational fix: run all future tests for at least 14 days and require that the treatment effect is stable (non-declining) in the last 7 days before shipping.
• Sanity-check the split, size the test up front, and reduce variance to go faster. Before reading any metric, run a sample-ratio-mismatch (SRM) check — a 50/50 design that comes back 55/45 signals a broken pipeline and makes every downstream number untrustworthy. Compute required sample size from a target MDE and 80% power beforehand, since underpowered tests both miss real wins and exaggerate the ones they report (winner's curse). CUPED (subtracting pre-experiment behaviour) cuts metric variance and shortens tests without more traffic. Roll out in stages: shadow → 1% canary → 5/10/25/50% → full, with automated rollback on guardrails.
• Mind interference, delayed metrics, and overlapping experiments. User-level A/B tests assume SUTVA (one user's outcome depends only on their own arm) — marketplaces, social networks, and shared-inventory systems violate it, so cluster-randomise or use switchbacks or the measured lift is biased. Short-window metrics (CTR) can move opposite to long-run ones (retention, LTV, refunds, churn); catch these with guardrails and a permanent holdout group. When many tests run at once, prevent contamination with mutual exclusion or experiment layers, and use factorial design only when you've confirmed independence and actually want the interaction.

Recap

• The novelty trap: a short test measures novelty, not the model: +18% CTR over 2 days can reverse to −4% revenue in two weeks because people click new things just because they're new, then settle back. The 2-day window sat entirely inside the novelty spike.
• Every ship / no-ship call ultimately rests on a controlled experiment: offline metrics, shadow mode, and proxy checks exist only to cheaply filter candidates so you spend real A/B tests on the few worth it.
• Shadow mode first: run the new model on the same live traffic while only the champion's predictions reach users — this proves the serving path works under load, catches weird output distributions and big model disagreements, all before a single user is affected. It catches infrastructure bugs offline testing simply can't.
• Three A/B rules people keep breaking: run at least two full business cycles (≥2 weeks, 4 if seasonal — a 2-day win hasn't outlived novelty); fix the primary metric and sample size *before* starting (daily peeking at α=0.05 pushes the real false-positive rate to ~30% by day 20 — use a sequential test if you must peek); and set guardrail metrics up front that aren't allowed to regress.
• "Significant" is not "worth it": a small p-value only says the effect is unlikely to be noise. At 10M daily users a 0.01% CTR bump hits p<0.0001 in a day — about a thousand extra clicks. "Is it real?" and "is it worth a new model to retrain, deploy, and maintain?" are different questions.
• SRM check before reading any metric: a 50/50 design that logs 55/45 signals a broken assignment or logging pipeline (a crashed arm dropping users, bot traffic, a logging gap) — run a chi-squared test on the counts first, and if it fails, stop and debug; the arms are no longer comparable populations.
• Size and de-risk the test: compute sample size from a target MDE and 80% power up front (underpowered tests exaggerate their wins — winner's curse); use CUPED (subtract pre-experiment behaviour) to cut variance and finish sooner; ramp in stages (shadow → 1% canary → 5/10/25/50% → full) with automated rollback. Watch for interference (marketplaces/social/shared-inventory violate SUTVA → cluster-randomise or switchback) and delayed metrics (retention, LTV, churn need a permanent holdout group).

Check your understanding