Online Experimentation for ML Launches
Power/MDE, SRM, guardrails, CUPED, sequential/peeking — the A/B machinery for shipping a model safely
Your challenger model beat the champion by 3% AUC offline. That number does not launch anything. Offline AUC is measured on logged data the *old* model shaped — the same feedback loops, the same exposure bias — and it says nothing about what happens when the new model actually changes what users see. The only way to know a model is better *for the product* is to run it live against the incumbent and measure the metric you actually care about: engagement, revenue, retention. That live comparison is an A/B test (champion vs challenger), and getting it right is a distinct skill from training the model. This module is the launch-safety subset of experimentation — the deeper causal-inference theory lives in PAL; here we cover what a DS/MLE must not get wrong when shipping a model.
Power and minimum detectable effect: size the test before you run it. A test's *power* is the probability it detects a real effect of a given size. Power depends on three things: the effect size you want to catch (the minimum detectable effect, MDE), the variance of the metric, and the sample size. They trade off — smaller MDE, or noisier metric, means you need more samples. The killer failure mode is the underpowered test: too few users to detect the effect you care about, so a genuine 1% lift comes back "not significant" and you either wrongly kill a good model or, worse, ship on a fluke. An underpowered test is a coin flip dressed up as evidence. Standard practice: fix α (false-positive rate, usually 0.05) and target power (usually 0.80), decide the MDE that matters commercially, then *solve for the sample size and runtime* up front. If the math says you need six weeks of traffic to detect a 0.5% lift and you only have one week, you know that *before* you burn the experiment.
Sample-ratio mismatch: the plumbing sanity check. You assigned 50/50, but the logs show 51.7% control and 48.3% treatment. That 1.7-point gap sounds tiny — but at scale it is astronomically unlikely by chance, and it means the *randomization itself is broken*: maybe treatment errors out and those users silently drop, maybe a redirect fails, maybe logging is lossy on one arm. When assignment is broken, the two groups are no longer comparable and every downstream metric is untrustworthy — including the win you were about to celebrate. The check is a chi-square goodness-of-fit test on the observed split against the intended ratio. If SRM fires, you stop and fix the plumbing; you do not interpret the results. It is the first thing a senior person looks at.
Guardrails: a win on the target metric is not a launch. Ranking models are notorious for this: the new model lifts click-through by 2% while quietly raising p99 latency by 40ms, or lifting engagement while depressing revenue-per-session, or boosting short-term clicks while increasing report/block rates. Guardrail metrics — latency, error rate, revenue, crash rate, unsubscribe rate — are the non-negotiables the launch must not harm. You monitor them alongside the target metric, and a target-metric win that trips a guardrail is *not a ship*. This is where model launches differ most from generic A/B tests: an ML model's failure modes (a latency regression from a heavier network, a fairness regression on a slice, a feedback loop that degrades over weeks) show up in guardrails, not in the headline metric.
CUPED: buy statistical power for free with pre-period data. Much of a metric's variance is baseline user difference, not treatment effect — a heavy spender spends a lot in both arms. CUPED (Controlled-experiment Using Pre-Existing Data) uses each user's *pre-experiment* behavior as a covariate to subtract off that predictable baseline: it forms an adjusted metric `Y_adj = Y − θ(X − E[X])`, where X is the pre-period value and θ is chosen to minimize variance. The treatment effect is unchanged (X is pre-treatment, so it can't be affected by the assignment), but the *variance* drops by roughly the squared pre/post correlation. Less variance means a smaller MDE at the same sample size — often a 30–50% variance reduction, which can turn a six-week test into three weeks. It is one of the highest-leverage tricks in the toolkit and costs nothing but a join to historical data.
Sequential testing and the peeking problem. A classic fixed-horizon test is only valid if you look *once*, at the pre-planned sample size. The temptation is to watch the dashboard and stop the moment p < 0.05. But every additional look is another chance for noise to cross the threshold — repeated peeking at a fixed-α test inflates the true false-positive rate far above the nominal 5% (peek continuously and it approaches 100%). Model launches make this worse because the pressure to ship or roll back *now* is intense. The fix is methods designed for continuous monitoring: sequential testing and always-valid p-values / confidence sequences (e.g. mixture sequential probability ratio tests, group-sequential boundaries), which spend the error budget across looks so you can stop early — for a win *or* a guardrail breach — without inflating false positives. If you need to peek, use a method built to be peeked at.
Key points
- Size the test before running it — power depends on MDE, variance, and n. Fix α (0.05) and power (0.80), pick the commercially meaningful MDE, then solve for sample size and runtime up front. An underpowered test is a coin flip: a real lift returns "not significant" and you kill a good model or ship a fluke. Roughly n ∝ variance / MDE² per arm, so halving the MDE quadruples the users needed.
- SRM is the first sanity check — a broken split invalidates every downstream number. A 50/50 assignment landing at 50.0/48.3 is astronomically unlikely by chance and signals broken randomization (treatment erroring out, lossy logging, failed redirects). Run a chi-square goodness-of-fit on the observed split; if it fires, stop and fix the plumbing before interpreting any metric.
- Guardrails gate the launch — a target-metric win that trips one is not a ship. Monitor latency, error rate, revenue, crash/report rates alongside the target metric. ML models fail specifically here: a heavier network adds p99 latency, a feedback loop degrades a slice. Guardrails, not the headline metric, catch the launch-specific damage.
- CUPED reduces variance for free using pre-period data — same effect, less noise, smaller MDE. Adjust Y with a pre-experiment covariate X: Y_adj = Y − θ(X − E[X]). Because X is pre-treatment the effect estimate is unbiased, but variance drops by ~ the squared pre/post correlation (often 30–50%), shrinking the MDE or runtime at the same n.
- Peeking at a fixed-horizon test inflates false positives — use sequential methods to stop early. Every extra look is another chance for noise to cross α; continuous peeking pushes the true FPR toward 100%. Always-valid p-values / confidence sequences / group-sequential boundaries spend the error budget across looks, so you can stop for a win or a guardrail breach without breaking the guarantee.
Shipping a model safely is an A/B discipline, not a modeling one: pre-compute the sample size from your MDE, α, power, and metric variance so the test isn't a coin flip; check SRM (chi-square on the split) before trusting any result; gate the launch on guardrails (latency/error/revenue), because a target-metric win that trips one is not a ship; use CUPED to cut variance with pre-period data and buy back power for free; and if you must watch the dashboard, use sequential / always-valid methods so peeking doesn't inflate your false-positive rate.
Recap
- Offline metrics don't launch models — live A/B does. Offline AUC is measured on data the old model shaped and ignores product impact. Ship via a champion/challenger test on the real metric (engagement/revenue/retention) plus guardrails.
- Size before you run: power = f(MDE, variance, n). Fix α=0.05 and power=0.80, pick the MDE that matters, solve for n and runtime. n ∝ variance/MDE² per arm — halving the MDE quadruples the users. An underpowered test is a coin flip.
- SRM first: a broken split invalidates everything. A 50/50 landing at 50/48 at scale is not chance — it's broken randomization/logging. Chi-square on the split; if it fires, stop and fix the plumbing before reading any metric.
- Guardrails gate the ship. Latency, error rate, revenue, report/crash rates must not regress. A target-metric win that trips a guardrail (e.g. +2% clicks, +50ms p99) is not a launch — this is where ML-specific failure modes surface.
- CUPED buys power for free; sequential methods make peeking safe. CUPED subtracts pre-period variance (Y_adj = Y − θ(X − E[X])) — unbiased effect, ~30–50% less variance, smaller MDE. And repeated peeking inflates false positives, so use always-valid / sequential tests to stop early without breaking α.
Check your understanding
Q1. Your challenger model shows +3% AUC offline. Your manager wants to ship it based on that number. Which two of the following are correct? Select two.
- A) Offline AUC reflects data the old model shaped, so it does not measure real product impact on its own, no matter how large the lift looks
- B) The right next step is a live champion/challenger A/B test, sized for the MDE that matters, and judged on the real metric plus guardrails
- C) Ship to 100% but keep a rollback plan ready to go — offline AUC combined with a fast rollback is functionally equivalent to a real A/B test
- D) Retrain with more data until offline AUC clears +5%, then ship directly, since a bigger offline gap removes any real need for a live test
Q2. You planned a 50/50 experiment. After a day, exposure logs show 51.1% control and 48.9% treatment across 2 million users. What should you conclude and do?
- A) A 2-point gap like this is well within normal randomization noise at this scale of traffic; proceed and interpret the metrics completely as usual
- B) At 2M users that split is astronomically unlikely by chance, so randomization or logging is broken; stop, run a chi-square, fix the plumbing first
- C) Treatment is simply less popular with users here in this particular case, which is itself taken as a valid result — log it as a negative finding overall
- D) Re-randomize only the treatment arm going forward to rebalance it back to 50/50, then simply continue running the very same experiment as before
Q3. Your new ranking model lifts the target metric (session clicks) by +2.1% with p < 0.01. During the test, p99 serving latency rose from 80ms to 130ms and revenue-per-session dropped 0.8%. Do you launch?
- A) Yes — the target metric already won decisively at p < 0.01, and latency plus revenue are only ever secondary to the primary success metric here
- B) No — latency and revenue are guardrails the launch must not harm; extra p99 latency plus dropped revenue is a breach here, not a ship decision
- C) Yes, but only roll it out to mobile users specifically, since this particular latency regression only really matters on slower connections overall
- D) Re-run the whole test again without logging latency at all this time, so that the guardrail cannot possibly block a statistically significant win
Q4. Your metric is noisy and the required sample size for a 1% MDE is six weeks of traffic you don't have. A colleague suggests CUPED. What does CUPED do and why does it help here?
- A) CUPED increases the measured treatment effect itself by adjusting outcomes upward across the board, so the same effect becomes easier to detect
- B) CUPED subtracts predictable pre-experiment variance via a covariate; effect stays unbiased since X is pre-treatment, but variance drops a lot
- C) CUPED simply raises the significance threshold alpha from 0.05 up to 0.10, which makes reaching significance easier without needing any more data
- D) CUPED just replaces the noisy metric outright with a different proxy metric that happens to have lower variance by definition from the start
Q5. A PM keeps refreshing the experiment dashboard and wants to stop the test "the moment it hits p < 0.05." Why is this dangerous on a standard fixed-horizon test, and what is the fix?
- A) It is fine to do this — p < 0.05 means 95% confidence whenever it is observed, so stopping early right at that threshold is always statistically valid
- B) Each extra look inflates the false-positive rate well above 5%; use sequential testing or always-valid p-values that spend the error budget across looks
- C) The only real danger is that stopping early reduces the sample size; peeking itself does not affect the false-positive rate, so a bigger planned n fixes it
- D) Simply switch to a one-sided test instead of a two-sided one, which halves the reported p-value and makes this kind of early stopping perfectly safe
Q6. You need to detect a 0.5% lift instead of a 1% lift, keeping α and power fixed. Roughly how does the required sample size change, and what's the cheapest way to offset it?
- A) It roughly doubles in size, since n scales as 1 divided by the MDE, and the only real fix available is to simply wait twice as long for more traffic
- B) It roughly quadruples since n scales as 1 over MDE-squared; the cheapest offset is CUPED, cutting variance 30-50 percent with pre-period data
- C) It stays exactly the same size regardless — the MDE only ever affects statistical power, not the required sample size, so no offset is needed at all
- D) It roughly halves in size instead, because a smaller effect is somehow easier to detect reliably, so you could actually shorten the test duration
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →