A/B Infrastructure

Traffic splitting, treatment assignment, exposure logging, interaction effects

A team runs an A/B test the simple way: even user IDs go to control, odd IDs to treatment. Three weeks later they realize their hashing put the *same* 15% of power users into treatment across all six experiments running at once. Those users saw six new experiences stacked on top of each other, and their behavior is now tangled up in all six at the same time. That's experiment interference, and it doesn't just complicate the reading — it structurally breaks all six results. The groups aren't comparable, the p-values aren't valid, and every decision made from them rests on corrupted evidence.

Why an A/B bug is scarier than a model bug: it hides behind real-looking statistics

A biased assignment still produces a p-value, a confidence interval, a tidy result summary — and every number in that chain is wrong. Nothing crashes, the experiment completes, and a decision gets made on bad data. That's why getting the infrastructure right isn't an implementation detail; it's the precondition for every statistical claim downstream.

The pieces that make assignment trustworthy

*Deterministic assignment* — hash(user_id + experiment_id) so a user always lands in the same bucket, making their events attributable. *Orthogonal splitting* — a different salt per experiment so simultaneous tests assign users independently and don't correlate. *An experiment registry* — tracks what's live and blocks conflicting experiments from overlapping. *Traffic ramp-up* — 1% → 5% → 20% → 50%, to catch bugs before full exposure. *Guardrail metrics* — automatic regression alerts that fire before anyone reads the primary metric.

One more piece, specific to testing ML model changes: latency and versioning. A new model is a new deployable, not just a new arm — before it ever sees production traffic, confirm it meets the latency SLA (a slower model changes the user experience by itself, independent of what it predicts), then run it through the same canary ramp (1–5% traffic) as any other change. Two details matter more here than in a generic A/B test: assignment must happen *before* the model runs, so a timeout or error in the challenger can't leak into which arm a request gets logged under; and every prediction must be logged with the exact model_version that produced it, or you can't tell which model's output actually rendered.

The single most important check: Sample Ratio Mismatch

You intended 50/50 and observed 52/48 — each arm is 2 percentage points off its target. That is *not* noise — it's the fingerprint of a systematic bug in assignment or logging. SRM is measured with a χ² test on the raw split, not by eyeballing the percentages: run it *before* you open any metric dashboard, and treat any split the test flags as statistically significant — a split like 52/48 fails it decisively at any real sample size — as disqualifying. Once SRM is flagged, every downstream comparison is invalid no matter how significant it looks; there is no valid analysis to do until the root cause (hashing, bot filtering, logging) is found and fixed.

A second, different kind of interaction effect: network effects between arms, not within them. Experiment interference (above) is the same user landing in multiple experiments. A SUTVA violation is different — SUTVA (the Stable Unit Treatment Value Assumption: one unit's outcome shouldn't depend on which treatment other units received) is broken here, because treatment leaks across arms through the system itself. In a marketplace test where treated sellers get a new dashboard, control buyers still interact with those treated sellers, so control buyer behavior shifts even though no control user was ever treated. Control and treatment are no longer independent populations, and a metric computed as if they were misstates the true effect. Detect it by watching guardrail metrics in the arm that received no treatment; mitigate it with geo- or marketplace-cluster randomization, so a whole region or seller cluster sits in one arm instead of mixing treated and control counterparties in the same market.

And for the long view: holdout groups. Permanently hold 5–10% of traffic out of *all* experiments and compare production against it over time. This catches novelty effects — the wins that look great in a two-week test but fade once the shine wears off — and shows the true cumulative impact of your ML work. All of this is why "A/B testing is just feature flags and if/else" misses the point: the flag is the mechanism; everything above is the safety system that makes the mechanism produce answers you can trust.

Key points

• Implement an experiment registry before running more than 2 simultaneous experiments — without it, experiment interference silently invalidates results and you make product decisions on corrupted data. hash(user_id + experiment_id) % 100 gives orthogonal assignments because experiment_id itself is the per-experiment salt — each experiment's distinct id string sends the same user to an effectively independent bucket in every experiment, with no separate salt value needed on top of it. Without the registry tracking what is running, two experiments may still accidentally share users in ways that correlate treatment assignments, confound their effects, and produce results that look significant but measure the interaction, not the treatment.
• Trap: running experiments past their planned duration on a fixed sample size is a form of p-hacking, regardless of your statistical reasoning for the extension. Once an experiment runs past its pre-specified duration, extending it is outcome-dependent stopping. You looked at the data, saw it was close to significance, and extended. That is exactly what p-hacking looks like from the outside. Plan sample size before starting — and plan for a minimum duration of 7-14 days regardless: shorter windows let day-of-week effects (weekday vs. weekend behavior swings a metric on its own) and novelty effects masquerade as a treatment effect. If you need to extend past the plan, use sequential testing methods like mSPRT (mixture Sequential Probability Ratio Test) -- built to let you check significance continuously without inflating the false-positive rate the way naive repeated peeking does -- to account for the additional looks.
• Diagnostic: audit your last 10 A/B test results — if more than 30% were positive, your testing framework likely has inflated Type I error. Under pure noise at α=0.05 with a single primary metric and no peeking, you expect 5% false positives. If your win rate is 30%, you are either measuring real effects (unlikely across all features) or you have peeking, multiple comparisons, or SRM issues inflating the Type I error. The win rate is the fastest diagnostic for whether your experimentation infrastructure is measuring reality.

Recap

• Experiment interference: same power users land in treatment across six concurrent tests → all six results structurally broken.
• Network interference (a SUTVA violation): a different problem from experiment interference — treatment leaks across arms through the system itself (e.g. treated sellers change what control buyers see); fix with geo/cluster randomization.
• An A/B bug is scarier than a model bug: it hides behind real-looking p-values and confidence intervals.
• Trustworthy assignment: deterministic `hash(user_id + experiment_id)`, orthogonal splitting (per-experiment salt), registry, traffic ramp 1→5→20→50%, guardrail metrics.
• Testing ML model changes specifically: confirm latency SLA compliance and run a canary before launch, assign traffic *before* the model runs, and log `model_version` with every prediction.
• Most important check = Sample Ratio Mismatch: intended 50/50, observed 52/48 is a bug fingerprint, not noise.
• Run a χ² test on the raw split *before* opening any metric dashboard. SRM is a χ²-test result, not a fixed percentage — treat any flagged split (like 52/48) as disqualifying.
• Extending a fixed-N experiment past its planned duration = p-hacking — use sequential methods (mSPRT) if you must look again.
• Permanent 5–10% holdout across all experiments catches novelty effects and shows true cumulative ML impact.

Check your understanding