The Two Failure Modes of A/B Tests (And How to Catch Them)

Most A/B testing mistakes aren't statistical errors — they're procedural ones. The two most common: peeking at results before the test ends (inflates your false positive rate by 2–5×), and failing to detect a sample ratio mismatch (makes all your metrics untrustworthy). Both are invisible unless you know where to look.

A/B testing looks simple: split traffic, measure a metric, compare. It's not simple. The statistical machinery underneath has failure modes that are invisible to the untrained eye, and that silently corrupt your product decisions.

The two failure modes that cause the most damage in practice are peeking and sample ratio mismatch. Here's what they are, why they're dangerous, and how to catch them.

Failure Mode 1: Peeking

Peeking is looking at your experiment results before the planned end date, and making a decision based on what you see.

The problem: p-values are not stable over time. If you run 1000 A/A tests (same experience for both groups) and check them each day until one hits p < 0.05, roughly 30% of tests will cross that threshold at some point during the run — even though there is no real effect. The standard p < 0.05 threshold assumes you check once, at the end.

Peeking inflates your false positive rate (calling a winner when there isn't one) from 5% to 20–30%, depending on how often you check. This means a large fraction of the "winners" you ship are actually noise.

Why everyone peeks anyway: experiment dashboards are updated in real-time. PMs and executives watch them. Someone sees a metric moving in the right direction and wants to ship. The pressure to stop early is enormous.

Fixes for peeking:

Sequential testing / always-valid inference: methods that allow continuous monitoring while maintaining the correct false positive rate. Implementations include CUPED (Microsoft), mSPRT (Uber), and mixture sequential probability ratio tests. Statsig and Optimizely both offer always-valid p-values by default.

Pre-registration: write down your sample size, primary metric, and end date before the experiment starts. Don't change them. An experiment log with these parameters committed before launch is sufficient.

Failure Mode 2: Sample Ratio Mismatch (SRM)

SRM is when the ratio of users assigned to treatment and control doesn't match the intended ratio. You randomise 50/50, but the treatment group ends up with 47% of users. The 3% difference seems small. It isn't.

When an SRM occurs, it means something is wrong with your randomisation or traffic routing — and that something has differential selection effects on your treatment and control groups. The users who are "missing" from one group aren't random; they have some systematic property (device type, browser version, geographic region, load time). Your control group and treatment group are no longer comparable populations.

This means all your metric comparisons are invalid. Not noisy — invalid. You cannot trust any metric that shows a difference, positive or negative, when an SRM exists.

How to detect SRM:

Run a chi-squared test on your assignment counts: expected ratio (50%) vs. actual ratio. If the p-value is below 0.01, you have an SRM. This test costs nothing and should run automatically on every experiment the moment it starts.

Chi-squared: `from scipy.stats import chisquare; chisquare([n_treatment, n_control], f_exp=[expected_n/2, expected_n/2])`

Common SRM causes: bot traffic filtered differently in treatment vs. control, JavaScript errors in the treatment that prevent logging, different caching behavior, geographic load balancing sending disproportionate traffic, or A/B assignment happening after the first meaningful user action.

Bonus failure mode: The novelty effect

Users in the treatment group interact differently with a new experience simply because it's new — not because it's better. This shows up as a spike in the treatment metric in week 1 that regresses toward control in weeks 2–3.

Fix: run experiments for at least 2 full novelty cycles. For consumer products with weekly engagement patterns, that's typically 2 weeks minimum. For features with low interaction frequency (monthly), longer.

The SRM check is the first check, always

Before you look at your treatment effect: run the SRM check. If there's an SRM, stop analysis. Fix the randomisation. Re-run the experiment. Reporting results from an experiment with known SRM is worse than no experiment at all.