The Two Silent Killers of A/B Tests: Peeking and SRM
Most A/B testing failures aren't statistical errors — they're procedural ones. Peeking: checking results before the planned end date and declaring a winner. SRM: the traffic split is 52/48 instead of 50/50, making all your metrics untrustworthy. Both are invisible in dashboards. Both corrupt your decision-making. Both are preventable.
A/B tests are the gold standard for validating product decisions. They're also fragile. Two failure modes account for the majority of false positives and invalid experiments: peeking and Sample Ratio Mismatch (SRM). Both produce convincing-looking results that are actually noise.
Failure mode 1: Peeking
Peeking is checking your experiment results before the planned end date, and potentially stopping early if the p-value crosses 0.05.
The statistical problem: p-values are not stable over time. If you run 1,000 A/A tests (identical treatment and control) and check them daily for 14 days, roughly 30% will cross p < 0.05 at some point — even with no real effect. The standard p < 0.05 threshold assumes you test once, at the planned end date.
Each additional peek is an independent hypothesis test. Your false positive rate inflates from 5% to 20–30% depending on how often you check and how long you run the experiment.
Production signal: you ship a "winner" that seemed significant at day 8. By day 21 (if you hadn't stopped early), the effect would have regressed toward zero and been non-significant. The "winner" was noise.
Why everyone peeks anyway: experiment dashboards update in real-time. PMs refresh them daily. Someone sees a metric moving in the right direction and the pressure to ship becomes intense.
Fixes for peeking:
Sequential testing (always-valid inference): methods like mSPRT (Uber), CUPED (Microsoft), or mixture sequential probability ratio tests allow continuous monitoring while maintaining the correct false positive rate. Statsig and Optimizely both offer always-valid p-values by default.
Pre-registration: write down your sample size, primary metric, and end date before the experiment starts. Lock the dashboard for interim results. P-value visible only after the pre-registered duration. This is less sophisticated than sequential testing but surprisingly effective.
Bonferroni correction as a blunt tool: if you peek N times, set your p-value threshold to 0.05/N. This overcorrects but prevents false positives. For 14 daily peeks, set your threshold to 0.05/14 ≈ 0.004.
Failure mode 2: Sample Ratio Mismatch (SRM)
SRM is when the ratio of users assigned to treatment and control doesn't match the intended ratio. You randomise 50/50, but the treatment group ends up with 47% of traffic. The 3% difference seems small. It isn't.
An SRM means something is wrong with your randomisation or traffic routing. And that something has differential effects on treatment and control groups. The users missing from one group have systematic properties — they're from a specific device type, browser version, geographic region, or time window where an error occurred. Your control and treatment groups are no longer comparable populations.
When an SRM exists, all your metric comparisons are invalid. Not noisy — invalid. You cannot trust any metric showing a difference.
Production signal: you find a 3% SRM in post-analysis. You ship the experiment anyway, thinking "3% is small." Six months later, in a meta-analysis of similar experiments, you notice that every experiment with SRM showed inflated lift — false positives at roughly 50% rate.
How to detect SRM:
Chi-squared test on assignment counts: ``` from scipy.stats import chisquare chisquare([n_treatment, n_control], f_exp=[expected_n/2, expected_n/2]) ```
If p-value < 0.01, you have an SRM. Run this check automatically the moment your experiment starts, not after it ends.
Common SRM causes: bot traffic filtered differently in treatment vs control, JavaScript errors preventing logging in one variant, different caching behaviour, geographic load balancing sending disproportionate traffic, or randomisation happening after the first meaningful user action (so some users are bucketed differently than intended).
Bonus failure mode: the novelty effect
A variant that's new generates engagement purely because it's novel, not because it's better. This shows up as a spike in week 1 that regresses toward control in weeks 2–3.
Fix: run experiments for at least 2 full novelty cycles. For products with weekly engagement patterns, that's 2 weeks minimum. For features used infrequently (monthly), 6–8 weeks.
The mandatory check sequence:
1. Before you look at anything: run the SRM check. If p < 0.01, stop analysis. Fix the randomisation and rerun.
2. Next: verify your experiment ran for the full pre-registered duration. If it didn't, and someone wants to stop early, use sequential testing results — not naive p-values.
3. Then: look at your primary metric. It's only trustworthy if the SRM check passed and you ran for the full duration.
Practice this in Experimentation frameworks to understand how to design bulletproof A/B tests and catch these failure modes before they corrupt your decisions.
```python from scipy.stats import chi2_contingency, norm import numpy as np
def pre_analysis_checklist(control_n, treatment_n, target_split=0.5, observed_metric=None, alpha=0.05): """Run this BEFORE looking at your primary metric. Always.""" results = {}
# 1. SRM check — chi-squared on traffic split total = control_n + treatment_n expected_c = total * target_split expected_t = total * (1 - target_split) chi2, srm_p, *_ = chi2_contingency([[control_n, treatment_n], [expected_c, expected_t]]) results['srm_p'] = round(srm_p, 4) results['srm_pass'] = srm_p >= 0.01 # fail → stop, do not analyse primary metric
# 2. Minimum detectable effect check if observed_metric is not None: se = np.sqrt(observed_metric * (1 - observed_metric) * (1/control_n + 1/treatment_n)) mde = norm.ppf(1 - alpha/2) * se * 2 results['mde_pct'] = round(mde * 100, 2)
return results
# Never peek at conversion rate before running this first ```