Multiple Testing, FDR, and Power Analysis: The Stats Every DS Gets Wrong

If you run 20 A/B tests and one comes back significant at p < 0.05, the expected number of false positives under the null is 1. You found nothing. Multiple testing correction is the discipline of not fooling yourself when running many tests. FDR control is the right tool when you want to discover true effects, not just avoid false ones. And power analysis is what you do before the test, not after.

Multiple testing and power analysis are the two most commonly skipped steps in experimental DS work — and the source of more false conclusions than any other statistical mistake.

The multiple testing problem

Every hypothesis test has a Type I error rate α = 0.05: you will reject a true null hypothesis 5% of the time by chance. If you run 20 independent tests, the probability of at least one false positive is 1 - (1-0.05)^20 = 0.64. Running many tests and reporting the significant ones is p-hacking, whether or not it is intentional.

In practice, multiple testing arises constantly: testing one feature on 20 different user segments, testing 5 metrics for one experiment (click rate, conversion, revenue, session length, return visit), running 10 variants of an email subject line simultaneously. Each additional test increases your false positive rate.

Bonferroni correction: too conservative

Bonferroni divides the significance threshold by the number of tests: α_adjusted = α / m. For m=20 tests, each must have p < 0.0025 to be significant. This controls the Family-Wise Error Rate (FWER) — the probability of even one false positive across all tests. Bonferroni is appropriate when even one false positive is catastrophic (clinical drug trials). In industry settings where you are looking for effects worth investigating further, it is excessively conservative and kills statistical power.

Benjamini-Hochberg: FDR control for exploration

False Discovery Rate (FDR) = expected proportion of significant results that are false positives. Benjamini-Hochberg (BH, 1995) procedure controls FDR at level q: sort p-values p_(1) ≤ p_(2) ≤ ... ≤ p_(m). Find the largest k such that p_(k) ≤ k*q/m. Reject all tests up to k. At FDR q=0.1, you expect 10% of your declared discoveries to be false positives.

BH is the right tool when: you are running many tests, false positives are costly but not catastrophic, and you want to prioritise findings for follow-up investigation. In feature importance analysis, variant selection, or gene expression studies, BH is the standard.

Power analysis: determining sample size before the experiment

Statistical power = P(correctly rejecting H0 when H1 is true) = 1 - β. Standard practice: power = 0.80 (80% chance of detecting a true effect). The required sample size depends on: effect size (MDE — minimum detectable effect), significance level α, power 1-β, and the metric's variance.

For a two-sample proportion test: n = 2 * (z_{α/2} + z_β)^2 * p̄(1-p̄) / δ^2, where p̄ is the average conversion rate, δ is the MDE, z_{α/2} = 1.96 (for α=0.05), z_β = 0.84 (for power=0.8). For a baseline conversion rate of 5% and MDE of 1 percentage point (detecting a 5% → 6% lift), this requires ~5,000 users per arm.

Common mistakes in power analysis

Underestimating the variance: metrics with high variance (revenue per user, session length) require much larger samples than binary metrics (clicked/not). Ignoring multiple metrics: if you are testing 5 metrics, each at α=0.05 and power=0.8, your effective power for detecting any one specific metric may be much higher — but you need to correct for the multiplicity.

Running tests until significant (optional stopping): repeatedly peeking at results and stopping when p < 0.05 inflates the Type I error rate. Bayesian sequential testing or SPRT (Sequential Probability Ratio Test) provide valid stopping rules. Many experimentation platforms now implement these.

Peeking and early stopping in industry

The standard error at peak sample size assumes the test ran to completion. Looking at results midway through and stopping early (if significant) or late (if trending) changes the effective α. Proper sequential testing designs account for this: O'Brien-Fleming spending functions, Pocock corrections, or Bayesian adaptive designs allow early stopping with controlled error rates.

Try on Colab: simulate running 100 A/B tests where H0 is true (no effect) for all of them. At α=0.05, count false positives. Apply Bonferroni correction — count false positives. Apply BH at q=0.1 — count false positives. Now repeat where 30 of 100 tests have a true effect. Compare the power (true positives recovered) of Bonferroni vs BH. The power difference will be stark — BH recovers significantly more true effects at similar FDR.