Hypothesis Testing
p-values, Type I/II errors, t-test, chi-squared, multiple comparisons
Your team builds a new checkout button — different color, slightly repositioned. You run it for two weeks. 10,000 users see the new version; 10,000 see the old one. Conversion: 3.2% versus 3.0%. The difference is 0.2 percentage points. Is this real, or did you just get lucky?
The question you need to answer: if the button had no effect whatsoever — if both groups were drawn from the same underlying population — how often would random sampling produce a gap of at least 0.2 percentage points just by chance? That probability is the p-value. Work it out for this test: pooled conversion rate ≈3.1%, standard error of the gap ≈0.245 percentage points, so the observed 0.2-point gap is under one standard error from zero — the p-value comes out to about 0.41. A p-value of 0.41 means: if the null hypothesis were true, you would see a difference this large or larger about 41% of the time by chance alone — nowhere near rare enough to reject the null. This particular gap is comfortably explained by noise, not a real effect.
If p falls below your threshold α (commonly 0.05), you reject the null. Type I error (false positive): you rejected when the null was true — happens with probability α. Type II error (false negative): you failed to reject when there was a real effect. The probability of *detecting* a real effect is power $= 1 - β$. Power is not determined by p-values — it is determined before the experiment by choosing your sample size.
Now suppose your product manager runs 20 A/B tests simultaneously, each at α = 0.05. Under the null, each test has a 5% chance of a false positive. With 20 tests, you expect $20 \times 0.05 = 1$ false positive. Finding three "significant" results is entirely consistent with all null hypotheses being true. The Bonferroni correction divides α by the number of tests: test each at $α/20 = 0.0025$. The Benjamini-Hochberg procedure controls the false discovery rate — the fraction of significant results that are false — and is less conservative.
NOT this. Most people read p < 0.05 as "there is a 95% probability the effect is real." Wrong. The p-value is a property of the *data* under the null hypothesis, not a probability about the hypothesis. It says nothing about P(null is true). To compute that, you need a prior — Bayesian territory. A p-value of 0.001 on a 10-million-user-per-arm dataset can come from a conversion lift as small as ≈0.025 percentage points — about a 1% relative lift on a 3% baseline — small enough that many teams would consider it commercially marginal. Statistical significance is not practical significance. Always pair p-values with effect sizes and confidence intervals.
Key points
- Use hypothesis testing when you need to decide whether an observed difference exceeds what chance alone can explain. Pre-commit to your α and the minimum detectable effect size before collecting data. The sample size formula $n = 2(z_{α/2} + z_β)^2 σ^2 / δ^2$ requires specifying $δ$ (the smallest effect you care about) and $β$ (acceptable miss rate) before a single observation. Post-hoc power calculations — done after seeing the results — are not valid.
- The production trap: peeking. Checking p-values as data accumulates and stopping when p < 0.05 inflates Type I error far above α. If you check after every 100 users, the effective false positive rate can reach 20–30% even at a nominal α = 0.05. Use sequential testing methods (always-valid p-values or alpha-spending functions) if you need continuous monitoring, or commit to a fixed sample size and look once.
- The diagnostic: separate statistical significance from practical significance. With n = 10,000,000 per arm, a lift of only ≈0.025 percentage points (about 1% relative, on a 3% baseline) achieves p < 0.001 — statistically significant, but easy to overstate as a business win. Report Cohen's d, percent lift, or absolute conversion change alongside every p-value. If the effect size is smaller than the minimum you pre-specified as meaningful, the result does not justify a ship decision regardless of the p-value.
A p-value measures how often chance produces this result, not how probable the hypothesis is. Effect size tells you whether the result matters. You need both before making a decision.
Recap
- p-value = P(data this extreme | null true), a property of the data — NOT P(hypothesis is true).
- Type I (α) = false positive when null true; Type II (β) = miss. Power = 1−β, fixed by sample size beforehand.
- Multiple comparisons: 20 tests at α=0.05 → expect 1 false positive; finding 3 "significant" is consistent with all nulls.
- Corrections: Bonferroni tests each at α/m (conservative, FWER); Benjamini-Hochberg controls FDR (less conservative).
- Statistical ≠ practical significance: at n=10M/arm, a ≈0.025-point (≈1% relative) lift is p<0.001 but easy to overstate as a business win.
- Peeking inflates Type I error to 20–30% — use sequential/always-valid methods or fix n and look once.
- Always pair the p-value with effect size (Cohen's d, % lift) and a confidence interval before deciding.
Check your understanding
Q1. You run a t-test comparing two groups and get p=0.048. Your colleague says 'we have 95.2% confidence that the effect is real.' Which two of the following identify genuine, correct problems with this statement?
- A) The statement is almost correct but uses the wrong confidence level. p=0.048 means 1−p = 95.2% is the confidence that the effect is real. The technically precise statement would be '95% confidence' (using the standard threshold), not '95.2% confidence' — the colleague is incorrectly using the exact complement of the p-value rather than rounding to the nearest standard confidence level.
- B) The colleague's statement is correct for a one-sided test but wrong for the two-sided t-test that was run, since two-sided tests split the rejection region across both tails. For a two-sided test, p=0.048 means 97.6% confidence on each side, giving 95.2% total under this (mistaken) framing. The colleague should have instead said '97.6% confidence that the effect is in the observed direction,' though even that framing is not statistically rigorous.
- C) Separately, the word 'confidence' here borrows language from confidence intervals and misapplies it to a p-value: a 95% confidence interval's 'confidence' describes the long-run coverage rate of the interval-construction procedure across repeated samples, not a probability statement about this one result. Computing '95.2%' as 1−p and calling it 'confidence' conflates two distinct statistical objects — a p-value and a confidence level — that happen to share a number here but not a meaning. A cleaner statement would simply be 'we reject the null at α=0.05,' with no confidence language attached at all.
- D) The p-value is NOT the probability that the null is false, nor 1−P(null is true). p=0.048 means: IF the null were true, there is a 4.8% chance of a test statistic this extreme or more, purely by chance. It says nothing about P(H₀ is false) — that needs a prior (Bayesian framework). Interpretation: 'if truly no effect, data this extreme occurs ~4.8% of the time' — confusing frequentist and Bayesian probability is the real error here.
Q2. A clinical trial detects p=0.001 with a treatment effect of 0.2 points on a 100-point quality-of-life scale. N=50,000. Is this finding clinically meaningful? Explain statistical vs practical significance.
- A) Statistical significance (p=0.001) and clinical significance are different. With N=50,000 the standard error is tiny, so even trivial effects produce very small p-values. A 0.2-point improvement on a 100-point scale is 0.2% — far below any clinical threshold (typically 5-10 points). The p-value answers 'is the effect nonzero?' — yes. Clinical significance asks 'is it large enough to matter?' — no. Report effect size and confidence intervals alongside every p-value, not the p-value alone.
- B) Yes, p=0.001 is clinically meaningful because it passes the stringent 0.001 threshold, far below the typical 0.05 cutoff used in most fields. A result significant at p=0.001 survived a much higher evidentiary bar than p=0.05, providing roughly three times more statistical evidence of a real, nonzero effect. The 0.2-point effect size should still be reported but does not undermine the significance.
- C) The finding has borderline clinical significance. A 0.2-point improvement is small but not negligible — over N=50,000 patients, the aggregate benefit across the whole population is 0.2 × 50,000 = 10,000 patient-points of improvement, which sounds substantial. Population-level impact metrics like this one should supplement individual per-patient effect sizes when evaluating clinical trials of this scale.
- D) The finding is not statistically significant. p=0.001 means there is only a 0.1% chance the result is real — not strong enough for clinical applications, which require p < 0.0001 to account for multiple comparison corrections across the many endpoints typically measured in a single trial. The 0.2-point effect size is itself consistent with a trial that is slightly underpowered for this scale.
Q3. You run 20 A/B tests simultaneously. 3 tests show p < 0.05. How many would you expect by chance, and what correction do you apply?
- A) Under the null (all 20 effects zero), each test has P(p<0.05)=0.05, so expected false positives = 20×0.05=1 — finding 3 'significant' tests is plausible even if all nulls are true. Bonferroni: threshold α/m=0.05/20=0.0025 per test (conservative FWER control). Benjamini-Hochberg: ranks p-values, rejects H_k if p_k≤k·α/m — controls the expected false-positive proportion, the standard choice in A/B testing.
- B) Expected false positives = 20 × 0.05 = 1 under the null. Finding 3 significant tests suggests all 3 are likely real — the expected count under the null is 1, and the Poisson probability of observing 3 or more by chance when λ=1 is P(X≥3) ≈ 0.08, small enough to conclude at least some tests are detecting genuinely real effects. No multiple-comparisons correction is needed when fewer than 5% of the tests come back significant.
- C) Expected false positives = 20 × 0.05² = 0.05. Under the null, the probability of any false positive across all 20 tests is 1−(1−0.05)²⁰ ≈ 0.64, meaning there is a 64% chance at least one false positive occurs somewhere in the batch. Finding 3 significant results is far more than the 0.05 expected, suggesting all 3 are likely true positives. Bonferroni is only needed once more than 10% of tests are significant.
- D) Expected false positives = 3 × 0.05 = 0.15 — only the 3 significant tests contribute to the false-positive expectation, since the other 17 non-significant tests cannot themselves produce false positives. Bonferroni correction divides the significance threshold by the number of significant tests: 0.05/3 ≈ 0.017. All 3 tests show p < 0.05 but not p < 0.017, so after applying this correction none remain significant.
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →