Sampling Distributions & CLT
CLT, standard error, confidence intervals, bootstrap
You run an A/B test. Treatment group (n=500) has CTR 4.3%. Control (n=500) has CTR 3.8%. The difference is 0.5 percentage points. Is this a real effect or just sampling noise? To answer, you need to know: if the true CTRs were equal, how variable would a 0.5% difference be purely from random sampling? The sampling distribution — the distribution of a statistic's value across many hypothetical repeats of the same sampling process — of (CTR_treatment - CTR_control) under the null hypothesis answers this exactly.
The sample mean X̄ of n i.i.d. draws from a population with mean μ and variance σ² has: E[X̄] = μ and Var[X̄] = σ²/n. Standard error = σ/√n. By the Central Limit Theorem, for large n, X̄ ≈ N(μ, σ²/n) regardless of the shape of the original distribution. This is why t-tests and z-tests work asymptotically for any distribution — they operate on means, and means become approximately normal.
The t-distribution: when σ is unknown and estimated from data as the sample standard deviation s, the statistic (X̄ - μ)/(s/√n) follows a t-distribution with n-1 degrees of freedom. The t-distribution has heavier tails than N(0,1) for small n, reflecting the extra uncertainty introduced by estimating σ. At n=30 or more, t(n-1) is nearly indistinguishable from N(0,1) — a separate convergence fact from (but numerically close to) the informal n≥30 rule of thumb often cited for the CLT itself; the two thresholds are not the same claim and neither derives the other.
Bootstrap sampling distribution: the empirical alternative to analytical formulas. Draw n samples with replacement from your data, compute the statistic, repeat 10,000 times. The distribution of the statistic across bootstrap samples is the sampling distribution. Works for any statistic — AUC, precision@K, NDCG — with no formula required.
NOT this. The CLT applies to any distribution for large n is not unconditionally true. The CLT requires finite mean and finite variance. For heavy-tailed distributions — Pareto with tail index less than 2, some financial return distributions — the variance does not exist and the CLT does not apply. The sample mean does not converge to a Gaussian; it converges to a stable distribution with heavier tails. For web latency, transaction sizes, and other power-law distributed data, checking whether the CLT applies before running a t-test is not paranoia — it is necessary.
Back to the A/B test: pooled CTR under the null is (0.043·500 + 0.038·500)/1000 = 0.0405, so SE of the difference = √(0.0405·0.9595·(1/500+1/500)) ≈ 0.0125, or 1.25 percentage points. The observed 0.5-point gap is about 0.4 standard errors from zero (z ≈ 0.40, two-sided p ≈ 0.69) — nowhere near the ~2 SE needed for significance, so this particular 0.5pp gap is comfortably explained by sampling noise alone, not a real effect.
Key points
- Always report the standard error (σ/√n) alongside any point estimate. An estimate without its standard error is not a scientific claim — it is a number without an indication of how much it would vary across repeated samples. For differences between groups, the standard error of the difference is √(σ₁²/n₁ + σ₂²/n₂), assuming independence between groups.
- Trap: using a z-test when n < 30 and the distribution is non-normal. The z-test uses critical value 1.96, which comes from N(0,1). For small n with unknown σ, the correct critical value comes from the t-distribution: t_{0.025, n-1}. For n=10, that critical value is 2.262 instead of 1.96 — using z inflates Type I error because the interval is too narrow.
- Diagnostic: if your A/B test p-value looks suspiciously small (< 0.001) or large (> 0.5), recheck the standard error computation. Common errors: not accounting for within-user correlation across multiple observations (inflates effective sample size), using population σ instead of sample s, or forgetting that the standard error of a difference requires variance from both groups, not just one.
The sampling distribution tells you how much a statistic varies across repeated samples. The CLT makes sample means approximately normal for large n — but large depends on tail behavior. Always verify the CLT assumption holds before running t-tests or z-tests on data with heavy tails.
Recap
- Sampling distribution answers "is this real or noise?" — how a statistic varies across repeated samples.
- Sample mean: E[X̄]=μ, Var[X̄]=σ²/n, standard error = σ/√n.
- CLT: for large n, X̄ ≈ N(μ, σ²/n) regardless of the original shape — why t/z-tests work on means.
- t-distribution (σ estimated by s) has heavier tails for small n; at n ≥ 30 it's ≈ N(0,1).
- Bootstrap = resample with replacement, recompute, repeat 10k× — sampling distribution for any statistic, no formula.
- CLT needs finite mean AND variance — fails for heavy tails (Pareto tail index < 2); check before a t-test.
- SE ∝ 1/√n: doubling n only shrinks it by √2 — precision is expensive; for a difference use √(σ₁²/n₁ + σ₂²/n₂).
Check your understanding
Q1. X₁,...,Xₙ ~ N(μ,σ²). Which two of the following statements about the sampling distribution of S² are correct?
- A) (n−1)S²/σ² follows a χ²(n−1) distribution rather than χ²(n) — one degree of freedom is lost because the deviations are computed from the sample mean X̄, not the true mean μ, and the deviations (Xᵢ−X̄) must sum to zero, confining them to an (n−1)-dimensional subspace.
- B) Dividing by n−1 rather than n is what makes S² an unbiased estimator of σ²: with the n−1 correction E[S²] = σ², whereas dividing by n instead would give the biased E[S²] = σ²(n−1)/n.
- C) (n−1)S²/σ² is claimed to follow N(0,1) by the CLT, on the reasoning that S² is a sample average of squared deviations and averages of i.i.d. quantities become approximately normal for large n, regardless of the actual distributional family involved.
- D) (n−1)S²/σ² is claimed to follow an F(n−1,n) distribution, on the reasoning that it is a ratio of two chi-squared variables — χ²(n−1) in the numerator divided by an implicit χ²(n) coming from the denominator σ² itself.
Q2. If X̄ ~ N(μ, σ²/n), what does the Central Limit Theorem say about non-Gaussian X, and when does it break down?
- A) CLT states that X̄ ~ N(μ, σ²/n) exactly for all distributions with finite variance, regardless of how large or small n is. The distribution of X̄ is claimed to be always exactly Gaussian whenever X has finite variance — supposedly following from the characteristic function of a sum of i.i.d. variables being the product of individual characteristic functions, which is then claimed to equal the Gaussian characteristic function by the additive property of cumulants.
- B) CLT: if X₁,...,Xₙ are i.i.d. with mean μ and finite variance σ², then √n(X̄ − μ)/σ →_d N(0,1) as n→∞. Practical rule of thumb: CLT works well for n≥30 for most distributions; for highly skewed or heavy-tailed distributions, it may need n≥100 or more. The CLT is claimed to never break down as long as the sample size is large enough — for any distribution with finite variance, there is supposedly always some n beyond which the normal approximation holds, with the required n depending only on the skewness and kurtosis of X.
- C) CLT: √n(X̄ − μ)/σ →_d N(0,1) as n→∞, regardless of the shape of P(X) — the standardised sample mean approaches a standard normal no matter the shape of the original distribution. Rule of thumb: n≥30 works for most distributions, heavy-tailed ones may need n≥100. Breakdown conditions: (1) infinite-variance distributions (Cauchy, Pareto with tail index α<2) — the sample mean itself has infinite variance, so Generalized CLT (stable distributions) applies instead; (2) non-i.i.d. autocorrelated data needs a Functional CLT; (3) extreme imbalance (binary data with p near 0) needs n·p·(1−p) large enough.
- D) CLT: X̄ converges to a Student-t distribution with n−1 degrees of freedom for non-Gaussian X. As n→∞, this t-distribution converges to N(0,1). For non-Gaussian X, the sample mean is claimed to have heavier tails than predicted by the normal approximation, which the t-distribution is said to correctly capture. Breakdown: for n>1000, the t and normal distributions become essentially identical, and the CLT approximation is claimed valid for any distribution with finite variance.
Q3. The t-distribution has heavier tails than the normal. Why does this matter when computing confidence intervals with small samples?
- A) With known σ: X̄ ± 1.96σ/√n. With unknown σ: X̄ ± tσ/√n. The t-distribution correction multiplies the CI width by the ratio t_{0.025,n−1}/1.96. For n=5, that correction factor is 2.776/1.96 ≈ 1.42 — about 42% wider than the naive normal-based interval. For n=30, the correction factor shrinks to roughly 1.04, nearly negligible. The practical consequence is that small-sample confidence intervals require using proper t-tables rather than the fixed z=1.96, otherwise the stated 95% coverage ends up being noticeably lower than advertised.
- B) With known σ: (X̄ − μ)/(σ/√n) ~ N(0,1), so the 95% CI is X̄ ± 1.96σ/√n. With unknown σ, we substitute S (sample std dev): (X̄ − μ)/(S/√n) ~ t(n−1). The t-distribution has heavier tails because S is only an estimate of σ, adding extra uncertainty; the critical value t_{0.025,n−1} > 1.96 for all finite n — e.g. 2.776 at n=5, 2.262 at n=10, 2.045 at n=30, converging to 1.96 as n→∞. Using z=1.96 with small samples gives an interval that is too narrow, with true coverage below 95%.
- C) The t-distribution has heavier tails because small samples produce more variable estimates — not, in this account, due to estimating σ at all, but because the sampling distribution of X̄ itself is claimed to have heavier tails whenever n is small. This is presented as an exact property of normal populations: for any n, the ratio (X̄−μ)/(σ/√n) is exactly N(0,1), while (X̄−μ)/(S/√n) is exactly t(n−1). The difference is said to disappear only once n>30, because S is claimed to converge to σ and the two ratios become identical.
- D) The t-distribution correction is claimed to be necessary only for σ² estimation, not for μ estimation at all. When computing a confidence interval for σ² (the variance itself), you use chi-squared critical values rather than t-values. Under this view, the t-distribution only ever applies when you are estimating both μ and σ² simultaneously — for μ alone with a known σ, you should always use z=1.96 regardless of how small the sample size n happens to be.
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →