RCT Design

Randomisation, SUTVA, Intent-to-Treat, treatment effect heterogeneity

The DAG module showed you how to find the right adjustment set when you're stuck with data you didn't get to design — draw the graph, block backdoor paths, leave colliders and mediators alone. Recall from Potential Outcomes: there's a design that skips needing an adjustment set at all. Randomization balances every confounder between arms — the ones in your DAG and the ones you never thought to draw — because it doesn't rely on you specifying anything correctly. This module is about building that design well, because "we randomized" is necessary and nowhere near sufficient.

An e-commerce company tests a new checkout flow: half of users see it, half see the old one. After two weeks, conversion is 4.8% in the new flow versus 4.3% in the old — significant at p = 0.02. Do they ship it?

If assignment was truly random — a coin flip, not a business rule dressed up as one — treatment and control are identical in expectation on every dimension: age, income, device, time of day, past behavior, and confounders nobody measured. Pause here: if a fair coin decided each user's arm, what's the chance the groups differ systematically on some variable you didn't think to record? Vanishingly small, and it shrinks as the sample grows — a guarantee no DAG-derived adjustment set can offer, since a DAG only protects against confounders you thought to draw. With randomization, the only systematic difference between arms is the treatment itself.

Getting there takes real decisions before anything is flipped. Unit of randomization: user-level gives one consistent experience with no within-user contamination; session-level gains power but the same user can land in both arms; page-level gains the most power at the highest contamination risk. Sometimes none of those units are isolated enough — when SUTVA forces it (a feature that propagates through a social graph, a marketplace, a city-level rollout), you randomize whole clusters (friend groups, cities) instead of individuals. That buys interference protection at a real, computable power cost: the design effect DEFF≈1+(m−1)×ICC tells you how much, where m is the average cluster size and ICC (intraclass correlation) is how similar outcomes are within the same cluster versus across clusters — at ICC=0.1 and m=100, DEFF≈10.9, meaning you need roughly 11× the individual-level sample size to hold power constant. Stratifying by known covariates (device, geography) before randomizing balances small samples chance alone might leave lopsided; block randomization within each stratum keeps counts even. Sample size isn't a number to guess afterward — run a power analysis first. Power is the probability of detecting a true effect if one exists, at a significance level α — a false-positive tolerance fixed by the experimenter, not something the data hands you. At 80% power and α = 0.05, compute the minimum sample size to detect the smallest lift that matters — the minimum detectable effect (MDE). That minimum scales roughly as 1/MDE²: halving the detectable effect roughly quadruples the users needed — why hunting a 0.1-point lift takes months and a 5-point lift takes days.

Correct randomization still doesn't protect against interference. If users share carts with friends, a treated user describing the new checkout to an untreated one changes that person's behavior even though they're assigned to control. The coin flip was fair, but propagation through the social graph means "control" is no longer the clean baseline promised — the same SUTVA violation that broke naive estimates in Potential Outcomes, breaking a properly randomized one too.

Randomization guarantees balance on assignment, not on what people do with it. Assign half of users to the new checkout and some never load it — a stale cache, an ad blocker, a bounce before render. Reversed: some assigned to the old checkout hit a cached copy of the new one and see it anyway — non-compliance can run either direction. Either way you're compared by the group you were assigned to, not the page you actually saw — that's the Intent-to-Treat estimate (ITT), and it's what you actually observe, since compliance can't be forced. ITT understates the effect on people who used the new checkout, diluted by the assigned-but-unexposed. The clean case is one-sided: nobody in control ever crosses into the new flow, only some of treatment fails to load it (monotonicity — no defiers: nobody does the opposite of their assignment). Concretely: if 80% of treatment actually loaded the new flow and ITT comes out to $6.40 per user, the effect among compliers — the Complier Average Causal Effect (CACE) — is ITT divided by that compliance rate: CACE = $6.40 / 0.80 = $8.00. The $8.00 is what tells you the redesign itself works; the $6.40 is what tells you what shipping it to everyone, non-compliers included, will actually move. If control also has crossover, the same idea generalizes to the Wald estimator: divide by the *difference* in take-up between arms instead of the raw 0.80.

None of this is what "statistically significant" certifies. p = 0.02 says the gap is unlikely under no effect — nothing more. That reading only holds if randomization was valid, the primary metric was pre-specified, the run lasted as planned, and the result is worth acting on. That last condition gets harder the more metrics you check: with 15 secondary metrics tested at α = 0.05, chance alone predicts roughly 15 × 0.05 = 0.75 false positives, so finding two or three "significant" secondaries isn't surprising on its own — a Bonferroni correction (dividing α by the number of tests) is the standard fix, and whether the metric driving your headline result was pre-registered as primary, versus fished out after the fact, changes how much weight its p-value deserves. A p = 0.001 result on a 0.01% lift costing $500K to build isn't a success story — significance means reliably nonzero, not worth building.

Key points

• Run a power analysis before starting — calculate the minimum detectable effect at 80% power and α = 0.05. This tells you the required sample size and experiment duration. Running an underpowered experiment and concluding "no effect" is a false negative that can kill good product ideas. The confidence interval of an underpowered experiment is wide enough to contain the true effect; the null result is not evidence of zero, it is evidence of insufficient sensitivity.
• Trap: peeking at results and extending the experiment when it looks close. Deciding to run longer after seeing "almost significant" inflates Type I error from 5% to well above 30% depending on how many times you peek. Use sequential testing (SPRT or always-valid inference) if you need to monitor results during the experiment. Committing to the analysis plan before looking at the data is the only protection against this form of p-hacking.
• Cluster-level randomisation costs power in a specific, computable way. When SUTVA forces you to randomize whole clusters instead of individuals — a social feature, a marketplace, a city-level rollout — the design effect DEFF≈1+(m−1)×ICC tells you the sample-size inflation: m is average cluster size, ICC is how correlated outcomes are within a cluster versus across clusters. At ICC=0.1 and m=100, DEFF≈10.9 — you need roughly 11× the individual-level sample to hold power constant, which is why cluster designs are used only when interference leaves no alternative.
• Checking many secondary metrics inflates false positives the same way peeking does. At α=0.05, testing 15 secondary metrics yields roughly 15×0.05=0.75 expected false positives from chance alone, so 2-3 "significant" secondaries isn't automatically a real finding. Apply a Bonferroni correction (α divided by the number of tests) to secondaries, and treat the primary metric's p-value as trustworthy only if it was pre-registered before the experiment ran — a metric fished out after seeing the data doesn't get the same interpretation.
• Diagnostic: after the experiment ends, run an AA test — randomly split the control group into two halves and test for a significant difference on your primary metric. The AA test should show no significant difference — but treat that single result the way you'd treat any hypothesis test run at α = 0.05: even with a perfectly unbiased randomization mechanism, one AA test comes up "significant" by chance alone about 5% of the time, so a lone hit is a prompt to investigate, not an automatic diagnosis of bias. Run the check a handful of times, or track it across experiments; only a hit rate well above ~5% means some feature of the assignment mechanism is creating groups that were not exchangeable before treatment began. Fix the randomization before trusting any AB result from the same infrastructure.

Recap

• The RCT is the gold standard because randomisation balances *all* confounders — observed AND unobserved — in expectation: the only systematic difference between arms is the treatment, so it needs no measuring of the right confounders, unlike every observational method.
• Unit of randomisation is a design trade-off: user-level gives each user a consistent experience and low contamination; session-level gives more power but the same user can see both variants; page-level has the most power and the highest contamination risk. When even page-level can't contain interference (a social feature, a marketplace, a city rollout), randomize whole clusters instead — at a computable power cost, DEFF≈1+(m−1)×ICC (m = cluster size, ICC = intraclass correlation): ICC=0.1 with m=100 gives DEFF≈10.9, roughly 11× the individual-level sample size needed.
• Run a power analysis first: at 80% power and α = 0.05, compute the sample size needed to detect the smallest effect that matters to the business — an underpowered "no effect" is a false negative that can kill a good idea, not evidence of zero.
• Interference still bites even after correct randomisation: if treatment propagates through the social graph (a treated friend describes the new checkout to a control user), the "control" group is indirectly treated and the estimate is contaminated.
• Non-compliance dilutes what you observe — ITT vs. CACE: you're compared by assignment, not exposure (compliance can't be forced), so the Intent-to-Treat estimate understates the effect on people who actually used it. Under one-sided non-compliance, CACE = ITT / compliance rate (e.g. $6.40 / 0.80 = $8.00) recovers the complier effect — the ITT figure is what shipping to everyone will actually move, the CACE figure is what the redesign itself is worth.
• Peeking inflates Type I error from 5% to well over 30% depending on how often you look — commit to the analysis plan before seeing data, or use sequential / always-valid testing if you must monitor.
• Run an AA test to validate the randomisation: split the control group in two and test for a difference on the primary metric — but one significant result happens ~5% of the time by chance alone even with unbiased randomisation (it's a hypothesis test at the same α), so a single hit is a prompt to investigate, not proof of bias; only a hit rate well above ~5% across repeated checks means the assignment infrastructure is broken and no AB result from it can be trusted.
• Statistical significance ≠ success: p = 0.001 tells you the estimate is reliably nonzero, not that it's worth acting on — a 0.01% effect costing $500K to ship is a significant failure. It also doesn't survive unchecked multiple testing: 15 secondary metrics at α=0.05 predict ~0.75 false positives by chance alone, so a few "significant" secondaries need a Bonferroni correction (and the primary metric needs to have been pre-registered) before you trust them.

Check your understanding