Potential Outcomes Framework
Rubin causal model, ATE/ATT, fundamental problem of causal inference
User 47 got the discount email and spent $100 over the next month. Would they have spent that anyway, without it? You'd need a second version of User 47 who never got the email — and it doesn't exist; you only watch what actually happened. Same gap, reversed, for User 12: no email, spent $70. You'd need the version of them who did get it, and you don't have that version either. Whichever branch of the world happened for a person is the only branch you ever see.
That missing branch has a name — the counterfactual, the outcome a unit would have had under the treatment it didn't receive. And the fact that it's missing for every unit, always, is the fundamental problem of causal inference: you observe exactly one of two potential outcomes per unit, never both.
Pause here: if the other branch never exists for anyone, how could any experiment ever produce a causal number at all? Hold that question — it resolves once you stop trying to estimate any one person's effect.
Rubin's potential outcomes framework gives this gap symbols: Y_i(1) is unit i's outcome if treated, Y_i(0) if untreated. For User 47, Y_47(1) = $100 is the branch actually watched — whatever assigned the treatment (here, a random draw into the campaign) decided which branch got realized; Y_47(0) is the missing counterfactual. For User 12 it's reversed: Y_12(0) = $70 is observed, Y_12(1) is missing. The individual treatment effect, ITE_i = Y_i(1) − Y_i(0), needs both numbers for one person — exactly what the fundamental problem rules out. No dataset, however large, ever gives you one person's ITE.
So estimate the average instead: the Average Treatment Effect, ATE = E[Y_i(1) − Y_i(0)]. Here's why that rescues you. Suppose — a God's-eye view no real dataset gives you, for one paragraph only — you could see both branches for four users: User 47, $100 treated / $80 untreated, effect $20. User 12, $90 / $70, effect $20. User 8, $60 / $50, effect $10. User 90, $40 / $30, effect $10. True ATE = (20+20+10+10)/4 = $15 — computable only because you were God for a paragraph.
In the real campaign only one column per user survives. Say the actual draw put User 47 and User 8 in treatment, User 12 and User 90 in control. All you can compute is the observed difference in means: mean($100, $60) − mean($70, $30) = $80 − $50 = $30 — double the true $15. Randomization didn't fail; one draw carries sampling noise like any estimator does. The guarantee is about averaging this estimator over every possible draw: with 4 users split 2-and-2 there are C(4,2) = 6 possible draws, and the same difference-in-means computed under each gives $55, $30, $10, $20, $0, and −$25 — averaging to 90/6 = $15, exactly the true ATE. One draw is noisy; the estimator is unbiased across draws. That's the entire justification for flipping the coin: not that one experiment nails the number, but that the design makes the noise average out to zero around the truth.
That guarantee needs three assumptions, each of which can break in this campaign. SUTVA (Stable Unit Treatment Value Assumption): treating User 47 doesn't change User 12's outcome. Break it — say User 47 forwards the discount code to User 12 — and User 12's "untreated" $70 is no longer the baseline, it's contaminated by spillover. Consistency: the outcome observed for a treated unit really is Y_i(1) as defined — a different subject line is a different treatment, not noise around one. Positivity (overlap): every user has some chance of landing in either arm — if users who spent nothing last month are never in the campaign at all, any number reported for them is pure extrapolation, not estimation. A fourth assumption sits underneath those three, under a different name: ignorability (also called unconfoundedness, or no unmeasured confounders) — treatment assignment has to be independent of the potential outcomes, at least once you condition on what you've measured. The random draw into the campaign satisfies it by construction, the same way it satisfies positivity; outside a randomized design it means no factor you failed to measure is quietly driving both who got treated and what they would have earned either way. Like the other three, it is a claim about how the world generated the data, not something the same data can certify — testing it would need both Y_i(1) and Y_i(0) for one unit, exactly what the fundamental problem rules out. The best you can do is balance checks, placebo tests, and sensitivity analysis on how strong a hidden confounder would have to be to overturn the result.
Estimand choice is a decision made before the method, not after. ATE averages across all users. ATT — the effect on the treated — averages only over users who actually got the email. CATE is the effect for a slice sharing feature X = x. These are different numbers with different policy implications: using ATT — the effect among users who happened to get the campaign — to justify emailing the entire user base is an estimand error, since the users already reached may respond nothing like the users who never got the chance.
None of this means causal inference is impossible without perfect data. It means every causal number carries assumptions the same data can't verify — SUTVA, consistency, and positivity are claims about how the world generated the data, not properties a p-value certifies. Argue them from domain knowledge, design them away through randomization, or bound them with sensitivity analysis — never assume they hold just because the difference in means came out significant.
Key points
- Always specify the estimand before choosing an estimation method. ATE = average effect across all units. ATT = average treatment effect on the treated. ATC = average treatment effect on the control. CATE = conditional average treatment effect for units with features X = x. These are different quantities with different identification assumptions and different policy implications. Confusing ATT for ATE when recommending a universal rollout produces the wrong answer for the wrong population.
- Trap: SUTVA violations from network effects. In a social network, treating 10% of users and comparing to untreated users underestimates the true treatment effect — untreated users are indirectly affected by their treated connections. Their Y_i(0) is not the baseline outcome; it is the spillover-contaminated outcome. Test for SUTVA violations by comparing outcomes in clusters with high treated-neighbor density versus low treated-neighbor density. The structural fix is cluster-level randomization — assign whole friend groups or geographic regions to a single arm so spillover stays inside a cluster instead of crossing between treatment and control.
- Diagnostic: if your estimated ATE changes substantially when you reweight to match population demographics, either treatment effects are heterogeneous across subgroups or you have a positivity violation for some subgroup. A region of covariate space with only treated units means any estimate there is pure extrapolation. Report the estimand precisely — including which subpopulation the estimate applies to — before interpreting the result.
Causal inference is a missing data problem: for every unit you observe one potential outcome and must assume something about the other — and those assumptions are unverifiable from the same data you used to estimate the effect.
Recap
- The fundamental problem of causal inference: for each unit you observe only one of Y_i(1) (outcome if treated) and Y_i(0) (if untreated) — the other is the *counterfactual* and permanently missing. Causal inference is at heart a missing-data problem.
- ATE = E[Y_i(1) − Y_i(0)] is estimable even though the individual effect (ITE) is not: you never get Y(1)−Y(0) for one person, but a design that makes the missing outcome recoverable *in expectation* (randomization) lets you estimate the average across units.
- Three identification assumptions make it work: SUTVA (no spillover — one unit's treatment doesn't change another's outcome), consistency (the observed outcome for a treated unit *is* Y_i(1)), and positivity/overlap (every unit has nonzero probability of being in either arm, or you're extrapolating).
- Pick the estimand *first*, before choosing a method: ATE (all units), ATT (the treated), CATE (units with X=x) are different quantities with different identification assumptions and different policy implications — a first decision, not an afterthought.
- Classic estimand error: using the ATT — the effect among the users who *happened to receive* the treatment — to justify a *universal* rollout, when that already-reached group may respond nothing like the users who never got the chance.
- SUTVA breaks on network effects: treating 10% of a social network and comparing to untreated users *underestimates* the effect, because untreated users are indirectly affected by their treated connections — their Y_i(0) is spillover-contaminated, not the true baseline.
- Ignorability can never be tested from the same data used to estimate the effect (that would need both potential outcomes) — argue it from domain knowledge, embed it via randomization, or bound it with sensitivity analysis.
Check your understanding
Q1. Observed data shows users who saw an ad (T=1) had 20% higher conversion than users who did not (T=0). Can you conclude the ad caused 20% lift? What would you need to make a causal claim?
- A) Yes — the law of large numbers guarantees convergence to the true population mean, so with a large enough sample the observed 20% gap is automatically a consistent, unbiased causal estimate regardless of how exposure was assigned
- B) No — the 20% likely includes a selection bias term: E[Y|T=1] − E[Y|T=0] = ATE + bias. Need an RCT, or an observational strategy satisfying ignorability by measuring what predicts exposure and conversion
- C) Yes — controlling for age and gender in an OLS regression removes all selection bias, since demographic covariates are the only variables known to jointly drive ad exposure and purchase intent in ad-tech pipelines
- D) No — only a randomized controlled trial can ever support a causal claim here; propensity matching, instrumental-variable designs, and even doubly robust estimators are all statistically invalid substitutes no matter how precisely the covariates are measured or how large the sample grows
Q2. A clinical trial shows ATE = +5 points on health scale. A policymaker wants to mandate the drug for everyone. Is ATE the right estimand? What if trial enrolled only volunteers?
- A) Yes — ATE is always the correct estimand for population-wide mandates, and volunteer enrollment has no bearing on this because randomization within the trial fixes any selection issue automatically
- B) ATE is only valid when the trial sample perfectly matches the target population in size; otherwise the policymaker should default to using ATC instead of ATE, regardless of who volunteered
- C) Yes — randomization inside the trial guarantees the +5 estimate applies uniformly to every subgroup in the national population, including people who would never have volunteered to enroll
- D) The target for mandating everyone is ATE over the full population, but volunteers may really give ATT — check covariate overlap with the target population and reweight before extrapolating
Q3. You run an A/B test for a new social sharing feature. Control group engagement unexpectedly increased. Select the two statements that correctly explain what is happening and how to fix it.
- A) This is a SUTVA violation from interference between units — treated users share posts that surface in control users' feeds, inflating control's Y_i(0) above the true baseline and making the naive estimate understate the real effect
- B) The fix is cluster-based randomization — assign whole friend groups or geographic regions to one arm so spillover from treated to control users is contained within a cluster rather than crossing arms
- C) This is a novelty effect — treated users generate more content out of initial excitement, which mechanically lifts the platform-wide average and has nothing to do with the control group's own measured behavior
- D) This is regression to the mean — the control group happened to start from an unusually low baseline before the test began, and the rise is simply reversion toward its long-run average engagement level
Q4. Why can the ignorability assumption never be tested from data? What is the best you can do to support it?
- A) Ignorability can be fully confirmed with a balance test on observed covariates — once every measured variable is balanced between arms, that guarantees no unmeasured confounder remains either, regardless of how the assignment mechanism actually worked
- B) Ignorability is untestable because it is never possible to specify a correct propensity model, but adding enough covariates to that model eventually forces the assumption to hold by construction anyway
- C) Testing it needs both Y(0) and Y(1) for the same unit, exactly what the Fundamental Problem rules out. Best support: balance checks, placebo tests, sensitivity analysis on hidden-confounder strength
- D) Ignorability cannot be tested because all observational data is inherently biased, so the only defensible response is to discard observational analysis entirely and require an RCT in every single case
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →