Sensitivity Analysis for Causal Claims
E-values, Rosenbaum bounds, placebo tests, falsification design
You have estimated that a job training program increases earnings by $3,000/year using a matching estimator. Your result assumes no unmeasured confounders — the unconfoundedness assumption. But what if motivation is unobserved and it causes both training program enrollment and higher earnings? How strong would this unmeasured confounder need to be to reduce your estimated effect to zero? If the answer is "only a moderate confounder," your result is fragile. If the answer is "a confounder stronger than any observed covariate," your result is robust. Sensitivity analysis quantifies this threshold.
Rosenbaum sensitivity analysis is designed for matching studies. It reports the Γ (Gamma) parameter: the maximum ratio by which the odds of treatment can differ between two matched units due to unobserved variables, while still rejecting the null hypothesis. Γ = 1.5 means: even if an unobserved confounder could cause 50% more treatment odds, you would still find a significant effect. Γ = 1.0 means any unobserved confounding overturns the result. Sensitive at Γ = 1.2 is fragile; robust to Γ = 3.0 is credible.
The E-value (VanderWeele and Ding) is more general. It is the minimum strength of association that an unmeasured confounder would need with both treatment and outcome to fully explain away the observed effect. E = RR + √(RR(RR − 1)) where RR is the observed risk ratio. Compare the E-value to the associations of observed covariates — if the E-value is smaller than your strongest observed confounder's association, an unmeasured confounder of that strength could explain your result away.
Placebo tests provide indirect evidence from a different angle. A placebo outcome is one the treatment logically cannot affect — if the analysis shows a significant "effect" on the placebo, something correlated with treatment is also correlated with the outcome, signaling confounding in the main estimate. An event study pre-period check in DiD is a placebo treatment test: pre-treatment coefficients should cluster near zero.
What sensitivity analysis does not do: prove the causal estimate is correct. It tells you how fragile or robust the estimate is to violations of the identifying assumption. A high Γ says the conclusion survives substantial hidden bias — this is reassuring but not proof of causal validity. It is a communication tool that makes the credibility of the claim explicit, not a proof.
Key points
- Always report an E-value or Rosenbaum Γ alongside any observational causal estimate. Without it, readers have no way to assess the credibility of the causal claim. This is becoming a standard requirement in top journals and should be standard in internal data science reports. Compare the E-value against the observed associations of covariates you already control for — if it is smaller than any of those, the residual confounding threat is concrete, not hypothetical.
- Trap: running multiple sensitivity analyses and reporting only the one with the highest Γ. Pre-specify your sensitivity analysis approach before seeing results. Sensitivity analysis p-hacking is less common than outcome p-hacking but follows the same logic — choosing the most favorable framing after seeing results. Use the same estimator and assumptions throughout, and report the full range of sensitivity estimates across specification choices.
- Diagnostic: if your E-value is smaller than the association of any variable in your observed covariate set — for example, E-value = 1.8, but industry sector has RR = 2.5 with outcome — the unmeasured confounder needed to explain away your result is weaker than covariates you are already controlling for. This is a warning sign of residual confounding. Report it honestly rather than treating the E-value as evidence of robustness when the comparison to observed covariates undermines it.
Sensitivity analysis does not prove a causal estimate — it reports how much hidden confounding it would take to overturn it, making the fragility or robustness of the claim explicit rather than hidden.
Recap
- The core question: given a result that assumes no unmeasured confounders, *how strong* would a hidden confounder have to be to reduce your effect to zero? "Only a moderate one" ⇒ fragile; "stronger than any observed covariate" ⇒ robust.
- Rosenbaum Γ (for matching studies): the maximum ratio by which two matched units' *odds of treatment* could differ due to unobserved variables while you still reject the null — it quantifies how much hidden bias the conclusion tolerates.
- Reading Γ: Γ = 1.0 means *any* unobserved confounding overturns the result; sensitive at Γ = 1.2 is fragile; robust up to Γ = 3.0 is credible.
- E-value = RR + √(RR(RR−1)): the *minimum* association strength an unmeasured confounder would need with *both* treatment and outcome to fully explain away the observed effect — more general than Γ, works beyond matching.
- Contextualise the E-value against your observed covariates: if it's *smaller* than the association of a confounder you already control for (e.g. E = 1.8 but industry sector has RR = 2.5), the residual-confounding threat is concrete, not hypothetical — report it honestly.
- Placebo tests give indirect evidence: run the analysis on an outcome the treatment logically can't affect — a "significant" effect there means something correlated with treatment is driving the outcome, signalling confounding in the main estimate.
- It does *not* prove causality: sensitivity analysis makes the fragility or robustness of a claim explicit — a high Γ is reassuring, not verification. It's a communication tool, not a proof.
Check your understanding
Q1. Your matching analysis shows treatment increases survival rates by 30% (RR=1.3). You compute an E-value of 1.9. What does this mean and how do you use it?
- A) An E-value of 1.9 means the study has roughly 90% statistical power to detect any confounder with RR ≥ 1.9, and any confounder weaker than that threshold is treated as too weak to explain away the observed effect
- B) An E-value of 1.9 is essentially a p-value transform confirming RR=1.3 is significant at p<0.05, adjusted to account for the multiple comparisons implicit in a typical matching study's many covariates and subgroups
- C) An E-value of 1.9 literally counts how many additional covariates — 1.9 of them — would need to be added to the matching model before the observed effect fully disappears from the estimate entirely
- D) A confounder needs RR≥1.9 with BOTH treatment AND survival to explain away RR=1.3 — compare to known ones like SES (RR≈1.4, too weak) or smoking (RR≈2.5, strong enough) to judge robustness
Q2. Your DiD estimate shows minimum wage increase reduced employment by 3%. A critic says treated and control counties had different pre-trends. How do you test and respond?
- A) Add pre-period interactions to the TWFE regression and plot an event study — coefficients should be near zero pre-treatment; if not, add trends or synthetic control and report the test alongside the estimate
- B) Respond that parallel trends is simply untestable by definition, since pre-treatment trend similarity logically says nothing whatsoever about post-treatment similarity, so the critic's concern cannot be addressed empirically at all
- C) Re-run the DiD using a much shorter pre-post window, such as one quarter instead of one year — if the estimate holds up across that narrower window alone, pre-trends are treated as fully irrelevant to the -3% result
- D) Dismiss the concern outright, since including county and time fixed effects in the TWFE regression is treated as sufficient by construction to absorb any pre-existing trend divergence, making pre-trend testing entirely unnecessary
Q3. You want to test whether IV analysis for effect of college education on earnings (using proximity as instrument) is confounded. Select the two genuine falsification tests.
- A) Run the same IV analysis on a cohort too old to have benefited from nearby colleges at the time of their schooling decisions — a significant "effect" there signals proximity is correlated with local labor markets directly, not just through education
- B) Regress pre-determined covariates such as parents' education and family income on the instrument — if proximity predicts these, it is capturing geographic family sorting rather than pure exogenous variation
- C) Compare IV estimates across rural and urban counties — a significant difference between the two subsamples is treated as sufficient on its own to prove proximity is only capturing urbanicity rather than genuine college access
- D) Add a second instrument such as state college subsidies and run only a Sargan-Hansen overidentification test — passing this single test is treated as conclusive proof the exclusion restriction holds for both instruments
Q4. After reporting a positive causal effect of a product feature on revenue using DiD, a sceptical executive asks: 'How fragile is this result?' How do you answer rigorously?
- A) Report the p-value and the confidence interval alone — a p-value under 0.01 with a CI that excludes zero is treated as sufficient by itself to establish that the whole result is not fragile in any way
- B) Run a specification check: vary the pre-post window, vary the control group, check related outcomes move together, vary the regression, and place the estimate against placebo pseudo-date estimates
- C) Run a retrospective power analysis — if the original test cleared 80% power, the result is treated as automatically robust, since underpowered tests are assumed to be the only possible source of fragile findings
- D) Declare the result non-fragile outright, since two-way fixed effects DiD is treated as robust to all forms of time-invariant confounding by construction, with sensitivity analysis needed only for purely cross-sectional designs
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →