Causal Price Experiments: A/B, Geo & Switchback

Why you almost never A/B test price at the user level, and how geo holdouts and switchback designs recover an unbiased elasticity

You cannot read elasticity off observational data — so you experiment. But price is the one variable where the obvious experiment, randomizing the price *per user*, is usually the wrong design: it is often unfair or illegal, and even when allowed it leaks and interferes, quietly breaking the assumptions that make an A/B test valid.

Per-user price randomization violates SUTVA and invites legal risk. A clean A/B test assumes the Stable Unit Treatment Value Assumption: one user's treatment doesn't affect another's outcome. Prices break this because users talk and compare — showing person A $9 and person B $12 for the identical item means B can see A's price, feel cheated, and change behavior (churn, screenshot, complain). The treatment leaks across units. On top of the interference, charging different people different prices for the same good is a fairness and often a legal problem. So the per-user RCT is frequently off the table before the stats even matter.

Geo experiments randomize the market, not the person. Assign whole cities/regions to treatment (new price) or control (old price), then read the effect with difference-in-differences (compare the treated-vs-control change over the pre/post window) or synthetic control (build a weighted combination of control cities that tracks the treated city pre-period, then measure the divergence after). Everyone *in* a market sees the same price, so it's fair and consistent; randomization is at the market level, so there are usually few units — the key limitation is statistical power and needing good matched controls.

Switchback designs randomize price over *time* — the fix for strong interference. In a marketplace where supply and demand slosh across the whole system (rideshare, food delivery), even a geo split interferes: a price change in one zone spills into neighbors. Switchbacks turn the *entire market's* price on and off in short randomized time windows (e.g., 30-minute blocks alternating high/low) and compare outcomes across windows. Because the whole market is treated at once, cross-unit spillover within a window is absorbed rather than contaminating a control group. The catch is temporal autocorrelation (adjacent windows aren't independent) and carryover, which the analysis must account for. Whichever design you pick, the payoff is the same: an unbiased %ΔQ / %ΔP — the causal elasticity that observational data could never give you.

Key points

• Per-user price randomization breaks SUTVA and courts legal risk. Users compare prices, so one person's price affects another's behavior (interference/leakage) — and charging different people different prices for the same item is a fairness and often legal problem. The obvious A/B test is usually off the table.
• Geo experiments randomize markets, not people. Assign whole cities to old vs new price; everyone in a market sees the same price (fair, consistent). Read the effect with difference-in-differences or synthetic control. The trade-off is few randomization units → limited power and a need for well-matched controls.
• Switchback designs randomize price over time to handle strong interference. In marketplaces where supply/demand slosh system-wide, even geo splits spill over. Turning the whole market's price high/low in short randomized windows absorbs within-window spillover; the cost is temporal autocorrelation and carryover the analysis must model.
• All three designs exist to recover an unbiased %ΔQ / %ΔP. The experiment is how you get the causal elasticity that observational data cannot: it forces price to vary for reasons unrelated to demand, breaking the endogeneity that biases naive regressions.

Recap

• Per-user price randomization usually fails twice: it breaks SUTVA (users compare prices → treatment leaks across users) and it is unfair/illegal to charge different people different prices for the same good. Off the table before the stats matter.
• Geo experiments randomize markets, not people: whole cities get old vs new price (fair, consistent within a market). Read via difference-in-differences or synthetic control. Cost: few units → limited power, needs matched controls.
• Switchback designs randomize price over time: flip the whole market high/low in short randomized windows; within-window system-wide spillover is absorbed rather than contaminating a control group. Cost: temporal autocorrelation and carryover to model.
• Choose by interference: clean markets → geo; strong system-wide marketplace spillover → switchback. Per-user only in the rare fair/legal, no-comparison case.
• The payoff is a causal %ΔQ / %ΔP: the experiment forces price to move independently of demand, breaking the endogeneity that biases observational elasticity.

Check your understanding