Network Effects in A/B Tests: SUTVA Violations and How to Handle Them

Standard A/B testing assumes that a user's outcome depends only on which treatment they receive — not on what treatment other users receive. In social networks, marketplaces, and ride-hailing, this assumption is violated. A user assigned to control is affected by the behaviour of users assigned to treatment. SUTVA violation is the statistician's name for this, and it makes standard A/B tests produce severely biased estimates.

The Stable Unit Treatment Value Assumption (SUTVA) is the bedrock of causal inference from randomised experiments. It states: the potential outcome for unit i depends only on the treatment assigned to unit i, not on the treatments assigned to other units. In consumer tech, SUTVA is routinely violated.

When SUTVA fails

Social networks: you assign 50% of users to see a new newsfeed algorithm (treatment) and 50% to the old algorithm (control). Treatment users post more (the new algorithm is more engaging). Control users see posts from treatment users in their feed. Control users' engagement increases because their feed has more content — not because of any change to their own algorithm. Your control group is contaminated by the treatment group's behaviour.

Ride-hailing: 50% of drivers receive a surge pricing incentive (treatment). Treatment drivers take more rides. Fewer drivers are available for control-group riders — their wait times increase. The control group is harmed by the treatment. Standard A/B estimation shows treatment drivers do better and control riders do worse, but the true effect on the overall market is different from what the individual-level comparison shows.

Marketplace supply/demand: 50% of buyers receive a discount (treatment). They buy more. Supply is diverted toward treatment buyers — control buyers find fewer items available. Control group outcomes are degraded by the treatment.

Quantifying the bias

SUTVA violation causes interference between units — the treatment effect estimated by standard A/B analysis is a biased estimate of the global average treatment effect. The direction of bias depends on the sign of the interference: if treatment helps the treated and hurts the control (ride-hailing), the observed treatment effect is an overestimate. If treatment helps everyone in the network (viral content), the control group also benefits and the observed treatment effect is an underestimate.

Cluster randomisation: the standard fix

Instead of randomising individuals, randomise clusters of individuals who are likely to interact. Geographic clusters: randomly assign cities or DMA regions to treatment and control. Social clusters: randomly assign connected components of the social graph. Each cluster is treated or controlled entirely — interactions within the cluster are allowed; between-cluster interference is minimised.

Requirements: clusters must be large enough for statistical power, and between-cluster spillover must be negligible. Geographic experiments require dozens of matched city pairs and weeks of data collection. The matching (pairing similar cities) reduces variance.

Switchback experiments: time-based randomisation

In ride-hailing and supply-demand markets where clustering by user is infeasible (supply is global), switchback experiments randomise treatment assignment over time for an entire market: treatment during odd hours, control during even hours. This ensures all users in the market experience both conditions. Analysis accounts for time-based confounders (rush hour, day of week). Netflix and Lyft use switchbacks for marketplace-level experiments.

Network experimental designs: ego-cluster and bipartite

Ego-cluster randomisation: assign treatment at the cluster level where each cluster is a user's local network (the user plus all their friends). This captures most social influence effects while maintaining many independent clusters.

Bipartite randomisation: in two-sided marketplaces (Airbnb, eBay), randomise on one side (hosts or buyers) while measuring outcomes on both sides. This gives unbiased estimates of the treatment effect on the randomised side, with carefully controlled leakage to the other side.

Variance estimation under interference

When clusters are the unit of randomisation, variance must be estimated at the cluster level, not the individual level. Using individual-level variance gives a severely underestimated standard error and inflated statistical power — you think you have 1000 independent observations (individuals) when you really have 20 independent observations (clusters). Cluster-robust standard errors correct for within-cluster correlation.

Try on Colab: simulate a social network (Erdős-Rényi graph, 1000 nodes). Assign treatment to 50% of users. Simulate an outcome where each user's outcome depends on their own treatment plus the fraction of their friends who are treated (network effect). Run standard A/B analysis (ignoring network) and compare to cluster-randomised analysis (randomise by connected component). Observe the bias in the individual-level analysis.