Distribution Shift

The core assumption of supervised learning — train and deploy distributions match — is almost always violated in production.

You trained a recommendation model on January user data and shipped it in March. By May, engagement is down 30%. No error, the API responds, the model happily returns predictions. They are just *wrong*. Welcome to distribution shift — the quiet killer of production models, and the reason "it worked in testing" is never the end of the story.

Here is the trap that makes it so dangerous: the model never tells you it is lost. A confidence score measures how far an input sits from the model's decision boundary — *not* how far that input sits from anything the model was trained on. So a user unlike anyone in training can still get a *high-confidence* prediction that happens to be completely wrong. You do not find out from the model. You find out weeks later, when the real engagement numbers arrive.

The world can shift in three different ways, and telling them apart decides whether you need a five-minute fix or a two-week data effort.

Covariate shift — the inputs move

The *kinds of users* changed (a February product change brought new behaviour), but the underlying rule linking behaviour to engagement still holds. Your old labels are still correct; you just have fewer training examples that look like today's users. This one you can sometimes patch *without* retraining, by importance weighting — lean harder on the training examples that resemble current traffic and less on the ones that do not.

Concept drift — the rule itself moves (the bad one)

Now the *meaning* of the features changed. Fraudsters in 2024 have learned to make fraudulent transactions look legitimate, so a pattern that screamed "fraud" in your training data now looks perfectly innocent. Reweighting old data cannot save you — the old labels are simply *wrong* about today's world. There is no shortcut: you need fresh labelled data and a retrain.

Prior shift — just the mix changes (the easy one)

The fraud *rate* rose from 0.1% to 0.3%, but fraud itself still looks the same. Here you can adjust the model's outputs by re-estimating the new class balance — no retrain required.

You cannot fix what you cannot see

All of this is invisible without monitoring, so detection has to come *before* diagnosis. Watch each important feature's distribution against its training baseline — a common gauge is PSI (population stability index): under 0.1 is calm, 0.1–0.2 says "go investigate," and over 0.2 is a retraining trigger. Watch the *prediction* distribution too; if it drifts while the inputs look stable, that is a fingerprint of concept drift. And a neat trick to confirm covariate shift: train a quick classifier to tell "training row" from "production row" — if it succeeds easily, the two worlds really have diverged. Build this monitoring in from day one, or the business will discover the shift before you do.

The three shifts, in notation

The names map cleanly to which probability moved. Covariate shift: P(X) changes, P(Y|X) holds — the inputs move but the rule is intact. Label/prior shift: P(Y) changes, P(X|Y) holds — the class mix moves but each class still looks the same. Concept drift: P(Y|X) changes — the *rule itself* moves, which is the one no reweighting can fix. Being able to say "which distribution changed?" is exactly how you pick the response, so it's worth carrying the notation, not just the stories.

Detection has a metric menu, not just PSI

PSI is the industry default, but know the alternatives and when they're better. The KS test measures the largest gap between two CDFs (good for continuous features) — but its p-value is sample-size-sensitive: with enough production traffic, even a trivial, practically meaningless shift reads as statistically significant, so always check the *magnitude* (e.g. via PSI) before acting on a low p-value alone. Wasserstein (earth-mover) distance captures *how far* the mass moved, which PSI's binning can miss. KL / Jensen-Shannon divergence quantify distributional difference (JS is symmetric and bounded). MMD (maximum mean discrepancy) is a kernel-based two-sample test that works in high dimensions. For categorical features, compare frequencies (chi-squared) and watch for new categories. Use several — a shift that hides from one metric often shows in another.

Label shift can be estimated without new labels

Prior/label shift has a neat property: you can often correct it *without* fresh labels. Black Box Shift Estimation (BBSE) uses your existing model's confusion matrix plus the *distribution of its predictions* on the new data to estimate the new class priors, then reweights the outputs. So if only the class balance moved (fraud rate 0.1% → 0.3%, fraud still looks the same), you re-estimate the prior and adjust — no relabelling, no retrain. This is why diagnosing the shift *type* pays off: label shift is the cheapest to fix.

Adaptation strategies, cheapest to most involved

Match the response to the shift. Importance weighting for covariate shift (up-weight training rows resembling current traffic). Prior/output correction (BBSE) for label shift. Domain adaptation methods (align feature representations between source and target) when you have unlabelled target data. Online / continual learning to keep updating from a stream, and test-time adaptation (adjust batch-norm statistics or a few parameters to the incoming batch) for mild drift. And for concept drift, the honest answer remains fresh labels and a retrain — there is no free lunch when the underlying rule has genuinely changed.

Key points

• Use PSI per feature as your first production monitoring signal, not aggregate accuracy. PSI buckets a feature's distribution into deciles and computes weighted divergence from the training baseline. PSI < 0.1 is stable; 0.1–0.2 warrants investigation; > 0.2 is a retraining trigger. Track per-feature, not aggregate — a single important feature shifting while others are stable will be invisible in any aggregate metric. Also monitor the prediction distribution: if P($\hat{y}$) shifts without any feature shift, you have concept drift.
• The most common production trap: scheduled retraining (weekly, monthly) in a domain where shift happens in days. Fraudsters observe your model's behaviour and adapt within weeks of a new deployment. A fixed monthly retraining schedule is already two to four weeks behind by the time it fires. Build trigger-based retraining: fire when PSI exceeds 0.2 on a key feature, or when performance on a labeled validation window drops beyond a threshold. Scheduled retraining is acceptable in stable domains; in adversarial or fast-moving domains, it guarantees you are always working with stale assumptions.
• Diagnose shift type before choosing a response. Stable feature PSI but degraded performance = concept drift signature (P(X) unchanged, P(Y|X) changed) — requires new labeled data and retraining, no shortcut. Shifted feature PSI but degraded performance = covariate shift candidate — try importance weighting first, which can buy weeks before a full retrain. To confirm covariate shift, train a logistic regression to classify "is this example from training or production?" If it classifies with high accuracy, the distributions are meaningfully different and importance weighting is appropriate.
• Name which distribution moved, and detect with more than PSI. Covariate shift = P(X) moves, P(Y|X) holds (importance-weight); label/prior shift = P(Y) moves, P(X|Y) holds (correct the prior); concept drift = P(Y|X) moves (retrain, no shortcut). Beyond PSI, detect with the KS test and Wasserstein distance for continuous features, KL/Jensen-Shannon and MMD for distributions, and chi-squared/new-category checks for categoricals — a shift hidden from one metric often shows in another.
• Match the fix to the shift, cheapest first — and label shift needs no new labels. Label shift can be corrected without relabelling via Black Box Shift Estimation (use the model's confusion matrix and its prediction distribution on new data to re-estimate class priors, then reweight outputs). Adaptation ladder: importance weighting (covariate), prior/output correction (label), domain adaptation with unlabelled target data, online/continual learning and test-time adaptation (mild drift), and fresh labels + retrain for genuine concept drift. Diagnosing the shift type is what tells you whether you need five minutes or two weeks.

Recap

• Shift is silent: the model outputs confident predictions on shifted data, degrading until ground-truth labels arrive. Only monitoring catches it first.
• Name which distribution moved: covariate shift = P(X) moves, P(Y|X) holds; label shift = P(Y) moves; concept drift = P(Y|X) moves.
• Monitor per-feature PSI, not aggregate accuracy: <0.1 stable, 0.1–0.2 investigate, >0.2 retrain — one shifting feature is invisible in aggregates.
• Diagnose type before responding: stable PSI + degraded performance = concept drift (no shortcut); shifted PSI = covariate shift (try importance weighting).
• Confirm covariate shift by training a classifier to tell train-vs-production apart — high accuracy means the distributions genuinely differ.
• Label shift needs no new labels: BBSE re-estimates class priors from the confusion matrix + prediction distribution, then reweights outputs.
• Scheduled retraining fails in fast/adversarial domains — fraudsters adapt in days; fire on PSI/performance triggers, not a monthly calendar.

Check your understanding