Monitoring Taxonomy
Data drift, concept drift, model decay, infrastructure drift
Your fraud model is suddenly catching 30% fewer frauds. Something changed — but what? Four very different causes could produce this exact drop. The `income` feature distribution shifted, so the model is seeing a different population than it trained on. Or fraud tactics evolved, so the same transaction profile now carries a different fraud probability. Or the overall fraud rate went up while the model's per-transaction accuracy is unchanged. Or the feature pipeline broke and is quietly sending nulls downstream. Each demands a completely different fix. Without a way to tell them apart, you are debugging blind.
Four causes, four precise questions
*Data drift* asks: has P(X) changed? The inputs shifted — your transactions now have a different income distribution than training. You detect it with statistical tests — PSI (Population Stability Index), the KS (Kolmogorov–Smirnov) test, and chi-squared — on feature distributions. It may or may not hurt performance, depending on whether the input-output relationship also moved.
*Concept drift* asks: has P(Y|X) changed? The relationship itself moved — a profile that was 3% fraud is now 15% because tactics evolved. You can only confirm it with labels, by watching performance degrade. This one genuinely needs retraining on recent data.
*Prior shift* asks: has P(Y) changed while P(X|Y) held? The base rate went from 1% to 3% — more fraud, not different fraud. Like concept drift, you can only confirm it with labels — but where concept drift shows the same transaction profile mapping to a different fraud rate, prior shift shows the overall base rate moving while each profile's own rate stays put. Often a threshold adjustment is enough.
*Infrastructure drift* asks: is the pipeline broken? Schema changed, latency spiked, a dependency degraded. Often the failure is silent: failed requests fall back to a default not-fraud prediction instead of erroring loudly, so recall drops from false negatives even though the features that did make it through the pipeline look completely normal. That's an engineering fix, not a model fix.
Why the taxonomy earns its keep
A broken pipeline and real concept drift can produce the *identical* drop in recall. But patch concept drift with an engineering fix and nothing improves; "fix" a broken pipeline by retraining and you burn a week to land exactly where you started. This is the difference between a 30-minute fix and a wasted week.
And that's why "just watch the output score distribution" isn't enough. Output monitoring tells you behavior changed, not *why.* Feature monitoring tells you what moved in the data. Infrastructure monitoring tells you if the cause isn't in the data at all. You need all four layers to separate a drift that needs retraining from a pipeline bug that needs a half-hour patch — and collapsing everything into "model problem" is the single most expensive habit in production ML.
Key points
- Build four monitoring layers in order: infrastructure → data → predictions → performance. Infrastructure (latency, error rate) and prediction monitoring give you real-time signals. Data drift and performance monitoring give you explanatory power. The order matters — a 5% endpoint error rate is infrastructure, not drift. Mixing up the layer diagnosis means applying the wrong remedy.
- Trap: conflating data drift with model performance degradation. A feature can drift significantly without affecting model performance (if the model is robust to that variation) or affect it minimally (if that feature's importance is low). Always measure whether performance actually degraded before triggering retraining. Reflexive retraining on every drift alert wastes pipeline compute and can introduce regressions on stable segments of the population.
- Diagnostic: when performance drops, check the four layers simultaneously and look for which combination fires. Infrastructure clean + data drifting + performance degrading = concept or data drift requiring investigation. Infrastructure clean + data clean + performance degrading = concept drift in a dimension not covered by your feature monitoring. Infrastructure errors + data clean = pipeline bug requiring an engineering fix. The combination is the diagnosis.
A drop in model performance has four possible causes — data drift, concept drift, prior shift, and infrastructure drift — and applying the wrong remedy to any of them wastes time while the problem compounds.
Recap
- Four causes, one symptom: a recall drop can be data drift, concept drift, prior shift, or infrastructure drift.
- Data drift = P(X) moved. Inputs shifted; detect with PSI/KS/chi-squared. May or may not hurt.
- Concept drift = P(Y|X) moved. Same profile, different fraud rate; only labels confirm it; needs retraining.
- Prior shift = P(Y) moved. More fraud, not different fraud; often just a threshold adjustment.
- Infra drift = pipeline broke. Schema/latency/nulls; an engineering fix, not a model fix.
- Four layers in order: infrastructure → data → predictions → performance. The combination that fires is the diagnosis.
- Wrong remedy = wasted week. Retraining a broken pipeline lands you exactly where you started.
Check your understanding
Q1. Your fraud model is flagging more transactions as fraudulent over time, but actual fraud labels (available after investigation) show no increase in fraud rate. What type of drift is this?
- A) Concept drift — P(Y|X) shifted, so the same features now map to a higher fraud probability
- B) Data drift — P(Y) is stable, feature shifts are mechanically pushing scores up
- C) Infrastructure drift — a library update altered score computation, inflating raw outputs
- D) Overfitting decay — the model memorized noise and now generalizes badly to new traffic
Q2. Recall dropped 30% overnight. Feature PSI is clean, prediction score distribution is unchanged, but the endpoint error rate jumped to 6%. Which layer is the cause?
- A) Concept drift — P(Y|X) shifted, and only labels can confirm the true magnitude of change
- B) Data drift — P(X) moved enough that the model now scores an unfamiliar population
- C) Infrastructure drift — clean features plus a 6% error rate means a broken pipeline
- D) Prior shift — the base rate P(Y) rose, leaving the fixed threshold in the wrong place
Q3. Which two statements correctly describe why reflexively retraining on every drift alert is the "single most expensive habit" the taxonomy warns against?
- A) A feature can drift with zero performance impact, so retraining wastes compute chasing a problem that may not exist
- B) Retraining on a drift alert can introduce regressions on population segments that were never actually affected
- C) Retraining is computationally free, so the only real cost of reflexive retraining is a short delay before deploy
- D) Drift alerts are almost always caused by infrastructure, so the correct fix is always to wait on the on-call queue
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →