Prediction Distribution Monitoring

Output distribution shift, score distribution, confidence calibration drift

Your fraud model is live. Labels won't arrive for 7 days — chargebacks take that long to confirm. Infrastructure looks perfect: normal latency, 0% errors. So you cannot measure accuracy until next week. And yet you can see the *predictions* right now, and three of them are already whispering that something moved.

Three signals you have before any label

*The mean score fell.* Last month the average predicted fraud probability was 3.1%; this month, 2.7%. The model is scoring this week's transactions as less risky. Right or wrong, that's worth a look.

*Confidence collapsed.* The share of predictions above 0.8 confidence dropped from 12% to 4%. The model is now hesitant about cases it used to call decisively.

*The histogram flattened.* Last month's scores were bimodal — clear fraud piled high, clear non-fraud piled low, a trough between. This month it's a single lump near 0.5. The model has lost its ability to separate the two classes.

What each one tends to mean

A falling score distribution means either fewer genuinely risky events (could be real) or a feature that shifted — which could be a genuine population change (a new customer segment, a seasonal cohort — see monitoring_taxonomy's income-shift example) or a pipeline bug; the shift alone doesn't tell you which. A dropping high-confidence rate means the inputs have moved outside the model's training range — it's seeing unfamiliar data. A flattening histogram means a key discriminative feature is missing or corrupted, leaving the model unable to tell the classes apart.

You measure all of this with the same tools as feature drift: PSI on the score distribution (same 0.1/0.2 bands), Jensen-Shannon divergence between current and reference histograms (bounded in [0, 1]), and a z-test on the mean-score shift — z = (current mean − baseline mean) / SE(baseline mean), flag when |z| > 2. That |z| > 2 rule is exactly keyPoint 1's alert-at-2σ check computed as a formal test statistic — the z-test isn't a separate metric, it's how the 2σ deviation gets measured.

What prediction monitoring is — and isn't. It tells you the model's *behavior* changed, not that the model is *wrong.* Maybe fraud genuinely dropped and the lower scores are correct; maybe a feature pipeline broke. That's exactly why it's your earliest warning system — it fires days ahead of the delayed labels — and exactly why the right first move when it fires is to *investigate the inputs,* not to assume the model is broken and retrain on reflex.

Key points

• Set reference baselines for score distribution, high-confidence rate, and histogram shape from the first 4 weeks of deployment — these are your "healthy" benchmarks. Alert when current statistics deviate more than 2σ from baseline. The 4-week window captures natural weekly variation so that normal Monday-vs-Friday patterns do not generate spurious alerts.
• Trap: setting alert thresholds too tight. If your score distribution naturally varies ±15% week-over-week due to business seasonality, a 2σ threshold generates constant alerts. Calibrate thresholds based on observed natural variation from the first month of production traffic, not on theoretical distributions. A threshold that produces more than one alert per week during normal operation needs to be widened.
• Diagnostic: when prediction distribution shifts, check the feature distributions of high-confidence positive predictions specifically. If a key feature — for example, `transaction_velocity` — has suddenly shifted for high-confidence predictions, the feature pipeline is the likely cause — but confirm before concluding that: a genuine population shift (a new market, a seasonal cohort, as in monitoring_taxonomy's income example) can produce the identical signature without any pipeline bug, so check pipeline logs and whether the new values are even plausible first. If feature distributions are clean for high-confidence predictions but the score distribution has still shifted, the change is in the data itself, not the pipeline — possible concept drift or genuine fraud rate change.

Recap

• See predictions now, labels in 7 days: three signals fire before any label.
• Falling mean score: fewer risky events (real) or a feature shifted — genuine population change or pipeline bug, the shift alone doesn't say which.
• Collapsing high-confidence rate: inputs moved outside the training range — unfamiliar data.
• Flattening histogram: a key discriminative feature is missing or corrupted — classes no longer separate.
• Same tools as feature drift: PSI on scores (0.1/0.2), Jensen-Shannon in [0,1], z-test on mean shift (|z|>2 — the formal version of the 2σ alert rule).
• Set baselines from first 4 weeks; alert at 2σ — captures Monday-vs-Friday variation.
• Behavior changed ≠ model wrong: earliest warning, so investigate the inputs before retraining on reflex.

Check your understanding