Calibration Monitoring in Production
Reliability diagrams from logged predictions, ECE over time, recalibration triggers
An insurance pricing model outputs a claim probability. On launch day it's beautifully calibrated: a prediction of 0.15 really does mean a 15% claim rate — dead on the reliability diagram's diagonal. Three months later, the same 0.15 predictions are actually claiming at 22%. The model is now *underestimating* risk by 7 points in that bucket. And the pricing team feeds its output straight into premiums, so for three months it has been silently undercharging exactly the high-risk customers. No alert fired. AUC is a steady 0.82.
AUC will never catch this, because AUC measures a different thing
AUC measures *ranking* — does the model put riskier customers above safer ones? Calibration measures *probability accuracy* — does 0.15 actually mean 15%? These are independent. A model can rank every customer perfectly (great AUC) while being wrong about every customer's *absolute* risk (terrible calibration). Anywhere the output is used as a real probability — pricing, risk scoring, expected-value math — you must monitor both, separately.
Why calibration drifts even when ranking holds
As the input distribution shifts, the model's probability estimates stop matching the true rates in the new population — even though it still knows *who* is riskier than whom. It just no longer knows *by how much.* You quantify this with Expected Calibration Error:
$ECE = \sum_{bins} (n_{bin}/n)\,|\,\text{confidence}_{bin} - \text{accuracy}_{bin}\,|$
An ECE tripling from 0.03 at launch to 0.09 in three months isn't a footnote — it's mispricing at scale.
Fixing it rarely means retraining
*Platt scaling* fits a small logistic regression on top of the model's outputs using recent labeled data — a few hours with ~1,000 examples. *Temperature scaling* uses a single scalar T: new_prob = σ(logit / T). Both are fast and leave the base model untouched. Trigger recalibration when ECE runs 0.03 above baseline for three days straight.
And kill the assumption that "stable AUC means calibration is fine." A model can post AUC 0.90 with ECE 0.15 — ranking everyone right while its probabilities are off by 15 points. For a fraud model, that means your chosen threshold no longer buys the precision-recall tradeoff you designed. For pricing, it means systematic mispricing. Monitor both, independently, on every batch of labels that arrives.
Key points
- Monitor ECE on every batch of ground truth labels that arrives — calibration drift is silent and systematic, exactly the kind of error that causes financial mispricing or risk misallocation to compound undetected. Plot the reliability diagram alongside the ECE scalar: ECE tells you the magnitude, the diagram tells you which buckets are miscalibrated and in which direction.
- Trap: applying temperature scaling fit on stale data. If you recalibrate using labels from 6 months ago, you are correcting for past calibration drift, not current. Fit recalibration only on recent labels from the last 30–60 days. A recalibration model trained on stale data can shift the current calibration in the wrong direction, making ECE worse instead of better.
- Diagnostic: if ECE is increasing but AUC is stable, apply temperature scaling as the first-line fix — it corrects the overall miscalibration in one parameter without retraining. If ECE improvement from temperature scaling is greater than 0.03, the recalibration was justified. If improvement is less than 0.01, the problem is per-bucket conditional miscalibration rather than a uniform scaling issue, and isotonic regression is required instead — a non-parametric monotonic step-function fit that can correct each probability bucket independently, unlike Platt/temperature scaling's single global parameter.
Calibration drift is invisible to AUC — monitor ECE on every label batch, recalibrate with Platt or temperature scaling on recent data when ECE crosses threshold, and never wait for a business stakeholder to notice the mispricing.
Recap
- AUC ≠ calibration: ranking (who is riskier) is independent of probability accuracy (does 0.15 mean 15%).
- Calibration drifts when inputs shift: model still knows who is riskier, not by how much.
- ECE: $ECE = \sum_{bins} (n_{bin}/n)\,|\text{confidence}_{bin} - \text{accuracy}_{bin}|$; tripling 0.03→0.09 is mispricing at scale.
- Fix without retraining: Platt scaling (small logistic) or temperature scaling ($\sigma(\text{logit}/T)$) on recent labels.
- Fit only on recent labels (30–60 days): stale recalibration can push ECE the wrong way.
- Trigger at ECE 0.03 above baseline for 3 straight days.
- Temperature scaling first; if improvement <0.01 it's per-bucket miscalibration — use isotonic regression.
Check your understanding
Q1. A loan default model's ECE has increased from 0.03 (at deployment) to 0.09 (current) over 4 months. Reliability diagram shows the model is now overconfident for predictions > 0.7. What is your response?
- A) No action needed yet — 0.09 sits below the commonly cited 0.1 threshold; review next quarter
- B) Retrain the full model right now — calibration decay of this size always means concept drift
- C) Run a chi-squared significance test on the score distribution before any remediation is committed
- D) ECE tripled — lower the threshold as a stopgap, recalibrate on recent labels, investigate drift
Q2. Which two statements correctly explain why a pricing model's AUC can hold steady at 0.82 while its 0.15 predictions actually claim at 22%?
- A) AUC and calibration measure independent things — one scores rank order, the other scores probability
- B) A model can keep ordering risky customers above safe ones correctly while its absolute scores drift off
- C) AUC of 0.82 isn't high enough to guarantee calibration; only AUC above 0.95 gives reliable probabilities
- D) Calibration drift always drags AUC down too, so the steady AUC means the diagram was mismeasured
Q3. You apply temperature scaling and ECE improves by only 0.008. What does this small improvement tell you?
- A) The recalibration ran on stale labels; refit on the last 30–60 days and ECE will drop as expected
- B) Temperature scaling never works alone, so full retraining is the only option left for this model
- C) A single-parameter rescale barely helped — this is per-bucket miscalibration, isotonic is needed
- D) A tiny gain under 0.01 confirms the model was already well calibrated, so no further step is needed at all
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →