Calibration Loss in Production: When 95% AUC Predicts 60% Precision
Your model has 0.95 AUC offline. In production, precision at the same threshold is 0.60. AUC doesn't predict absolute precision because AUC only measures rank ordering — it doesn't care about calibration. A model can have perfect AUC and completely miscalibrated probabilities. Here's what calibration actually means, why production systems need it, and how to detect and fix calibration loss before it breaks your downstream decisions.
Calibration is a property that nearly every ML engineer misunderstands. Here's the correct definition: a model is calibrated if, when it predicts probability p, the actual frequency of the positive outcome among those predictions is approximately p.
Example: if a model predicts P(fraud) = 0.8 for 1000 transactions, and 800 of them are actually fraudulent, the model is well-calibrated. If only 200 are fraudulent, the model is vastly overconfident.
AUC measures rank ordering, not calibration. A perfectly ranked model with AUC 0.99 can be completely miscalibrated — assigning probabilities 100× too high or 100× too low while maintaining perfect rank order.
Why calibration matters in production:
A fraud model with miscalibrated scores produces downstream effects:
How calibration loss happens:
1. Class imbalance without calibration: A model trained on 1% fraud learns to output probabilities in [0.001, 0.05] — matching the base rate. If you deploy it to a segment with 10% fraud (different population), the same scores now calibrate to [0.01, 0.5]. The model has shifted segments without recalibration.
2. Probability scaling mismatch: A neural network trained with sigmoid activation outputs probabilities. A tree-based model using LightGBM outputs raw scores. If you mix them in an ensemble without recalibration, the ensemble probabilities are nonsense.
3. Train-test distribution shift without adjustment: A model trained on data with P(Y=1) = 0.02 learns to scale its outputs toward that base rate. Deployed to a segment with P(Y=1) = 0.15, the same scores now underestimate probability.
4. Threshold calibration without probability calibration: You choose a threshold (e.g., 0.5) to maximise F1. That threshold is optimal for your training data's base rate. Deployed to a different base rate, the same threshold is suboptimal — and worse, your probability estimates are still miscalibrated.
Detection: reliability diagrams
A reliability diagram plots predicted probability vs observed frequency:
Production signal: your reliability diagram shows that predictions in the 0.8–0.9 bin actually have 0.4–0.5 positive rate. The model is massively overconfident in this range.
Three forms of miscalibration and their fixes:
1. Confidence miscalibration (model too confident)
The model's high-confidence predictions occur less frequently than predicted. Scores of 0.9 correspond to actual rates of 0.5.
Fix: Platt scaling. Train a logistic regression: targets = true labels, features = model scores. The fitted logistic curve rescales the model's scores to match reality.
```python from sklearn.calibration import CalibratedClassifierCV calibrated_model = CalibratedClassifierCV(model, cv='precomputed', method='sigmoid') calibrated_model.fit(X_val, y_val) ```
Cost: minimal. One additional logistic regression fit on a validation set.
2. Systematic base-rate shift (deployed to different segment)
Model trained on 2% base rate, deployed to 20% base rate. Same scores, very different calibration.
Fix: re-fit a calibration curve on data from the target segment (if available). If labels are delayed, retrain the entire model on segment-representative data.
3. Probability output type mismatch (mixing different model families)
An ensemble mixes a neural network (sigmoid outputs) with LightGBM (raw scores). The ensemble aggregates them naively without scaling.
Fix: calibrate each model separately to [0, 1] before ensembling. Use isotonic regression (more flexible than Platt scaling) if you have enough calibration data (500+ examples).
```python from sklearn.calibration import IsotonicRegression iso_reg = IsotonicRegression(out_of_bounds='clip') iso_reg.fit(lgbm_scores, y_val) lgbm_probs = iso_reg.transform(lgbm_scores) ```
The production checkpoint:
Before shipping a model: 1. Generate a reliability diagram on a validation set 2. For each decile of predicted probability, verify that actual positive rate ≈ predicted probability 3. If any decile has actual rate that diverges > 0.1 from predicted, apply Platt scaling or isotonic regression 4. Recompute the reliability diagram after calibration — all deciles should align with the diagonal 5. Ship the calibrated model
In production:
1. Log predictions and outcomes (with label delay) 2. Monthly: recompute reliability diagram on recent data with available labels 3. If calibration has drifted (decile actual rates diverge from predicted), retrain the calibration curve or retrain the full model
Why this is critical downstream:
If your fraud model outputs uncalibrated probabilities (0.8 when true probability is 0.3), downstream systems break:
Calibration is invisible to AUC. It's visible to precision, to downstream decision rules, and to any system that uses probabilities rather than rankings.