Calibration & Brier Score

Reliability diagrams, Platt scaling, isotonic regression, Brier decomposition

A model can rank perfectly and still lie to you. Picture a mortality-risk model with a flawless AUC of 1.0 — it always ranks sicker patients above healthier ones. A doctor reads "90% risk" off it and prepares for the worst. But among all the patients this model stamps "90%," only 60% actually die. The *ranking* is perfect; the *number* is badly wrong. And the moment a real decision hangs on that number — a treatment, a price, an alert threshold — the wrong number costs you.

That gap has a name. Calibration asks: when the model says 0.9, does the thing happen 90% of the time? It is a completely separate question from *ranking* (which is what AUC measures). A model can nail one and fail the other.

Seeing it: the reliability diagram

Bucket the model's predictions (everything near 0.1, near 0.2, and so on) and, for each bucket, plot the predicted probability against the *actual* fraction that came true. A perfectly calibrated model sits on the diagonal where "predicted = actual." Points that fall *below* the diagonal mean overconfidence — the model says 0.8 for things that happen only 0.55 of the time. Modern neural networks are almost always overconfident, and it gets worse toward the high end, because they are trained with a loss that rewards confident predictions even on noisy labels.

Putting a number on it: ECE and Brier

Two numbers summarise calibration. ECE (expected calibration error) is just the average gap between those buckets and the diagonal — lower is better, zero is perfect. It is handy but coarse: a low ECE can still hide bad miscalibration in one probability range, so always look at the diagram too, not only the single number.

The Brier score goes further. It is simply the mean squared error of the probabilities — the average of (predicted − actual)², where actual is 0 or 1. Lower is better. What makes it special is that it rolls *both* things you care about into one number: how well the model *separates* the classes (discrimination) *and* how honest its probabilities are (calibration). So a model with great AUC but poor calibration shows a good discrimination part and a bad calibration part inside its Brier score — which is exactly why a model with *lower* AUC can still have the *lower* (better) Brier score, by paying its way in calibration.

Fixing it — on a separate calibration set

You do not usually retrain; you patch the probabilities afterward, using a held-out calibration set (never the training or test data). Three common tools, cheapest first. Temperature scaling (for neural nets): divide the logits by a single number T before the softmax — T > 1 softens overconfident outputs, and it never changes which class wins, only the probabilities. One parameter, impossible to overfit, and it fixes the most common kind of neural-net miscalibration, so try it first. Platt scaling: fit a small logistic curve on top of the scores — good for the smooth, one-directional miscalibration of SVMs and boosting, and it works with little data. Isotonic regression: fit a flexible staircase that can straighten any shape of miscalibration — more powerful, but it needs more data or it just memorises. And the rule you cannot break: fit the correction on a *separate* slice. Calibrate on training data and it is fooled by memorised outputs; calibrate on test data and you have spoiled your only honest score.

The Brier decomposition, by its actual terms

We said Brier folds two things together — here they are precisely. The Murphy decomposition splits it into three: Brier = reliability − resolution + uncertainty. Reliability is the calibration error (how far predictions sit from the true rate in their bucket) — you want it *low*. Resolution is how much the predictions vary from the base rate and correctly separate outcomes (discrimination) — you want it *high*, and it's *subtracted*. Uncertainty is the irreducible difficulty set by the base rate, which no model controls. The critical consequence: a lower Brier does not always mean better calibration — it can drop purely because *resolution* improved (better discrimination) while calibration stayed the same or worsened. So don't read Brier as a pure calibration metric; it mixes calibration, discrimination, and base-rate difficulty. sklearn warns about exactly this.

ECE's fine print

ECE is more fragile than its single number suggests. It depends heavily on bin count (10 bins vs 20 gives different ECE) and bin placement; empty or tiny bins make the estimate noisy; equal-width bins waste resolution when predictions cluster (fixable with adaptive/equal-count binning). And in multiclass you must choose *what* to measure: top-label ECE (is the confidence in the predicted class honest?) versus classwise ECE (is every class's probability honest?) — they answer different questions and can disagree. Always pair ECE with the reliability diagram so a small number can't hide localised miscalibration.

Calibrating more than two classes

Multiclass calibration is genuinely harder. The one-vs-rest approach fits one calibrator per class, but then the per-class probabilities no longer sum to 1 and need renormalising. You also decide between calibrating the top label only versus classwise (every class), and you inspect classwise reliability diagrams rather than one curve. Temperature scaling is popular here precisely because scaling all logits by a single T keeps the softmax normalised and sidesteps the renormalisation headache.

Model-specific miscalibration patterns

Different models miscalibrate in characteristic directions. Random forests are pushed *away* from 0 and 1 by tree-averaging (a truly-positive case rarely gets every tree to vote yes), giving a sigmoid-shaped reliability curve — under-confident at the extremes. Modern neural networks are commonly overconfident (curve sags below the diagonal), though this is a tendency, not a universal law — it depends on architecture, loss, and regularisation. Knowing your model's usual distortion tells you which correction shape to expect.

Calibration decays after deployment — monitor it

Calibration measured at launch is not permanent. Covariate shift (the input mix moves) and base-rate shift (the positive rate changes) both break it even when the original test calibration looked perfect. So calibration is something to monitor in production — track ECE/reliability over time and, crucially, *by cohort*: time window, geography, device, and segment, since a model well-calibrated overall can be badly off for a subgroup whose prevalence shifted.

Calibration is not thresholding

Keep the two steps separate. Calibration makes the probability *truthful* (0.7 means 70%). Thresholding picks the *decision cutoff* that turns a probability into an action, chosen from costs and capacity. They're related — an honest probability makes the threshold meaningful and transferable across contexts — but distinct: you calibrate so the number can be trusted, *then* threshold so the decision is optimal. Fixing one does not fix the other.

Key points

• Calibration and ranking are different things — a model can ace one and fail the other. Ranking (AUC) asks whether riskier cases score higher than safer ones. Calibration asks whether "0.9" actually happens 90% of the time. A model that always predicts the base rate is perfectly calibrated yet useless; a model with AUC 1.0 can be 30 points overconfident. You need both, and which matters more depends on whether a real decision reads the probability itself (medical risk, pricing, fraud thresholds, or feeding another model) or only the order (top-k ranking, where calibration barely matters).
• See miscalibration with a reliability diagram; summarise it with ECE or Brier. Bucket the predictions and plot predicted probability against the actual rate — points below the diagonal mean overconfidence. ECE is the average distance from that diagonal (simple, but it can hide trouble in one range, so look at the diagram too). The Brier score — mean squared error of the probabilities — is richer: it folds discrimination and calibration into one number, which is why a model with lower AUC can still post the better (lower) Brier score by being better calibrated.
• The fix is a post-hoc patch on a separate calibration set — start with the simplest. Temperature scaling (divide neural-net logits by one number T) is the first thing to try: it fixes typical overconfidence, cannot overfit, and never changes which class wins. Platt scaling (a small logistic curve on the scores) suits the smooth miscalibration of SVMs and boosting. Isotonic regression can fix any shape but needs more data. Whichever you pick, fit it on a held-out calibration slice — never on training or test.
• Read Brier and ECE carefully — neither is a pure calibration number. Brier = reliability − resolution + uncertainty, so a lower Brier can come entirely from better discrimination (resolution) while calibration is flat or worse — check the reliability term or diagram directly. ECE is binning-sensitive (bin count, empty bins; use adaptive/equal-count bins) and in multiclass you must choose top-label vs classwise ECE, which can disagree. Random forests miscalibrate *away* from 0/1 (sigmoid curve, under-confident at extremes) while neural nets are commonly — not universally — overconfident.
• Handle multiclass, monitor for drift, and keep calibration separate from thresholding. One-vs-rest multiclass calibration needs renormalising (per-class probabilities won't sum to 1); temperature scaling avoids this by scaling all logits together. Calibration decays under covariate and base-rate shift even when AUC is unchanged, so monitor ECE/reliability over time and by cohort (geography, device, segment) and re-fit on recent data. And calibration (making the probability truthful) is a different step from thresholding (choosing the decision cutoff from costs) — fixing one doesn't fix the other.

Recap

• Ranking and calibration are independent — a model can ace one and fail the other: an AUC-1.0 model ranks every sicker patient above every healthier one, yet among the patients it stamps "90%," only 60% may actually die. AUC asks whether risky cases score higher; calibration asks whether "0.9" happens 90% of the time. The moment a real decision reads the *number* (treatment, price, threshold), the wrong number costs you.
• See miscalibration with a reliability diagram: bucket predictions and plot predicted probability against the *actual* fraction that came true — a perfectly calibrated model sits on the diagonal, and points *below* it mean overconfidence (says 0.8 for things that happen 0.55 of the time). Modern neural nets are almost always overconfident, worse toward the high end.
• Summarise with ECE and Brier: ECE = average gap from the diagonal (simple but coarse — a low ECE can hide bad miscalibration in one range). Brier = mean squared error of the probabilities, which folds *both* discrimination and calibration into one number — which is why a model with *lower* AUC can post the *lower* (better) Brier by being better calibrated.
• Fix with a post-hoc patch on a *separate* calibration set, simplest first: temperature scaling (divide neural-net logits by one T, T>1 softens overconfidence, never changes which class wins, can't overfit — try first) → Platt scaling (a small logistic curve, good for SVM/boosting) → isotonic regression (any shape, but needs more data). Never fit the correction on training or test data.
• Brier = reliability − resolution + uncertainty, so a lower Brier isn't proof of better calibration: reliability is the calibration error (want low), resolution is discrimination (want high, and it's subtracted), uncertainty is irreducible base-rate difficulty. Brier can drop purely from better resolution while calibration stays flat or worsens — check the reliability term or the diagram directly.
• ECE is binning-sensitive: bin count and placement change it, empty/tiny bins make it noisy, and equal-width bins waste resolution when predictions cluster (use adaptive/equal-count bins). In multiclass, choose top-label ECE (is the predicted class's confidence honest?) vs classwise ECE (is every class honest?) — they can disagree, so always pair ECE with the diagram.
• Calibration decays after deployment even when AUC is stable: covariate shift and base-rate shift both break it because the score→true-rate mapping depends on the distribution. Monitor ECE/reliability over time and *by cohort* (geography, device, segment) and re-fit on recent data. And calibration (making the probability truthful) is a separate step from thresholding (choosing the decision cutoff from costs) — fixing one doesn't fix the other.

Check your understanding