Model Evaluation · ML Systems Lab

AUC Is Not Your Friend: A Guide to ML Metric Selection

ROC-AUC is the default metric for classification problems. It's in every sklearn tutorial and every Kaggle competition leaderboard. It's also the wrong metric for most real production ML problems. Here's why — and what to use instead.

ROC-AUC measures rank ordering: given a random positive and a random negative, what's the probability your model scores the positive higher? This is a useful property. It's also completely divorced from what most production ML systems actually need.

The class imbalance problem.

ROC-AUC uses True Positive Rate (recall) and False Positive Rate (FPR). FPR = FP / (FP + TN). With 1% positive class, you have 99× more negatives than positives. A model that correctly identifies 80% of positives but has a 10% FPR looks great on ROC-AUC (0.85+). But at 1% prevalence, that 10% FPR means for every true positive you catch, you generate ~12 false positives. In fraud detection, that's 12 legitimate transactions blocked for every fraud caught. Your fraud ops team will revolt.

PR-AUC fixes this. Precision-Recall AUC is dominated by performance on the positive class. A high PR-AUC requires both high precision (few false positives) and high recall (few false negatives) — there's no "free" performance from classifying negatives correctly.

When to use what:

Balanced classes, rank ordering matters → ROC-AUC

Imbalanced classes, minority class is the target → PR-AUC

Threshold-based decisions, single operating point → F1 or F-beta

Probability outputs used downstream → Brier score + calibration

Business metric is revenue/cost → custom loss or direct business metric

The threshold problem.

F1, precision, recall — all computed at a fixed threshold. Move the threshold and your metrics change. ROC-AUC and PR-AUC are threshold-independent. But ultimately, your model operates at a threshold in production. Report threshold-independent metrics for comparison, but always report what the model does at your actual operating threshold.

The calibration problem.

A model can have 0.92 AUC and be completely uncalibrated. Uncalibrated means: when the model says P(fraud) = 0.8, the actual frequency of fraud in those predictions might be 0.3. This matters every time you use the score as a probability — risk scoring, expected value calculations, multi-model ensembles. Check with reliability diagrams. Fix with Platt scaling or isotonic regression.

Continue interactively

Read this post inside ML Systems Lab — with Simplify toggle, interview Q&As, inline glossary, and the MLE Path forward pointer.

Open in MSL →