ROC Curve & AUC

FPR/TPR, what area means, PR-AUC for imbalanced classes

Here is the problem with reporting one number for a classifier: it hides everything. Take that same fraud detector. At a threshold of 0.5 it gets precision 80%, recall 60%. Drop the threshold to 0.3 and precision falls to 65% while recall climbs to 80%. Raise it to 0.7 and you get precision 90%, recall 40%. The model's quality is not a single number — it depends entirely on where you draw the line. So how do you compare two models *before* committing to a threshold?

The ROC curve: try every threshold at once

The ROC curve answers exactly that. For every possible threshold, compute two numbers: the TPR (true positive rate, also called recall — the fraction of real frauds you catch) and the FPR (false positive rate — the fraction of legit transactions you wrongly flag). Plot TPR up the side and FPR along the bottom. As you lower the threshold you catch more fraud (TPR rises) but also raise more false alarms (FPR rises), and the curve traces that whole trade-off in one picture.

A model that guesses randomly gives the diagonal line (TPR always equals FPR), so its area under the curve — AUC — is 0.5. A perfect model hugs the top-left corner (catch everything, flag nothing), so its AUC is 1.0. The AUC is a single score for the model across *all* thresholds at once.

What AUC actually means

There is a lovely second reading of AUC that needs no curve at all. AUC is exactly the probability that the model scores a random real fraud higher than a random legit transaction. So AUC = 0.91 means: pick any one fraud and any one legit transaction at random, and the model ranks the fraud above the legit one 91% of the time. That is all AUC measures — how well the model *ranks* positives above negatives, averaged over every possible pair. (That is also why it ignores your threshold: it is a pure ranking score. Statisticians know it as the normalized Mann-Whitney U statistic. Concretely: line up every possible (fraud, legit) pair — a pair is concordant if the model scores the fraud higher — and AUC is exactly the fraction of all pairs that are concordant.)

Where it quietly lies: rare positives

Now the catch. FPR is FP / (FP + TN), and on imbalanced data that TN is enormous — 9,900 legit transactions for every 100 frauds. So even 500 false alarms give FPR = 500 / 9,900 ≈ 0.05, which looks tiny. The ROC curve sits comfortably in the top-left, AUC = 0.91, everyone is happy. But look at precision: assuming this scenario catches all 100 frauds (100% recall here), precision is 100 / (100 + 500) = 0.17. Five out of every six alerts your fraud team chases are wrong.

The fix is the precision-recall (PR) curve, which plots precision against recall and never touches TN at all. When positives are rare, it is the honest picture — a model that looks production-ready on ROC-AUC can be exposed as a false-alarm machine on PR-AUC. Rule of thumb: if your positive class is under about 10% of the data, use PR-AUC, not ROC-AUC.

One last thing to hold onto: AUC of either kind is a *threshold-independent* summary, so it tells you nothing about the specific cutoff you will actually run. AUC is for *choosing the model*; setting the threshold is a separate business decision driven by your cost matrix. Pick the model with AUC; pick the operating point with costs.

Partial AUC: sometimes only one corner matters

Full AUC averages ranking quality over *every* threshold — including regions you'd never operate in. In fraud or medical screening you only ever run at very low FPR (you cannot flag 40% of legit traffic), so ranking performance in the high-FPR region is irrelevant, yet full AUC rewards it. Partial AUC restricts the area to the FPR range you actually care about (say FPR < 0.05), giving a score that reflects the operating region instead of a whole-curve average. When two models tie on full AUC, partial AUC in your real operating band often separates them.

AUC says nothing about calibration

A crucial blind spot: AUC is a *pure ranking* score, so a model can have a superb AUC and badly wrong probabilities. Multiply every predicted probability by 0.5 and the ranking — and therefore the AUC — is unchanged, but every probability is now a lie. So if you use the probability itself (pricing, expected value, a downstream model), AUC is not enough; check calibration separately with a reliability diagram and the Brier score. High AUC, good calibration is what "trustworthy probabilities" requires.

When ROC curves cross, one AUC hides two stories

AUC collapses a whole curve to one number, so two models with the *same* AUC can have crossing ROC curves — model A better in the low-FPR region, model B better in the high-recall region. The single AUC averages that away. If your operating point is low-FPR, you want model A even if its total AUC is slightly lower. Always look at the curves in your operating band, not just the scalar.

Average precision is not exactly trapezoidal PR-AUC

A subtle library gotcha: sklearn's `average_precision_score` and `auc(recall, precision)` are *not* the same number. Average precision (AP) is a weighted mean of precision values, weighted by the increase in recall at each threshold — a step-wise summary that avoids the optimistic interpolation that trapezoidal area under the PR curve can introduce. When someone reports "PR-AUC," check whether they mean AP (usually what sklearn gives) or trapezoidal area; the two can differ meaningfully on small data.

Precision moves with prevalence — the formula

This is why the same model can look fine offline and terrible in production. Precision is tied to the base rate: with prevalence π, TPR, and FPR,

$\text{precision} = \dfrac{\pi \cdot TPR}{\pi \cdot TPR + (1-\pi)\cdot FPR}$

The ROC curve (TPR vs FPR) doesn't change when prevalence shifts — but precision does, dropping as positives get rarer. So a model validated at 5% fraud can post far worse precision when live fraud falls to 1%, with identical ROC-AUC. Always recompute expected precision at the *production* base rate.

Multiclass AUC

AUC is binary by construction, so for K classes you extend it. One-vs-rest (OvR) computes each class's AUC against all others and averages (macro or weighted). One-vs-one (OvO) averages AUC over every pair of classes and is more robust to imbalance. sklearn's `roc_auc_score` supports both via `multi_class='ovr'/'ovo'`; name which one you used, since the averaging choice changes the number.

Key points

• ROC-AUC or PR-AUC? It comes down to whether the negatives are rare or common. Use ROC-AUC when the classes are roughly balanced, or when getting the negatives right genuinely matters (like credit scoring, where correctly approving good applicants counts, not just catching defaulters). Use PR-AUC when negatives massively outnumber positives — fraud, disease screening, anomaly detection — because there the ocean of true negatives inflates the FPR denominator and makes ROC-AUC look better than it really is. Rule of thumb: if your positive class is under about 10% of the data, reach for PR-AUC.
• The trap: shipping a model with a great AUC on rare-positive data, then watching the alert queue overflow. AUC = 0.91 on a 1%-fraud dataset can sit right next to precision = 0.17 at the threshold you actually deploy — the ROC curve looked fine only because true negatives flooded the FPR denominator. Always check precision at your intended recall before calling a model ready. If you need 80% recall and precision there is 15%, the model is not production-ready no matter what AUC says.
• The habit: AUC picks the model, the cost matrix picks the threshold. After comparing models by AUC or PR-AUC, choose your deployment threshold by plotting the precision-recall trade-off and finding the point your costs demand. A concrete check: at 80% recall, how many alerts per day does that produce? If your team can handle 200 and the model would fire 2,000, the threshold has to move regardless of AUC. AUC told you which model; the costs tell you where to run it.
• AUC is a ranking score — blind to calibration, blind to your operating region, and blind to prevalence shift. Scaling every probability by 0.5 leaves AUC unchanged but makes the probabilities lies, so check calibration (reliability diagram, Brier) separately when you use the probability itself. Full AUC averages over thresholds you'd never run — use partial AUC in your real FPR band, and inspect the curves directly since two models with equal AUC can have crossing ROC curves (one wins at low FPR, the other at high recall). And precision = π·TPR / (π·TPR + (1−π)·FPR): the ROC doesn't move with prevalence but precision does, so recompute expected precision at the production base rate.
• Mind the library and multiclass details. sklearn's average precision (a recall-weighted mean of precision) is not identical to trapezoidal PR-AUC and avoids its optimistic interpolation — so confirm which "PR-AUC" someone means. For K classes, AUC extends via one-vs-rest (each class vs the rest, averaged) or one-vs-one (every pair, more robust to imbalance); state which averaging you used because it changes the number. Treat the "positives < 10% → use PR-AUC" rule as a helpful heuristic, not a law — PR-AUC has its own quirks under shifting prevalence and isn't universally superior.

Recap

• The ROC curve tries every threshold at once: plot TPR (recall — fraction of frauds caught) up the side against FPR (fraction of legit wrongly flagged) along the bottom, sweeping all thresholds. Random guessing gives the diagonal (AUC 0.5); a perfect model hugs the top-left (AUC 1.0).
• AUC is a pure ranking score with a clean probabilistic meaning: AUC = P(model scores a random real positive above a random real negative). AUC 0.91 means it ranks a random fraud above a random legit transaction 91% of the time — which is why it ignores your threshold entirely. It equals the normalized Mann-Whitney U statistic.
• Rare positives quietly fool ROC-AUC: FPR = FP/(FP+TN), and on imbalanced data that TN is enormous (9,900 legit per 100 fraud), so even 500 false alarms give FPR ≈ 0.05 — the curve sits in the top-left, AUC 0.91 — while precision — assuming all 100 frauds are caught (100% recall in this scenario) — is only 100/(100+500) = 0.17. Five of every six alerts are wrong.
• When positives are under ~10%, use PR-AUC: precision-recall curve plots precision vs recall and never touches TN, so it exposes the false-alarm floods ROC-AUC hides. It's a heuristic, not a law — PR-AUC has its own quirks under shifting prevalence.
• AUC picks the model; the cost matrix picks the threshold: AUC is threshold-independent, so it says nothing about the specific cutoff you'll run. Choose the model by AUC, then set the operating point by plotting the precision-recall trade-off against your costs — at 80% recall, how many alerts/day does that make?
• AUC is blind to calibration: it's pure ranking, so multiply every predicted probability by 0.5 and the AUC is unchanged while every probability is now a lie. If you use the probability itself (pricing, expected value, a downstream model), check calibration separately (reliability diagram, Brier score).
• Precision moves with prevalence — the formula: precision = $\pi\cdot TPR / (\pi\cdot TPR + (1-\pi)\cdot FPR)$. The ROC (TPR vs FPR) doesn't move when prevalence shifts, but precision drops as positives get rarer — a model validated at 5% fraud posts worse precision when live fraud falls to 1%, at identical ROC-AUC. Use partial AUC in your operating FPR band, and state OvR vs OvO for multiclass since the averaging changes the number.

Check your understanding