Evaluation Metrics: Precision, Recall, AUC-ROC, and AUC-PR From First Principles
Accuracy is almost never the right metric. Understanding which metric to use — and why — requires understanding the confusion matrix, the precision-recall trade-off, and what AUC-ROC actually measures geometrically (a probability, not just an area). This post builds every classification metric from the confusion matrix up.
Choosing the wrong evaluation metric is one of the most common production ML mistakes. A fraud detection model with 99.9% accuracy that predicts "not fraud" for every transaction is useless — but optimising accuracy would rate it highly. The right metric depends on the cost structure of your specific problem.
The confusion matrix
Binary classification with a threshold τ: predict positive if P(y=1|x) ≥ τ. For each test sample, one of four outcomes: True Positive (TP): predicted positive, actually positive. False Positive (FP): predicted positive, actually negative. Type I error. False Negative (FN): predicted negative, actually positive. Type II error. True Negative (TN): predicted negative, actually negative. The confusion matrix arranges these as a 2×2 grid. From these four numbers, every classification metric is derived.
The basic metrics
Accuracy = (TP + TN) / (TP + FP + FN + TN). Fraction correctly classified. Fails catastrophically with class imbalance. Precision = TP / (TP + FP). Of all predicted positives, what fraction are truly positive? Also called Positive Predictive Value (PPV). Measures: when the model raises an alarm, how often is it right? Recall = TP / (TP + FN). Of all true positives, what fraction did the model catch? Also called Sensitivity or True Positive Rate (TPR). Measures: what fraction of the real positives did we find? Specificity = TN / (TN + FP). Of all true negatives, what fraction did the model correctly label? = 1 - FPR. False Positive Rate = FP / (FP + TN). Fraction of negatives incorrectly labeled as positives.
The precision-recall trade-off
Precision and recall trade off against each other as you change the classification threshold τ. Raise τ: model predicts positive only when very confident → fewer false positives → higher precision, lower recall. Lower τ: model predicts positive more aggressively → catches more true positives → higher recall, more false positives → lower precision. There is no universally correct threshold — it depends on the costs. For fraud detection: false negatives (missed fraud) are expensive → lower threshold, higher recall. For medical screening follow-up (expensive test): false positives (unnecessary procedures) are costly → higher threshold, higher precision.
F1 score: the harmonic mean of precision and recall
F1 = 2 × (Precision × Recall) / (Precision + Recall) = 2TP / (2TP + FP + FN). Why harmonic mean, not arithmetic? The harmonic mean is dominated by whichever term is smaller. A model with precision=1.0 and recall=0.01 has arithmetic mean = 0.505 (looks decent) but harmonic mean = 0.02 (correctly reflects that the model misses 99% of positives). F1 penalises extreme imbalances between precision and recall. Fβ generalises F1: Fβ = (1+β²) × P × R / (β²P + R). β > 1 weights recall more (catching cases matters more); β < 1 weights precision more.
ROC curve and AUC-ROC
The ROC (Receiver Operating Characteristic) curve plots TPR (recall) vs FPR as the threshold τ varies from 1 to 0. Every point on the curve corresponds to one threshold value. AUC-ROC is the area under this curve, ranging from 0.5 (random classifier, diagonal line) to 1.0 (perfect classifier). Probabilistic interpretation: AUC-ROC = P(score(positive) > score(negative)) for a randomly drawn positive-negative pair. A model with AUC = 0.85 ranks a random positive above a random negative 85% of the time. This interpretation is threshold-free — AUC measures the quality of the ranking, not a specific threshold decision. AUC-ROC is insensitive to class imbalance: because FPR divides by total negatives and TPR divides by total positives, a large number of negatives doesn't inflate either axis. Failure case: when the positive class is very rare, FPR stays tiny even with many false positives — the curve looks optimistic.
Precision-Recall curve and AUC-PR
The PR curve plots precision vs recall as τ varies. AUC-PR (area under the PR curve) is more informative than AUC-ROC when the positive class is rare (< 5% prevalence). At low recall (high threshold): precision ≈ 1 (only very confident positives predicted). As recall increases, more true positives are caught but precision generally falls. A random classifier on a 1:99 imbalanced dataset has AUC-ROC ≈ 0.5 but AUC-PR ≈ 0.01 (only 1% of random predictions would be correct). This starkly shows PR's sensitivity to imbalance. Use AUC-PR for: fraud detection, rare disease detection, spam filtering — any case where the positive class is rare and false positives and false negatives both matter.
Multi-class metrics
Macro average: compute the metric for each class, then average equally across classes. Treats all classes equally regardless of support. Micro average: aggregate TP, FP, FN across all classes first, then compute. Gives larger classes more influence. Weighted average: weight each class's metric by its support (number of true instances). Appropriate when you want accuracy-weighted performance. For imbalanced multi-class problems: use macro F1 to equally weight minority classes.
Interview questions on this topic
"When would you prefer AUC-PR over AUC-ROC?" — When the positive class is rare (< 5%). AUC-ROC can be high (0.9+) even for models that perform poorly on the positive class, because the denominator of FPR (total negatives) is large — a large number of false positives is still a small fraction. AUC-PR directly measures how well the model performs on the positive class.
"Your model has high recall but low precision. What does this mean and when is it acceptable?" — The model catches most true positives but also raises many false alarms. Acceptable when the cost of a false negative (missing a true positive) greatly exceeds the cost of a false positive. Example: cancer screening — missing a cancer is far worse than a follow-up test. Not acceptable when false positives are costly: spam filtering (too many legitimate emails marked as spam damages trust).
"Explain AUC-ROC as a probability." — AUC-ROC = P(model scores a randomly drawn positive sample higher than a randomly drawn negative sample). A model with AUC = 0.75 correctly orders 75% of positive-negative pairs. This interpretation comes from the equivalence between the ROC AUC and the Wilcoxon-Mann-Whitney statistic.
"You have 95% negatives and 5% positives. Your model has 98% accuracy. Is this good?" — No. A model that always predicts negative achieves 95% accuracy. The 98% accuracy model may only be marginally better. Check recall on the positive class — if it is near zero, the model is useless for the minority class. Use AUC-PR or F1 on the positive class as the primary metric.
Try on Colab: use the Credit Card Fraud dataset (Kaggle) — 0.17% positive rate. Train a logistic regression. Compute AUC-ROC and AUC-PR. Plot both curves. Then plot the precision-recall curve for threshold values from 0.01 to 0.99 and identify the threshold that maximises F1. Compare: does AUC-ROC overstate model quality relative to AUC-PR?