Eval Metrics from First Principles

Confusion matrix, precision, recall, F1, why accuracy fails

Imagine you build a fraud detector. Your dataset has 10,000 transactions, and 100 of them are fraud — just 1%. You train a model and it scores 99% accuracy. The team celebrates. Then you look closer: the model predicted "not fraud" for every single transaction. It caught zero frauds. The accuracy number was lying to you the whole time.

That is the trap accuracy sets on imbalanced data. It blends four very different outcomes into one number that the majority class dominates. To actually understand a classifier, you have to pull those four outcomes apart.

The four outcomes: the confusion matrix

Every prediction lands in one of four boxes. TP (true positive): correctly caught fraud. FP (false positive): flagged a legit transaction — a false alarm, a customer's card blocked for nothing. FN (false negative): missed a fraud — it goes through, the bank eats the loss. TN (true negative): correctly cleared a legit transaction.

The whole point is that these four do *not* cost the same. A false alarm annoys a customer; a missed fraud loses real money. Accuracy pretends they are interchangeable. They never are.

Two numbers that actually mean something

From those four boxes come two ratios, each measuring a different way to fail.

Precision = TP / (TP + FP): when the model shouts "fraud!", how often is it right? Low precision means you are drowning your operations team in false alarms.

Recall = TP / (TP + FN): of all the real frauds, how many did you actually catch? Low recall means fraud is slipping past you.

And here is the tension that runs through all of classification: these two pull against each other. Make the model more cautious (raise the threshold) and precision climbs while recall falls — fewer false alarms, but more fraud missed. Loosen it and the reverse happens. This is not a flaw in the metrics; it is a real business trade-off, and where you land depends entirely on what each mistake costs you.

Combining them — and doing it honestly

People often reach for F1, which blends precision and recall into one number: $F1 = 2 \cdot P \cdot R / (P + R)$. It uses the *harmonic* mean on purpose, because that punishes lopsided scores — a model with precision 0.9 and recall 0.1 gets an F1 of 0.18, not a flattering 0.5.

But F1 makes a silent assumption: that precision and recall matter equally. If a missed fraud costs ten times more than a false alarm, they do not. F-beta lets you tilt the balance: $F_β = (1 + β^2) \cdot P \cdot R / (β^2 \cdot P + R)$, where β > 1 weights recall more and β < 1 weights precision more. The right β comes from your costs, not from habit.

So the real first step is not choosing a metric — it is writing down what each mistake costs. In fraud, a missed fraud (FN) usually dwarfs a false alarm (FP), so you lean toward recall. In spam filtering it flips: a real email lost to the spam folder (FP) destroys trust, so you lean toward precision. In cancer screening, a missed tumor (FN) is catastrophic, so recall rules. Translate TP, FP, FN, TN into money or risk first. Every metric choice follows from that.

The rest of the confusion-matrix vocabulary

Precision and recall look at the *positive* column, but interviewers expect the full set. Recall is also called sensitivity or the true-positive rate (TPR). Its mirror on the negative side is specificity, or the true-negative rate (TNR) = TN / (TN + FP) — of all the truly-legit transactions, how many did you correctly clear? One minus specificity is the false-positive rate (FPR), sometimes called fallout — the share of legit transactions you wrongly flagged. And false-negative rate (FNR) = FN / (TP + FN) = 1 − recall. Two rules of thumb: recall/TPR and FPR are the axes of the ROC curve, and specificity is the metric to quote when *correctly clearing negatives* is what matters (e.g. a screening test you don't want firing on healthy people).

More than two classes: macro, micro, weighted

With several classes you compute precision/recall/F1 per class, then average them — and *how* you average changes the story. Macro averages the per-class scores equally, so a tiny class counts as much as a huge one — use it when rare classes matter. Micro pools all the TP/FP/FN across classes first and then computes one score, so it's dominated by the frequent classes and equals overall accuracy for single-label problems — use it when every *instance* matters equally. Weighted averages the per-class scores weighted by class size, a middle ground. A big macro-vs-micro gap is a signal: macro high, micro low means the model nails small classes but stumbles on the big one (or vice versa).

Single numbers that survive imbalance

F1 ignores true negatives entirely, which is why two better summaries exist for skewed data. Balanced accuracy is the average of recall across classes — it doesn't reward always-predict-majority. In the binary case specifically, that average is just the mean of sensitivity and specificity, since specificity is recall on the negative class; with more than two classes there's no single "specificity" to pair it with, so it stays the plain average of per-class recall. Matthews correlation coefficient (MCC) uses all four boxes of the confusion matrix in one correlation-style score from −1 to +1, and is widely regarded as the most honest single number under imbalance because a model can't fake it by ignoring a class. When you need one number and the classes are skewed, prefer balanced accuracy or MCC over raw accuracy or F1.

When you can only act on the top K

Often the real constraint is capacity, not a threshold: a fraud team reviews the top 500 alerts, a search page shows 10 results. Then the metric is precision@K (of the top K ranked by score, how many are truly positive) and recall@K (of all positives, how many landed in the top K). The model only has to get the worst cases to the *top of the list* — a global threshold is the wrong framing when the action budget is fixed.

Curves beat single thresholds: ROC-AUC vs PR-AUC

Precision and recall are measured *at one threshold*; to summarise a model across all thresholds you use an area-under-curve. ROC-AUC plots TPR against FPR — but on rare-positive problems it can look flatteringly high, because a huge TN count keeps FPR tiny even when the model floods you with false positives relative to the few real positives. PR-AUC (precision vs recall) ignores true negatives and so exposes that failure. Rough heuristic: when positives are scarce, PR-AUC is usually the more honest summary — though it's a heuristic, not a law (PR-AUC has its own quirks under shifting prevalence). This section only covers what you need to choose between them as a summary metric — how each curve is actually built and read threshold-by-threshold is where the next module, ROC Curve & AUC, picks up.

Pick the threshold on validation, freeze it, then report on test

One discipline ties it together. The decision threshold is a *parameter you tune*, so tune it on the validation set — sweep thresholds, pick the one matching your cost/precision/recall target — then freeze it and report final performance on the test set *once*. Tuning the threshold on the test set is the same sin as tuning weights on it: the reported number becomes optimistic and won't hold in production.

Key points

• When to use it: always define your cost matrix before picking a metric. In fraud detection: a missed fraud (FN) costs the full transaction amount plus investigation time. A false alarm (FP) costs a customer service call and customer inconvenience. If FN costs 10x more than FP, optimize for recall ($F_\beta$ with $\beta = 2$ or higher). In spam filtering, FP (blocking a real email) is the catastrophic failure — optimize for precision ($\beta < 1$). Accuracy is appropriate only when classes are roughly balanced and all errors cost the same, which is rarely true in production.
• The most common production trap: reporting F1 without asking whether FP and FN cost the same. F1 weights precision and recall equally. On a fraud model where missing fraud costs 50x more than a false alarm, optimizing F1 will under-weight recall and leave real money on the table. Before training, write down: what does a FP cost? What does a FN cost? As a rough rule of thumb, once those costs diverge by a large margin, F1 is probably the wrong metric — but there's no precise universal cutoff, so don't treat any single ratio as a bright line. $F_\beta$ (with $\beta$ larger when recall matters more) tilts the balance, but treat it as a rough proxy — the cleaner tool when you have real costs is to minimise expected cost directly by choosing the threshold that minimises $FP \cdot cost_{FP} + FN \cdot cost_{FN}$, rather than encoding the ratio into a single $\beta$.
• The diagnostic: when your model looks suspiciously good, check whether it is predicting the majority class. The tell: high accuracy, recall near 0. Compute recall separately. If recall ≈ 0 on a 1% fraud dataset, the model learned to predict "not fraud" for everything and achieved 99% accuracy by doing nothing. Also check: if F1 is decent but all FPs come from the same subgroup, you have a slice-level failure the aggregate metric is hiding. Disaggregate by segment before declaring the model ready.
• Know the full vocabulary and the right single number under imbalance. Recall = sensitivity = TPR; specificity = TNR = TN/(TN+FP); FPR (fallout) = 1 − specificity; FNR = 1 − recall — TPR and FPR are the ROC axes. For multiclass, macro averages classes equally (rare classes count), micro pools instances first (frequent classes dominate, equals accuracy for single-label), weighted sits between. And since F1 ignores true negatives, prefer balanced accuracy (mean of per-class recall) or MCC (uses all four cells, −1 to +1) as the honest single summary on skewed data.
• Match the summary to how you'll act, and tune the threshold on validation only. When action is capacity-limited (review top 500, show top 10), optimise precision@K / recall@K — the model just needs the worst cases at the top. To compare models across thresholds use an AUC, and prefer PR-AUC to ROC-AUC when positives are rare (ROC-AUC's huge TN count hides false-positive floods) — a heuristic, not a law. Critically, the decision threshold is a tuned parameter: sweep it on the validation set, freeze it, then report on test once. Tuning the threshold on test inflates the number exactly like tuning weights on test.

Recap

• Accuracy lies on imbalanced data: a model that predicts "not fraud" for every transaction scores 99% accuracy on a 1%-fraud dataset while catching zero frauds (recall 0). Accuracy blends four different outcomes into one number the majority class dominates — to understand a classifier you must pull those four apart.
• The confusion matrix is TP / FP / FN / TN, and the four don't cost the same: TP = caught fraud, FP = false alarm (a customer's card blocked for nothing), FN = missed fraud (the bank eats the loss), TN = correctly cleared. A false alarm annoys; a missed fraud loses real money — accuracy pretends they're interchangeable.
• Precision and recall each measure a different way to fail: precision = TP/(TP+FP) = when the model shouts "fraud," how often is it right (low → drowning in false alarms); recall = TP/(TP+FN) = of all real frauds, how many you caught (low → fraud slipping past). They trade off via the threshold — raise it and precision climbs while recall falls.
• F1 blends them but assumes they matter equally: F1 = 2PR/(P+R), the *harmonic* mean, which punishes lopsided scores (P=0.9, R=0.1 → F1=0.18, not 0.5). F-beta tilts by cost — $F_\beta = (1+\beta^2)PR/(\beta^2 P + R)$, where β>1 favours recall, β<1 favours precision.
• Write the cost matrix first — every metric choice follows from it: translate TP/FP/FN/TN into money or risk before picking a metric. Fraud: a missed fraud dwarfs a false alarm → lean recall. Spam: a lost real email destroys trust → lean precision. Skipping this step is how teams optimize the wrong number for months.
• Know the full vocabulary: recall = sensitivity = TPR; specificity = TNR = TN/(TN+FP); FPR (fallout) = 1 − specificity; FNR = 1 − recall. TPR and FPR are the ROC axes; quote specificity when correctly clearing negatives is what matters (a screening test you don't want firing on healthy people).
• Under imbalance prefer balanced accuracy or MCC over F1 (F1 ignores true negatives; MCC uses all four cells, −1 to +1, and can't be faked by ignoring a class). Use precision@K / recall@K when action is capacity-limited (review top 500, show top 10). And tune the threshold on validation, freeze it, then report on test once — tuning it on test inflates the number exactly like tuning weights on test.

Check your understanding