Class Imbalance

SMOTE, threshold tuning, class weights, precision@K

The last module was about trusting a model's predicted probabilities — checking whether its batch of 0.7-confidence calls really come true about 70% of the time, and fixing it with Platt or isotonic scaling when they don't. Put that same trained model on an imbalanced problem and a second, nastier failure shows up before calibration even becomes the issue — one that hides in the headline number itself. Imagine you are building a fraud detector for a bank, and a thousand transactions come in: 950 legitimate, 50 fraud — a 5% positive rate, imbalanced but not exotically so. Here is a "model" that never looks at a single feature: label *every* transaction legitimate. Score it. True positives: 0. False positives: 0. False negatives: 50 (every real fraud, missed). True negatives: 950. Accuracy = (0+950)/1000 = 95.0% — and it is utterly worthless, because it never catches a single fraud, which was the whole point. This is the accuracy trap, and it makes class imbalance one of the sneakiest problems in machine learning: your headline number looks fantastic while the model does nothing useful.

Now score a real classifier trained on the same 1000 transactions, at the default 0.5 threshold (the cutoff score above which the model calls something fraud): it flags 70 transactions, and 35 of those are genuine fraud. True positives = 35, false positives = 35, false negatives = 15 (fraud it missed), true negatives = 915. Accuracy = (35+915)/1000 = 95.0% — *identical* to the do-nothing model, to the decimal point. Two models, same accuracy, and one of them is actually finding fraud. Accuracy cannot tell them apart, because it was never built to see the rare class at all.

Step one: stop measuring with accuracy

If accuracy lies, what do you look at instead? Two numbers that actually care about the rare class, scored on the same real classifier above:

Recall (also called sensitivity): of all the real frauds, what fraction did the model catch? Here, 35 of the 50 real frauds = 35/50 = 70%. The do-nothing model's recall is 0/50 = 0% — instantly exposed, where accuracy hid it completely.

Precision: of all the transactions the model flagged as fraud, what fraction really were? Here, 35 of the 70 flags = 35/70 = 50% — half the fraud team's alerts are false alarms. (The do-nothing model never flags anything, so its precision is undefined — 0 by convention — a second signal accuracy missed.)

F1, the harmonic mean of the two, folds them into one number: 2×(0.5×0.7)/(0.5+0.7) = 0.7/1.2 = 58.3% for the real classifier, versus 0% for the do-nothing one. F1 finally shows the gap that a 95.0%-vs-95.0% accuracy tie completely erased.

There is usually a tug-of-war between precision and recall — flag more aggressively and you catch more fraud (higher recall) but with more false alarms (lower precision). The PR-AUC (the precision-recall area) captures that whole tradeoff in one number, and it is far more honest than the more common ROC-AUC when the positive class is rare, because ROC-AUC hands the model easy credit for correctly ignoring the huge majority.

Step two: make the model care about the rare class — and watch what the fix actually costs.

By default, training treats every example as equally important, so the rare class gets drowned out — nineteen "legit" examples for every "fraud" shout it down. Two ways to fix that.

The simplest, and usually the first to try, is class weights: tell the loss function that getting a fraud example wrong costs roughly 19 times more than getting a legit one wrong (950:50, the actual ratio in this data) — no new data, just a reweighted loss. Pause and predict before the retrain: with fraud weighted 19× more heavily against a false alarm, do you expect precision to rise, fall, or stay about the same once the model is retrained under that weighting? Retrain the same classifier with that weighting, same threshold, and here's the real effect on this dataset: true positives rise to 45 (of 50), false negatives fall to 5, but false positives rise to 90 and true negatives fall to 860. Recall = 45/50 = 90%, up sharply from 70%. Precision = 45/(45+90) = 45/135 = 33.3%, down from 50%. F1 = 2×(0.333×0.9)/(0.333+0.9) = 0.6/1.233 ≈ 48.6% — actually *lower* than the unweighted classifier's 58.3%, even though recall improved. And accuracy? (45+860)/1000 = 90.5% — it went *down*. Every one of these numbers moved in a real direction: class weights bought +20 points of recall for −16.7 points of precision, at the cost of accuracy and even F1. Whether that trade is worth it depends entirely on whether a missed fraud actually costs more than 19× a false alarm — which is exactly the question the cost matrix below makes precise, because F1 alone can't answer it.

Alternatively you can rebalance the data itself: oversample the rare class (duplicate it, or with SMOTE, create synthetic in-between examples) or undersample the majority (throw some of it away). These help distance-based models and neural nets, but for tree-based models class weights are usually cleaner and reach a similar recall/precision trade.

Step three: choose the threshold on purpose

Here is the step almost everyone forgets — recall that both models scored above used 0.5 as that cutoff without anyone actually deciding it was right. A classifier gives a *score*; turning it into a yes/no needs a threshold, and the default of 0.5 is almost never right for an imbalanced, unequal-cost problem. Recall that the class-weighted retrain above reached 90% recall at 0.5 — but the same recall gain is often reachable a cheaper way: keep the *original* unweighted classifier's scores, and simply lower the cutoff instead of retraining. If missing a fraud is far worse than a false alarm, you want to flag on much weaker suspicion — a far lower threshold. So after training, pick the threshold *deliberately*: look at the precision and recall you get at each possible cutoff, and choose the one that matches what the problem actually costs. Train the model to rank well; then set the threshold to act well.

Precision@K: when you can only act on a few

Often the real constraint isn't a threshold at all — it's *capacity*. A fraud team can investigate maybe 500 alerts a day; a doctor can review only so many flagged scans. In that world the question changes from "what's our recall?" to "of the top 500 cases the model ranks as riskiest, how many are real fraud?" That's precision@K — precision measured on the top K by score. Its partners are recall@K (of all fraud, how much did the top K capture?) and lift@K (how much better than random is the top K?). When action is capacity-limited, optimise the ranking for precision@K, not a global threshold — the model just has to get the *worst* cases to the top of the list.

The cost matrix, made explicit

"Missing a fraud is worse than a false alarm" can be written down precisely as a cost matrix: a dollar cost for each cell of the confusion matrix, drawn from the business itself rather than guessed — suppose the bank's own loss data says a false negative costs \$2,000 (the average unrecovered fraud amount) and a false positive costs \$5 (an analyst's few minutes clearing the alert), with true predictions costing 0. That's the number that actually settles the class-weight trade above: at \$2,000 per miss and \$5 per false alarm, the unweighted classifier's 15 misses and 35 false alarms cost 15×\$2,000 + 35×\$5 = \$30,175, while the class-weighted classifier's 5 misses and 90 false alarms cost 5×\$2,000 + 90×\$5 = \$10,450 — the weighted model is cheaper by a wide margin, even though its F1 was lower. Once you have a cost matrix, the optimal decision isn't a guessed threshold — it's the one that minimises expected cost: flag whenever the expected cost of flagging is below the expected cost of not flagging, which for a calibrated probability gives an exact optimal threshold. This is the rigorous version of "pick the threshold from the costs," and it's why calibrated probabilities matter here.

SMOTE has sharp edges

The cost-matrix math above assumed the rebalancing choice behind it — class weights or resampling — was already sound. One of those options, SMOTE, deserves its own warning: it's popular but easy to misuse. The cardinal rule: apply it only inside cross-validation folds, after the split — never before. Oversample first and copies of the same synthetic points land in both train and validation, leaking and inflating your score. Beyond that, SMOTE struggles in high-dimensional sparse data (its "in-between" points are meaningless), with noisy labels (it amplifies the noise), with overlapping classes (it synthesises into the other class's territory), and with time-series (it invents points that violate temporal order). It's a tool, not a default.

Resampling distorts your probabilities

A subtle consequence interviewers love: oversampling or undersampling changes the class balance the model trains on, so its predicted probabilities no longer reflect the true base rate — they come out systematically too high for the minority class. SMOTE in particular tends to over-estimate minority-class probabilities. So if you resample *and* you need real probabilities (for cost-based thresholds or downstream use), you must recalibrate afterward, or correct the prior back to the true rate. Class weights avoid this problem, which is another reason to prefer them when probabilities matter.

Match the fix to the model

The right lever depends on the algorithm. Tree ensembles and boosting usually do best with class weights / `scale_pos_weight` (the same reweighting knob named in the gradient boosting module, roughly negatives/positives) plus threshold tuning — resampling buys them little. Linear and distance-based models (logistic regression, k-NN, SVM) are more sensitive to the geometry, so sampling and careful feature scaling can help them more. Don't apply one imbalance recipe blindly across model families.

The fuller metric menu — for when precision/recall/F1 alone don't settle an argument in a room.

Beyond precision/recall/PR-AUC, know the wider toolkit: balanced accuracy (average recall across classes — useful when you need one number a non-technical audience can read as "how good," without precision and recall's two-number nuance), F1 and its macro/micro/weighted variants (macro treats classes equally, weighted accounts for size), MCC (Matthews correlation — one number computed from all four confusion-matrix cells at once, from −1 to +1, so a model can't game it by ignoring the rare class the way accuracy can; robust on imbalance and often the best single summary when you want just one trustworthy figure), specificity (recall's mirror image: of the real negatives, what fraction did the model correctly call negative?) and the false-positive/false-negative rates (those same misses, read as fractions of the actual-negative and actual-positive totals instead), and — always — the confusion matrix read at your chosen threshold so you see the actual counts, not just a summary.

When imbalance gets extreme

At 1-in-100,000 (rare diseases, novel fraud), the classification framing itself starts to break, and you switch strategies. Frame it as anomaly detection (model "normal," flag deviations) rather than two-class classification. Use a two-stage retrieval-then-rank pipeline. Design explicitly around human review capacity (precision@K), delayed labels (the truth arrives weeks later), and alert fatigue (too many false positives and reviewers stop trusting the system). Extreme imbalance is a systems problem, not just a loss-function tweak.

Key points

• On an imbalanced problem, accuracy is a trap — use precision, recall, and PR-AUC instead. On 950 legitimate / 50 fraud, a model that always guesses "legitimate" scores 95.0% accuracy and catches nothing — and a real classifier catching 35 of the 50 frauds also scores exactly 95.0%, an identical tie that hides a real gap. Recall (35/50 = 70% vs 0%) and precision (35/70 = 50% vs undefined) actually track the rare class, and F1 (58.3% vs 0%) finally shows the difference accuracy erased. PR-AUC sums up the recall/precision tradeoff across every threshold in one number, and it's far more honest than the more common ROC-AUC when positives are rare, because ROC-AUC hands out easy credit for correctly ignoring the huge majority.
• The real problem is not the imbalance — it is that the two kinds of mistake cost different amounts, and fixing recall has a real, computable price. Missing a fraud can cost thousands; a false alarm costs a moment. Accuracy pretends they are equal. Class weights (roughly 19:1, matching 950:50) push recall from 70% to 90% by retraining — but precision falls from 50% to 33.3%, F1 actually drops (58.3%→48.6%), and accuracy falls too (95.0%→90.5%). None of that means the fix failed: on a cost matrix of \$2,000 per missed fraud and \$5 per false alarm, the reweighted model costs \$10,450 against the original's \$30,175 — cheaper, despite the lower F1. The lesson: don't judge an imbalance fix by accuracy or even F1 alone — judge it against what the mistakes actually cost.
• The step everyone forgets: choose the decision threshold on purpose, not at the default 0.5. A classifier gives a score; turning it into a yes/no needs a cutoff, and 0.5 almost never matches an imbalanced, unequal-cost problem. If missing a positive is far worse than a false alarm, flag on weaker suspicion — a lower threshold. After training, look at the precision and recall you get at each cutoff and pick the one that matches what the problem actually costs. Train the model to rank; then set the threshold to act.
• When action is capacity-limited, optimise precision@K and derive the threshold from an explicit cost matrix. If a team can only review the top 500 alerts, the metric is precision@K (of the top K by score, how many are real) with recall@K and lift@K — the model just needs the worst cases at the top of the list. And "pick the threshold from costs" has a rigorous form: write a cost matrix (dollar cost per confusion-matrix cell) and choose the threshold that minimises expected cost, which for a calibrated probability is exact — another reason calibrated probabilities matter. Round out judging with balanced accuracy, macro/weighted F1, MCC, and the confusion matrix at your chosen threshold.
• SMOTE has sharp edges, resampling distorts probabilities, and the right fix depends on the model. Apply SMOTE only inside CV folds after the split (before it leaks), and avoid it with high-dimensional sparse data, noisy or overlapping labels, and time-series. Resampling changes the training class balance, so predicted probabilities come out too high for the minority class — recalibrate afterward if you need real probabilities (class weights sidestep this). Match the lever to the family: tree/boosting → class weights + `scale_pos_weight` + threshold tuning; linear/distance models → sampling and scaling. And at extreme imbalance (1-in-100k), switch framing to anomaly detection, two-stage retrieval/ranking, and design around review capacity, delayed labels, and alert fatigue.

Recap

• Accuracy is a trap — 950/50 dataset: do-nothing model and a 70%-recall classifier both score 95.0% accuracy, identically.
• Precision/recall/F1 break the tie: do-nothing = 0%/0%/0%; real classifier = 50%/70%/58.3%.
• Real issue: the two kinds of mistake cost different amounts — not the imbalance itself.
• Class weights (≈19:1) traded recall for precision: 70%→90% recall, 50%→33.3% precision, F1 58.3%→48.6% (lower!), accuracy 95.0%→90.5%.
• Cost matrix settles it: at \$2,000/miss, \$5/false-alarm, the weighted model costs \$10,450 vs \$30,175 — cheaper despite the lower F1.
• Set the decision threshold deliberately to match the cost — don't leave it at 0.5.
• When action is capacity-limited, optimise precision@K and derive the threshold from a cost matrix.
• SMOTE has sharp edges; resampling distorts probabilities — the right fix depends on the model.

Check your understanding