Class Imbalance
When 99% of examples are one class, accuracy is a lie — learn the techniques that actually work.
You are building a fraud detector. In your data there are 999 legitimate transactions for every 1 fraud. You train a model, it scores 99.9% accuracy, and your stakeholder is thrilled. You should not be — because a "model" that simply labels *everything* legitimate, learning nothing and looking at no features at all, *also* scores 99.9%. Accuracy is measuring the wrong thing: how often you agree with the majority class, which you can ace by ignoring the rare class entirely.
The rare class is not a flaw in your data — fraud genuinely *is* rare. The trouble is how ordinary training reacts to it. The loss adds up mistakes across all examples, and with 999 legit transactions per fraud, getting the legit ones right dominates the total 999-to-1. The gradient points almost entirely away from fraud, so the model learns to shrug it off. There are three places to fix this, at three points in the pipeline.
Fix 1 — reweight the loss (start here)
The cleanest first move is class weights: tell the loss that a mistake on a fraud example counts as much as roughly 999 mistakes on legit ones (set `class_weight='balanced'`, or `scale_pos_weight` in XGBoost). Now the rare class pulls on the gradient as hard as the common one. No data added or removed — just a reweighted loss.
Fix 2 — manufacture more minority examples (SMOTE), carefully
SMOTE takes a different tack: it invents new fraud examples by interpolating *between* real ones in feature space — pick two nearby frauds and drop a synthetic fraud on the line between them. This gives the model a denser minority region to learn a boundary from. But it has a real failure mode: when fraud and legit heavily *overlap*, interpolating between two frauds can plant a synthetic "fraud" right in the middle of legit territory — a contradictory, misleading training point. So SMOTE is not a default; it shines mainly when the minority class is genuinely sparse (a few hundred examples), and it needs care when the classes mix.
Fix 3 — move the decision threshold
This one does not touch training at all. A classifier outputs a *probability*; turning it into a yes/no needs a threshold, and the default 0.5 is almost never right here. If a missed fraud costs 10,000 and a false alarm costs 50 in review time — a 200:1 asymmetry — you should flag on much weaker suspicion: the cost-minimizing threshold works out to roughly 0.005, not 0.5. The threshold is a *business-cost* decision, not a modeling one: plot precision against recall across thresholds and pick the point that minimises your expected cost.
And above all: stop reporting accuracy
The deepest fix is the metric itself. On imbalanced data, use precision, recall, and PR-AUC, which actually measure how you do on the rare class. Even ROC-AUC can read a flattering 0.97 while the model catches almost no fraud, because the huge pile of true negatives swamps its denominator. Accuracy on an imbalanced problem is not a partial truth — it is actively misleading.
The fuller metric menu
Precision/recall/PR-AUC are the start; know the rest so you can pick the honest single number. Balanced accuracy (average recall across classes) doesn't reward always-predict-majority. MCC (Matthews correlation) uses all four confusion cells and is often the best single summary under imbalance. Macro/micro/weighted F1 average per-class F1 differently: macro treats classes equally, weighted scales by class size, and micro pools every prediction into one global count — which collapses to plain accuracy in binary classification, the exact number this module just told you not to trust. Specificity (TNR) and the FPR/FNR matter when the cost of each error type differs. And when action is capacity-limited — a fraud team reviews the top K — precision@K, recall@K, and lift@K are the right frame, because the model only has to rank the worst cases to the top.
A fair word on ROC-AUC — and how resampling breaks calibration
ROC-AUC isn't *useless* under imbalance — it's a valid ranking metric — but it's misleading because the huge true-negative pile keeps FPR tiny, so it can read 0.97 while precision is terrible; PR-AUC is usually more informative for rare positives. A subtler cost: class weighting and resampling distort probability calibration. Both change the class balance the model trains on, so its predicted probabilities come out too high for the minority class. If you need real probabilities (for cost-based thresholds), recalibrate afterward (Platt or isotonic) and check the reliability curve — the ranking may be fine while the numbers lie.
Validating rare events
With few positives, careless validation is noise. Use stratified splits so every fold holds enough positives (a random split can leave a fold with almost none), prefer repeated cross-validation to average out the high variance of a single split, and put confidence intervals around recall and precision — "recall 0.8" on 20 positives has an enormous interval. For fraud and any time-ordered data, use a temporal split (train on the past, test on the future), because a random split lets the model peek across the fraud timeline.
Sampling alternatives, and losses built for imbalance
Beyond plain class weights and vanilla SMOTE, know the toolkit: random undersampling (drop majority examples — fast, throws away data), random oversampling (duplicate minority — risks overfitting), the SMOTE + cleaning hybrids SMOTE-Tomek and SMOTE-ENN (oversample then remove the confusing points near the boundary), focal loss (down-weights easy majority examples so training focuses on the hard minority — popular in detection), and balanced random forests (each tree trained on a balanced bootstrap). Match the tool to the model and the imbalance severity rather than reaching for SMOTE reflexively.
The cost-sensitive formula, made explicit
"Pick the threshold from costs" has an exact form. Write the per-error costs and choose the threshold that minimises expected cost = FP·cost_FP + FN·cost_FN over the validation set (a true-positive/true-negative usually costs 0). Equivalently, when there's a hard review capacity, set the threshold to fill that queue with the highest-risk cases (precision@K). This turns "which threshold?" from a guess into an optimisation against numbers you can write down.
In production, imbalance keeps moving
Rare-event models need specific monitoring. Track alert volume (a spike means the model or the base rate shifted), precision drift (are the flags still real?), and base-rate drift (the fraud rate itself changes, which silently moves precision even if the model is unchanged). Account for delayed labels (confirmation arrives weeks later) and review capacity, and close the loop by retraining on confirmed cases as they come in. And at extreme imbalance (1:10,000+), stop treating it as one classifier: use a two-stage system — a high-recall candidate generator narrows millions to a manageable pool, then a precision-focused ranker or human-review queue orders that pool. Extreme rarity is a systems-design problem, not a loss-function tweak.
Key points
- Use cost-sensitive training as your first move on any imbalanced tabular problem. Set class_weight='balanced' in sklearn or scale_pos_weight in XGBoost. This requires no data modification, carries no SMOTE-before-split leakage risk, and integrates cleanly into cross-validation. Reserve SMOTE for cases where the minority class is so sparse that the model literally cannot learn its decision boundary — roughly, fewer than a few hundred minority examples in training.
- The most common production trap: applying SMOTE to the full dataset before the train-test split. Synthetic minority samples are generated by interpolating between real minority examples. If those real examples are in both train and test, the synthetic samples are geometrically close to test-set points. The test set is contaminated with structure derived from training data. Evaluation looks strong; production collapses. Always split on real data first, then apply SMOTE only inside the training fold.
- Diagnose your model with a precision-recall curve, not a single threshold. Plot precision vs. recall across all possible thresholds. The shape of the curve tells you how the tradeoff behaves at your operating point. Then compute the business cost at each threshold — multiply false negative count by the cost of a missed fraud, false positive count by the cost of a false review — and pick the threshold that minimizes total expected cost. A model with recall 0.9 and precision 0.3 may be exactly right if the cost asymmetry is 200:1 in favor of catching fraud.
- Know the full metric menu and that resampling breaks calibration. Beyond precision/recall/PR-AUC: balanced accuracy, MCC (best single number under imbalance), macro/micro/weighted F1, specificity, FPR/FNR, and precision@K/recall@K/lift@K when action is capacity-limited. ROC-AUC isn't useless but is misleading under rare positives (huge TN pile keeps FPR tiny) — prefer PR-AUC. Crucially, class weighting and resampling shift the training class balance and inflate minority probabilities, so recalibrate (Platt/isotonic) and check the reliability curve if you need real probabilities.
- Validate rare events carefully, pick the sampling tool deliberately, and monitor drift. Use stratified splits (enough positives per fold), repeated CV, confidence intervals on recall/precision, and temporal splits for fraud. Beyond class weights and SMOTE: random under/oversampling, SMOTE-Tomek/SMOTE-ENN (oversample then clean the boundary), focal loss (down-weight easy majority examples), and balanced random forests. Choose the threshold by minimising expected cost = FP·cost_FP + FN·cost_FN (or to fill review capacity). In production, monitor alert volume, precision drift, and base-rate drift, handle delayed labels, and at 1:10,000+ use a two-stage high-recall-then-precision pipeline.
Accuracy on an imbalanced dataset measures how well the model predicts the majority class — which it can do by ignoring minority examples entirely. The fix starts with the metric, then the loss function, then the decision threshold. Resampling is a last resort, not a default.
Recap
- Accuracy measures the majority class — a model can hit 99% by ignoring the minority entirely.
- Fix order: the metric first, then the loss function, then the decision threshold. Resampling is a last resort, not a default.
- Cost-sensitive training is the first move: `class_weight='balanced'` / `scale_pos_weight` — no data change, no leakage, clean in CV.
- SMOTE-before-split leaks: synthetic points interpolated from real minority rows land near test points. Split first, SMOTE inside the train fold only.
- PR curve over a single threshold; pick the threshold that minimizes expected cost = FP·cost_FP + FN·cost_FN.
- Metric menu: PR-AUC over ROC-AUC under rare positives; MCC as best single number; precision@K when action is capacity-limited. Resampling breaks calibration — recalibrate.
- Extreme imbalance (1:10,000+) is a systems problem: two-stage high-recall candidate generator → precision ranker/human review.
Check your understanding
Q1. Your fraud model achieves 99.2% accuracy and your colleague is satisfied. What would you check?
- A) Check for class imbalance in the training set and verify the model's F1 score on the test set. If F1 is above 0.9, the 99.2% accuracy is genuine and the model is working correctly.
- B) Check whether the model was trained with sufficient regularization — high accuracy on imbalanced data is often a sign that L2 regularization is too weak and the model has overfit to the majority class.
- C) Check the AUC-ROC score — if AUC-ROC comes back above 0.95, the accuracy figure is meaningful and the model is genuinely distinguishing fraud from legitimate transactions well.
- D) First check the baseline: predicting "not fraud" for everything already yields ~99% accuracy here. Check recall, precision, and AUC-PR directly — they measure performance on the minority class, which accuracy hides.
Q2. Why is applying SMOTE to the full dataset before splitting into train and test sets invalid?
- A) SMOTE interpolates between real minority samples. Applied before splitting, synthetic points can land near real test examples, so the holdout is no longer valid. Split first, then SMOTE the training set.
- B) SMOTE applied before splitting is invalid because it changes the class balance of the test set, making it impossible to compute meaningful precision and recall metrics on the held-out data.
- C) SMOTE applied before splitting is invalid because it requires knowing the class labels of the test set, which means the model has implicitly seen the test labels during preprocessing.
- D) SMOTE applied before splitting is mainly computationally wasteful — synthetic samples get discarded once the test set is held out, so applying it inside training alone is simply more efficient.
Q3. Compare class weighting and SMOTE for a 50:1 imbalanced tabular dataset. Which TWO of the following are true?
- A) Always use SMOTE for tabular data regardless of minority size — class weighting only adjusts the loss function, while SMOTE physically creates new training examples with real decision-boundary detail.
- B) Class weighting is almost always the first choice for tabular data — no data modification, integrates cleanly with cross-validation, and scales cleanly to any imbalance ratio you throw at it.
- C) SMOTE is worth reaching for specifically when the minority class is genuinely sparse in feature space — say only 50 rows at 50:1 — where reweighting alone can't give the model enough boundary to learn.
- D) Choose between them purely based on model type: class weighting always works best for gradient-boosted trees, while SMOTE always works best for logistic regression and neural networks.
Q4. You lower classification threshold from 0.5 to 0.2 and recall increases 0.6 to 0.9 but precision drops 0.8 to 0.3. Is this an improvement?
- A) No — the F1 score decreased overall. F1 is the harmonic mean of precision and recall, and a precision drop from 0.8 to 0.3 clearly outweighs a recall gain from 0.6 to 0.9 in the combined score.
- B) Depends entirely on the relative cost of false negatives versus false positives — compute expected cost at each threshold (fraud cost x FN plus review cost x FP) and pick the minimum.
- C) Yes — recall is always the primary metric for fraud detection specifically, so any improvement in recall is unconditionally an improvement in a fraud model, regardless of the precision impact.
- D) Yes — the model now catches 90% of all fraud cases, which comfortably exceeds the widely cited industry-standard threshold of 85% recall required for production fraud systems.
Q5. What is the SMOTE failure mode when minority and majority classes heavily overlap in feature space?
- A) When classes overlap heavily, SMOTE generates synthetic samples that become visually indistinguishable from majority examples, causing the model to wrongly learn that entire high-density regions are minority.
- B) Heavy class overlap causes SMOTE to generate synthetic samples that end up as near-exact duplicates of existing minority points, adding no real new geometric information and wasting the oversampling effort.
- C) Heavy class overlap makes SMOTE noticeably slower, because its k-NN search must examine a much larger fraction of the dataset to find each sample's k nearest minority neighbors reliably.
- D) SMOTE interpolates between minority samples. When classes overlap, some sit surrounded by majority points, so synthetic points can land INSIDE the majority region — a signal that SMOTE-ENN later cleans up.
Q6. You fix imbalance with class weighting, and the model's ranking (PR-AUC) is excellent, but downstream cost-based thresholding behaves oddly because the predicted probabilities seem systematically too high for the fraud class. What happened, and what do you do?
- A) Nothing is actually wrong here — class weighting mathematically never affects predicted probabilities in any way, so the downstream thresholding logic must simply contain a bug.
- B) Class weighting changes the effective class balance trained on, inflating minority-class probabilities even when ranking is fine. Recalibrate afterward and check the reliability curve.
- C) Switch entirely from class weighting to plain accuracy as the reported metric, which will automatically make the model's predicted probabilities correct again.
- D) The probabilities themselves are completely fine as they are; the real fix is to always use a fixed 0.5 threshold regardless of any underlying business costs involved.
Q7. You have a 1:50,000 imbalance (a few hundred positives in tens of millions of rows) and single-classifier approaches keep failing. What overall design and validation approach fits?
- A) Simply crank class_weight higher and higher until a single classifier fully separates the classes — extreme imbalance like this is always solvable given a large enough weight value.
- B) At this rarity, treat it as a systems problem: a two-stage pipeline — a high-recall generator narrows millions to a pool, then a ranker orders it against review capacity.
- C) Randomly undersample the majority class down to a clean 1:1 ratio and train a single logistic regression — this fully and permanently solves extreme imbalance with absolutely no downsides.
- D) Simply report plain accuracy as the headline metric, which at a 1:50,000 imbalance ratio will read near 100% and conclusively prove the model is working correctly.
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →