Cross-Validation Strategies

k-fold, stratified, group k-fold, time-series CV, nested CV

You are training a random forest on 5,000 examples. You tune max_depth with a single 80/20 split. By luck, the test set landed on mostly *easy* examples — the ones any model gets right. You read 87% accuracy, pick the max_depth that scores best on this one split, and call it done. You deploy. Performance: 79%. What went wrong?

A single split is just one of the many ways you *could* have divided the data. You got a lucky draw, your estimate had high variance, and you staked the whole tuning decision on it.

k-fold: don't trust one split, average many

k-fold cross-validation runs several splits and averages them. With k=5 on 5,000 examples: cut the data into 5 groups of 1,000; train on 4,000 and test on the held-out 1,000; rotate so each group is the test set exactly once; average the 5 scores. Now every example is tested once, and your estimate is far steadier. Standard choices are k=5 (trains on 80% each time, tests on the other 20%) or k=10 (trains on 90%, tests on 10%, but slower). Leave-one-out (k = n) sounds ideal but is actually noisy — a single weird test example swings each fold's score.

Plain k-fold is right only when your rows are independent draws from the same distribution. Step outside that and you need a different flavour.

Stratified — when the classes are imbalanced

With a 5% positive rate, a careless random split can hand you one fold with 1% positives (too easy) and another with 9% (too hard); averaging scores from folds that are secretly different problems is misleading. Stratified k-fold forces every fold to hold roughly the same 5% positives. Stratify for any classification task where the positive class is under about 20%.

Group — when the same subject repeats

Suppose you have 50,000 hospital visits from 5,000 patients, and you will deploy on *new* patients. With plain k-fold, patient 142's January visit trains the model and their June visit tests it — so the model quietly memorises patient 142 and "predicts" a patient it already knows. Group k-fold keeps all of one patient's visits on the same side, so the test fold is always patients the model has never seen — which is the real deployment situation.

Time-series — walk forward, and mind the gap

On time-ordered data plain k-fold is not just rough, it is *wrong*: it lets the model train on month 10 to predict month 3. The fix is walk-forward validation — train on an early window, test on the next, then expand: train on [start … t], test on [t … t+1], and keep going, always keeping time in order. One subtlety most people miss: if any feature uses a rolling window (a 7-day average, say), a training row from day 89 and a test row from day 91 share raw data, because day 91's feature is built from days 85–91. So add a purge gap — drop the rows within one feature-window of the boundary — so the two sides never share ingredients.

One honest caveat

Cross-validation estimates performance on *new data from the same distribution, drawn the same way.* It does not save you if the world shifts before deployment, and it cannot undo leakage baked in before the split. And if you tune hyperparameters on the very folds you then report, you have quietly cheated: trying 200 configurations and reporting the best fold score is biased upward by the search itself. The clean fix is nested CV — an outer loop estimates quality on data the tuning never touched, while the inner loop does the full hyperparameter search. Slower, but it is one honest way to score your whole pipeline — a separate holdout test set the tuning never touches works too, if you can afford to set data aside instead of looping.

When the folds themselves are noisy: repeated k-fold

On small datasets a single 5-fold split is still one arbitrary partition — shuffle the rows differently and the estimate moves. Repeated k-fold runs the whole k-fold procedure several times with different random shuffles and averages, shrinking the variance of the estimate. It costs more compute but gives a steadier number when data is scarce or fold scores are unstable (like the [0.83, 0.84, 0.92, 0.85, 0.83] case where one fold is an outlier).

When you need both balance and isolation: stratified group k-fold

Stratified k-fold keeps the class ratio; group k-fold keeps entities together — but real problems often need *both*. Imbalanced medical data with multiple visits per patient wants every fold to hold the same rare-disease rate and never split a patient across folds. Stratified group k-fold does both at once. It's an approximation (you can't always satisfy both perfectly), but it's the right tool when you have imbalance and repeated subjects together — a very common combination.

Rolling versus expanding windows

Walk-forward comes in two flavours, and the choice is about drift. An expanding window always trains on *all* history up to time t — more data, better when the relationship is stable. A rolling (sliding) window trains only on the most *recent* fixed span, deliberately forgetting old data — better under strong concept drift, where ancient patterns actively mislead. Rule of thumb: expanding by default, rolling when the world changes fast enough that stale data hurts more than it helps.

Nested CV estimates the pipeline, not the final model

A subtlety people miss: nested CV gives you an unbiased estimate of your *model-selection procedure's* quality — but the model you actually ship is usually retrained on all available training data using the hyperparameters that procedure chose. Nested CV answers "how good is my process," not "here's the exact model." So report the nested-CV score as your honest performance estimate, then retrain on everything for deployment. Don't confuse the two.

Fold scores aren't independent — mean ± std isn't a confidence interval

Reporting mean ± std across folds is useful, but treat it carefully: the folds overlap in training data (each pair of 5-fold training sets shares most of their rows), so the fold scores are correlated, not independent draws. That means the naive standard error understates the true uncertainty, and the std across folds is *not* a valid 95% confidence interval. Use it as a rough spread and an instability flag, not as a rigorous statistical bound.

Stratifying regression and multi-label targets

Stratification isn't only for single-label classification. For regression, stratify on *binned* target values so each fold spans the full range (otherwise one fold can get all the cheap houses). For multi-label problems, use iterative/multi-label stratification that balances each label's distribution across folds. Naive random splitting can leave a rare label almost absent from some folds, making those folds unrepresentative.

Key points

• Match the CV strategy to your deployment assumption — it is not a style choice. Plain k-fold (k=5 or 10): rows are independent, no time order, no repeated subjects. Stratified k-fold: any imbalanced classification, so every fold keeps the same class mix. Group k-fold: the model will score entities (users, patients, stores) it has never seen, so all rows from one entity stay on one side. Walk-forward: any time-series data, always. Use the wrong one and you are measuring a deployment scenario that does not exist.
• The most common trap: plain k-fold on time-series data. It produces validation scores that look fine and mean nothing. The tell: CV says AUC 0.85, then it drops sharply the moment you run a proper time-ordered holdout. Always split by time for time-series, default to an expanding walk-forward window, and add a purge gap (say 7 days) between train and validation so rolling-window features cannot bleed across the boundary.
• The diagnostic: if you tuned hyperparameters and reported the best fold score, that number is inflated. Trying 200 configurations and reporting the best one's validation score is selection bias — you implicitly fit to that partition, and the more you tried, the worse the inflation. Fix it with nested CV (search only in the inner fold) or a separate test set never touched during tuning. Quick check: retrain with the chosen settings and score on data the tuning never influenced — if it drops a lot, the bias was real.
• Reach for the right variant, and read fold spread honestly. Repeated k-fold shrinks estimate variance on small/unstable data; stratified group k-fold handles imbalance and repeated subjects together; expanding windows use all history (stable world) while rolling windows keep only recent data (strong drift). Stratify regression on binned targets and multi-label with iterative stratification so no fold misses a rare label.
• Know what nested CV estimates and that folds aren't independent. Nested CV gives an unbiased estimate of your model-selection *procedure*; the shipped model is then retrained on all training data with the chosen hyperparameters, so report the nested score but don't mistake it for the exact deployed model. And because k-fold training sets overlap heavily, fold scores are correlated — mean ± std is a useful spread and instability flag but not a valid confidence interval, so don't treat the std as a rigorous statistical bound.

Recap

• A single split has high variance — k-fold averages many: one 80/20 split can land on easy examples (87%) and mislead your tuning (deploy → 79%). k-fold cuts the data into k groups, trains on k−1 and tests on the held-out one, rotates, and averages — every example tested once, a far steadier estimate. k=5 (trains on 80%, tests on 20%) or k=10 (trains on 90%, tests on 10%, slower); leave-one-out is actually noisy (one weird example swings each fold).
• Stratified — for imbalanced classes: at a 5% positive rate a careless split can hand you one fold with 1% positives and another with 9% — secretly different problems. Stratified k-fold forces each fold to hold roughly the same class ratio; use it for any classification under ~20% positive.
• Group — when the same subject repeats and you deploy on new entities: plain k-fold lets patient 142's January visit train and their June visit test, so the model memorises a patient it already knows. Group k-fold keeps all of one entity's rows on one side, so test folds are always unseen entities — the real deployment situation.
• Time-series → walk-forward, and mind the gap: plain k-fold on time-ordered data is *wrong* (trains on month 10 to predict month 3). Train on [start…t], test on [t…t+1], expand, always keeping time in order. Add a purge gap — drop rows within one feature-window of the boundary — so a 7-day rolling feature can't share raw data across the split.
• Expanding vs rolling window is a call about drift: an expanding window trains on *all* history (more data, better when the relationship is stable); a rolling window keeps only the recent span and deliberately forgets old data (better under strong concept drift, where stale patterns mislead). Expanding by default, rolling when the world changes fast.
• Tuning on the folds you then report is cheating: trying 200 configs and reporting the best fold score is biased upward by the search itself. Fix with nested CV — an outer loop estimates quality on data the tuning never touched while the inner loop does the full hyperparameter search.
• Nested CV scores the *procedure*, not the shipped model: it gives an unbiased estimate of your model-selection process; the deployed model is retrained on *all* training data with the chosen hyperparameters, so report the nested score but don't confuse it for the exact model. And because k-fold training sets overlap heavily, fold scores are correlated — mean ± std is a useful spread and instability flag, not a valid confidence interval.
• Repeated k-fold shrinks the noise of small-data estimates: run the whole k-fold procedure several times with different random shuffles and average — costs more compute but gives a steadier number when data is scarce or fold scores are unstable (the [0.83, 0.84, 0.92, 0.85, 0.83] case above).
• Stratified group k-fold handles imbalance and repeated subjects together: when you have both a rare class and multiple rows per entity — imbalanced medical data with several visits per patient, say — it keeps every fold's rare-disease rate steady *and* keeps each patient on one side. It's an approximation, but the right tool for that common combination.
• Stratification isn't just for single-label classification: for regression, stratify on *binned* target values so each fold spans the full range (otherwise one fold can get all the cheap houses); for multi-label problems, use iterative/multi-label stratification so a rare label isn't left almost absent from some folds.

Check your understanding