Validation Set Traps & Data Leakage

Leakage taxonomy: temporal, group, label leakage, how to audit

You are building a model to predict whether a stock will go up over the next week — a next-week up/down classification. Among your features you include one called "whether the stock rose over the next 7 days." You train, validate, and get 95% accuracy — numbers you have never seen. You ship it. In production, accuracy is 2%. What happened?

The feature cheated. "Whether the stock rose over the next 7 days" is computed from the very thing you are trying to predict. The model did not learn anything — it read the answer straight off that feature. During validation the answer was sitting right there in the inputs; in production that information does not exist yet, so the model is worthless. This is data leakage: information from the future, or from the outcome itself, sneaking into the training features.

The dangerous part is that a normal train/test split does *not* catch it — because both halves are contaminated the same way. Leakage comes in three main flavours, each with its own fix.

Temporal leakage — peeking at the future

You have time-ordered data and you split it into train/test by picking rows at random. Now some training rows come from *after* the test rows' prediction time, so the model gets to peek ahead. The fix is a strict time cutoff: every feature for a row must be built only from data that existed *before* that row's prediction moment. Train on the past, test on the future — never a random shuffle across dates.

Group leakage — the same subject on both sides

The same user (or patient, or store) shows up in both train and test. Your fraud model trains on some transactions from users A, B, C and is tested on *other* transactions from those same users — so it memorises each person's spending habits, patterns that simply will not exist for brand-new users at deployment. The fix is to split by *group*: every row from a given user goes entirely to one side.

Label leakage — the right time, the wrong direction

This one is subtle. The feature has a valid timestamp, but it is a *consequence* of the outcome rather than a cause of it. Classic example: predicting whether a patient was hospitalised, using "days in the ICU" as a feature — that number only exists *because* they were hospitalised. The audit question is not "when was this computed?" but "does this feature come *before* the outcome in the causal chain, or *after* it?"

The one question that catches almost everything

For every feature, ask: *would this exact value exist at the moment of prediction in production, knowing nothing about future events?* If the answer is no, it is leakage — full stop. And a few red flags should trigger an immediate audit: validation AUC above 0.97 on a problem where even human experts disagree; one feature towering over all the others in importance; train and validation accuracy nearly identical with no gap; or a mediocre model suddenly looking brilliant right after you added one new feature group.

Preprocessing leaks too — fit everything on the training fold only

Leakage isn't just about features you engineer; it hides in *preprocessing*. Any step that *learns something from the data* — a StandardScaler's mean and variance, an imputer's fill values, a PCA's components, feature selection, SMOTE oversampling, target encoding — must be fit on the training fold only, then *applied* to validation/test. Fit your scaler on the whole dataset before splitting and the training rows already "know" the test set's statistics. The clean fix is a scikit-learn Pipeline that bundles every transform with the model, so cross-validation re-fits the whole chain inside each fold and leakage becomes structurally impossible.

The fuller leakage taxonomy

Beyond temporal, group, and label leakage, name the rest: train-test contamination (a preprocessing step or a duplicate bleeds test info into train); duplicate / near-duplicate leakage (the same row, or an augmented copy, in both splits); proxy leakage (a feature that's an innocent-looking stand-in for the label, like a customer-ID range that encodes signup cohort); post-outcome features (computed after the event you're predicting); aggregation-window leakage (a rolling average whose window reaches past the prediction time); and survivorship bias (training only on entities that survived — e.g. still-active accounts — so the model never sees the ones that churned/failed).

Three timestamps, not one

"When is this feature available?" is really three questions. The event timestamp (when the thing happened), the feature-computation timestamp (when your pipeline calculated it), and the data-availability timestamp (when the value was actually queryable in production). A value can *exist* historically but not have been *available* at decision time — a label confirmed a week later, a nightly-batch feature not ready until 2am. Point-in-time correctness means joining features as of their availability timestamp, not their event timestamp. Feature stores exist largely to get this join right.

Real systems need group and time splits together

The splits aren't mutually exclusive. A fraud or churn system usually needs both: hold out *future* data (temporal) *and* make sure a held-out user's rows never appear in training (group). A pure temporal split can still leak a user's identity across the cutoff; a pure group split can still let the model peek at the future. The split you choose should mirror the production question — new users, new sessions, new transactions, or future events — and often that's a combined group-plus-time holdout.

Negative controls: sanity tests that catch leaks

A few cheap experiments smoke out leakage. Shuffle the labels and retrain — a properly-built model should collapse to chance; if it still scores well, a feature is leaking the label. Compare a random split against a temporal split — a big gap means temporal leakage. Drop the top suspicious feature — if AUC craters from 0.97 to 0.75, that feature was carrying the leak. Train on future-only features as a deliberate check of what's genuinely available. These controls turn "I have a bad feeling" into evidence.

How leakage shows up in production

Leakage often isn't caught offline at all — it surfaces after deploy. The signatures: an offline-online collapse (0.97 validation, 0.64 live), a null-rate spike on a feature that turns out to be unavailable at serving time, feature-freshness violations (a value that was instant offline takes hours to compute live), or a top feature that simply *can't be computed* in the real-time path. Monitoring these in production is your last line of defence when the offline audit misses a leak.

Key points

• When to audit for leakage: any time validation looks too good to be true. Red flags: AUC above 0.97 on a problem where domain experts only manage 70–80%; a single feature with far higher importance than all the others (can it even be known at prediction time?); train and validation accuracy within a point or two of each other with no regularisation (the test set is probably contaminated); or AUC jumping ten-plus points right after you add one new feature group (check that group's causal link to the label).
• The most common trap: target encoding computed on the full dataset before the split. Target encoding replaces a category with the average outcome for that category. Compute those averages over the whole dataset (test rows included) and each test row's feature now carries information from its own label — a direct leak. Fix: compute the encodings out-of-fold (each row encoded using only the other folds), or wrap it in a scikit-learn Pipeline inside cross-validation. It is easy to miss because the code looks completely innocent.
• The diagnostic: for every feature, ask whether its value would exist at the exact moment of prediction. For each feature, write down what data it uses, when that data becomes available, and whether that is before or after the prediction time. Any feature whose data only arrives after the prediction moment is leakage. You can do this whole audit in a spreadsheet. Inheriting a model? Check its top five features by importance and confirm each one passes the timestamp test.
• Preprocessing leaks too, and the taxonomy is bigger than three types. Anything that learns from data — scaler, imputer, PCA, feature selection, SMOTE, target encoding — must be fit on the training fold only; wrap it in a scikit-learn Pipeline so CV re-fits it inside each fold. Beyond temporal/group/label, watch for duplicate/near-duplicate leakage, proxy features (an ID range encoding cohort), post-outcome features, aggregation-window leakage (a rolling window reaching past prediction time), and survivorship bias. And distinguish three timestamps — event, feature-computation, and data-availability — joining features as of *availability* (what point-in-time feature-store joins enforce).
• Mirror production in the split, prove it with negative controls, and monitor after deploy. Real systems often need a combined group-plus-time holdout (future data AND unseen users), matched to the production question (new user/session/transaction/event). Catch leaks with negative controls: shuffle labels (a clean model drops to chance), compare random vs temporal split (a gap = temporal leak), and drop the top suspicious feature (a crater = it carried the leak). Leakage frequently surfaces only in production — an offline-online collapse, a feature null-rate spike, a freshness violation, or a top feature that can't be computed in the serving path — so monitor these as the last line of defence.

Recap

• Leakage = future or outcome information sneaking into features — a feature like "whether the stock rose over the next 7 days" reads the answer straight off, giving 95% validation and 2% production. A normal train/test split doesn't catch it because both halves are contaminated the same way.
• Temporal leakage — peeking at the future: a random split of time-ordered data lets some training rows come from *after* the test rows' prediction time. Fix with a strict time cutoff — every feature built only from data that existed before that row's prediction moment; train on the past, test on the future.
• Group leakage — the same subject on both sides: the same user (or patient, or store) shows up in both train and test — trained on some of their transactions, tested on others — so it memorises habits that won't exist for brand-new subjects at deployment. Fix by splitting by group — all of one subject's rows go entirely to one side.
• Label leakage — right time, wrong causal direction: the feature has a valid timestamp but is a *consequence* of the outcome, not a cause ("days in ICU" only exists because the patient was hospitalised). The audit isn't "when was it computed?" but "does it come before or after the outcome in the causal chain?"
• The one question that catches almost everything: would this exact value exist at the moment of prediction in production, knowing nothing about future events? If no, it's leakage. Red flags: AUC > 0.97 where experts disagree, one feature towering over all others, no train/val gap, or a mediocre model suddenly brilliant after one new feature group.
• Preprocessing leaks too — fit on the training fold only: anything that *learns* from data (StandardScaler's mean/variance, imputer fills, PCA components, feature selection, SMOTE, target encoding) must be fit on train and applied to val/test. Wrap it all in a scikit-learn Pipeline so CV re-fits the whole chain inside each fold and leakage becomes structurally impossible.
• Real systems need group + time together, proven with negative controls, monitored in prod: hold out future data AND unseen users, matched to the production question. Confirm with negative controls (shuffle labels → a clean model collapses to chance; compare random vs temporal split → a gap means temporal leak; drop the top feature → a crater means it carried the leak). Leakage often surfaces only after deploy — an offline-online collapse, a feature null-rate spike, a freshness violation — so monitor these as the last line of defence.

Check your understanding