Data Splits and Leakage
Understand why models that look great in development fail in production — and the exact mistakes that cause it.
You are predicting hospital readmission from records of 5,000 patients — and some patients appear many times, once per visit. You do a normal random 80/20 split, train, and get 99% validation accuracy. You ship it. Production accuracy: 73%. Nothing errored. Where did 26 points go?
Here is the leak. Patient 147 has nine visits in your data. The random split scattered seven of them into training and two into validation. So during training the model memorised patient 147 — their exact labs, age, history — and when it "predicted" their two validation visits, it was not generalising to a new patient at all; it was recognising someone it had already studied. Your validation score was partly measuring *memorisation*, and memorisation does not exist in production, where every patient is new. This is leakage: information from the evaluation set sneaking into training. It fires no error, the metrics look great, and the model fails on real data. It comes in four flavours.
The same entity on both sides
That was patient 147: related rows (same patient, user, or household) split across train and test, so the test set is not truly independent — this is group leakage. The fix is a group split — every row from a given patient goes entirely to one side, so validation always contains patients the model has never seen.
Training on the future
With time-ordered data, a random split lets the model train on March to predict January — the reverse of reality, where you never have tomorrow's data today. That's temporal leakage. The fix is a time cutoff: train on everything before a date, validate on everything after.
The quiet one
Fit a scaler (or imputer) on all 5,000 records *before* splitting, and its mean and spread were computed partly from the validation rows — so the training transform is tainted by the data you are supposed to be judging on. This is preprocessing leakage. Target encoding leaks worse, and differently. A target encoder replaces each category with the *mean of the label* for that category — fit it on all 5,000 records before splitting, and every validation row's encoded value was computed using the labels of the very validation rows it will later be graded on. That is not a distributional statistic sneaking across the boundary like a scaler's mean and spread; it is the label itself, smuggled into a feature column before training even starts. Fit every transformer — scaler, imputer, or target encoder — on the *training* data only, then apply it to validation. A pipeline enforces this so you do not have to remember.
The answer hiding in a column
A feature like "rehospitalised_within_30_days" on a readmission-prediction row *is the label wearing a disguise* — it could only be known after the outcome. This is feature leakage: any feature that needs knowledge of the future is unavailable at prediction time. The classic tell: a single new feature makes accuracy jump 15 points. Real features never do that.
The through-line: a held-out test set alone does not save you. If a feature was built using future information — even for rows that ended up in *training* — the model learned a pattern that will not exist once it is deployed. Real protection is structural: split *before* fitting anything, judge only on data the model never touched, and check every feature for whether you would actually have it at prediction time.
Duplicates and near-duplicates leak too
Group leakage's cousin: exact or near-duplicate rows split across train and test. The same product listing scraped twice, the same image at two resolutions, an augmented copy of a training example, the same document under two IDs — any of these landing on both sides means the model is tested on something it effectively trained on. Deduplicate (and near-deduplicate on a similarity key) *before* splitting, and make sure augmentation happens *after* the split so an original and its augmentations never straddle the boundary.
The target-leakage taxonomy is bigger than one column
"A feature that is the label in disguise" comes in several forms worth naming: post-outcome features (computed after the event), label-derived features (a transform of the target), proxy labels (an innocent column that's a near-perfect stand-in, like a case-ID range that encodes the outcome), aggregation-window leakage (a rolling stat whose window reaches past the prediction time), and future-window leakage (any feature summarising events after T). The audit question is always the same — *would this exact value exist at prediction time, knowing nothing about the future?* — but the disguises are many.
Test-set reuse and nested CV
Leakage isn't only about features — it's also about *decisions*. Every time you tune against the same validation/test set — trying architectures, thresholds, feature sets and keeping whatever scores best — you leak your own choices into it, and the reported number drifts optimistic (you've overfit to the test set through the back door). Guard it: touch the test set once, at the very end. And when you tune hyperparameters or features aggressively, use nested cross-validation — an outer loop for the honest estimate, an inner loop for all the tuning — so the reported score reflects data the selection never saw.
Time-series CV: windows, gaps, backtesting
A single time cutoff is the start; robust temporal validation has more structure. Expanding-window CV trains on all history to date and tests the next slice (more data, stable relationships); sliding-window trains on a fixed recent span (better under drift). Add a gap/embargo between train and test so rolling-window features can't bleed across the boundary. And backtest across multiple cutoffs rather than one — a strategy that works at one date and fails at three others isn't real. One split is an anecdote; a backtest is evidence.
Match the split to the production question
The split should mirror what you'll actually predict. Deploying to new users? Split by user (group split). Predicting new sessions for existing users? Split by session. Scoring new transactions, new products, or future events? Split accordingly — by transaction, by product, or temporally. The distinction between entity-level and event-level deployment decides the split: if production always sees brand-new entities, your validation must too, or your number answers a question you'll never be asked.
Feature selection leaks, and features must survive to serving
Two final structural traps. Selection leakage: correlation filtering, mutual-information ranking, PCA, RFE, and any *target-based* feature selection must happen inside the CV folds — pick features using the full data before splitting and you've let the test set influence which features exist. And production parity: every feature you keep must be *computable at prediction time* with the same freshness, latency, and timestamp constraints as offline — a feature that's trivial to compute over historical tables but unavailable (or stale) in the real-time path is a leak that only surfaces after deployment. Feature selection itself — which method to use, wrapper versus filter tradeoffs, and stability across resamples — is deep enough to earn its own treatment: that's exactly where the next module, Feature Selection, picks up.
Key points
- Use group-based splits whenever examples share an entity — patient, user, household, time series. A random split on a medical dataset with ten records per patient puts the same patient in both train and test; validation measures how well the model memorizes patients, not how well it generalizes to new ones. Group k-fold assigns all of a given patient's records to a single fold. This is non-negotiable if entity-level generalization is what you are deploying for.
- The most common production trap: fitting preprocessing transformers outside the cross-validation loop. Fitting a StandardScaler or SimpleImputer before the loop means its statistics were computed on data that includes every validation fold. Each fold's validation data contaminated the scaler. The correct order: inside each fold, fit all transformers on the training portion, apply fitted transformers to validation. Use sklearn Pipeline to make this structurally impossible to get wrong.
- Diagnose leakage with a feature correlation audit and a single-feature accuracy test. Before training on a new feature: (1) check its correlation with the target — above 0.8 on a complex real-world problem is suspicious; (2) train a model using only that single feature and check accuracy — suspiciously high single-feature performance often indicates label derivation; (3) verify the feature's computation timestamp is strictly before the label timestamp in your data pipeline. A feature that causes a 15+ point accuracy jump in isolation is almost certainly leaking.
- Deduplicate before splitting, expand the target-leakage taxonomy, and don't reuse the test set. Exact/near-duplicate rows (rescraped listings, augmented copies, same doc under two IDs) split across folds leak like group leakage — dedupe first and augment after the split. Target leakage includes post-outcome features, label-derived features, proxy labels, and aggregation/future-window leakage, all caught by "would this value exist at prediction time?" And test-set reuse is decision leakage: tuning repeatedly against the same set drifts the metric optimistic, so touch test once and use nested CV when tuning aggressively.
- Do temporal CV properly, match the split to deployment, and select features inside folds. Use expanding or sliding windows with a gap/embargo and backtest across multiple cutoffs, not one. The split must mirror the production question — new users → group split, new sessions → session split, future events → temporal split (entity-level vs event-level deployment decides it). Feature selection (correlation/MI/PCA/RFE/target-based) must run inside CV folds, and every kept feature must be computable at serving time with the same freshness and timestamp constraints or it's a leak that surfaces only after deploy.
Leakage fires no error and produces no warning — the model trains cleanly, metrics are excellent, and the system ships. It fails when real data arrives. The only protection is structural: enforce the split before any transformer is fit, audit every feature for temporal validity, and never reuse the test set.
Recap
- Leakage fires no error: the model trains cleanly, metrics look excellent, it ships — and fails on real data.
- Group-split shared entities (patient, user, household) — a random split with 10 rows/patient measures memorization, not generalization.
- Fit preprocessing inside each CV fold on the training portion only — scalers and imputers leak distributional stats, target encoders leak the label itself; use a Pipeline so it's structurally impossible to get wrong.
- Dedupe before splitting — exact/near-duplicate rows across folds leak like group leakage; augment after the split.
- Target-leakage test: "would this value exist at prediction time?" catches post-outcome, label-derived, proxy, and future-window features.
- Never reuse the test set — repeated tuning drifts the metric optimistic; touch test once, use nested CV when tuning hard.
- Do temporal CV with structure: expanding or sliding windows, a gap/embargo so rolling features can't bleed across the boundary, and backtest across multiple cutoffs — one split is an anecdote.
- Match the split to deployment: new users → group split, new sessions → session split, future events → temporal split.
- Select features inside CV folds, not before: correlation/MI/PCA/RFE/target-based selection on the full data before splitting lets the test set influence which features exist; every kept feature must also be computable at serving time with production-matching freshness.
Check your understanding
Q1. You build a model to predict customer churn. You include 'support_tickets_after_churn_date' as a feature. The model achieves 98% accuracy. What is the problem?
- A) Label leakage: this feature can only be known AFTER the customer churns, so it's essentially the answer in disguise. For a currently active customer it's unavailable, and accuracy collapses once deployed.
- B) The model is simply overfitting to the raw support-ticket count, which is an inherently noisy signal — the fix is adding L2 regularization and capping the feature at the 95th percentile.
- C) The 98% accuracy is actually completely legitimate here — support-ticket behavior is a genuinely strong predictor of churn intent, since dissatisfied customers naturally submit more tickets before leaving.
- D) The feature mainly introduces multicollinearity, since support-ticket count correlates with tenure, causing the model's coefficient estimates to become unstable and the reported 98% accuracy unreliable.
Q2. Why does random train-test splitting fail for time-series forecasting, and what is the correct splitting strategy?
- A) Random splitting fails mainly because it changes the class balance between the train and test sets; time-series data instead requires stratified splitting by time period to preserve the distribution.
- B) Random splitting fails because time-series data carries autocorrelation, so randomly shuffled examples violate the independence assumption most ML algorithms rely on, inflating the reported accuracy.
- C) Random splitting shuffles examples across time, so the model might train on March data to predict January — backwards, since March data doesn't exist yet. Correct: temporal split, train on 0 to T, validate afterward.
- D) Random splitting fails mostly because weekly, monthly, and annual seasonality patterns get split across train and test, so the model learns incomplete cycles that never generalize to full ones in production.
Q3. You run 5-fold cross-validation on a medical imaging dataset with 500 patients and 20 images per patient. You get CV accuracy of 94%. You deploy and get 71%. What happened?
- A) The 5-fold CV setup simply used too few folds — with 500 patients, 10-fold or full leave-one-out CV is required for a genuinely unbiased estimate, and 5 folds optimistically inflates it.
- B) Group leakage: the random split put different images of the same patient in different folds — the model learned patient traits and validated on patients it already knew. Fix: grouped k-fold.
- C) The model was trained on a dataset drawn entirely from one scanner, but the deployed model encounters images from different scanners — a distribution-shift problem unrelated to the splitting strategy.
- D) The 94% CV accuracy was computed largely on augmented images during training, while the 71% production number reflects the model's true performance on unaugmented images once deployed.
Q4. List three preprocessing operations that can cause leakage when applied before the train-test split.
- A) Feature selection, hyperparameter tuning, and model selection all cause leakage specifically when performed repeatedly before the split, because they use test-set performance to guide decisions.
- B) Outlier removal, duplicate detection, and missing-value imputation all cause leakage because each one uses global dataset statistics that structurally incorporate rows from the held-out test set.
- C) Log transformation, binning, and interaction-feature creation all cause leakage because they change the statistical properties of training features based on the entire dataset's distribution.
- D) (1) Scalers fit on the full dataset reflect test statistics. (2) Imputers' fill values are influenced by test rows. (3) Target encoding leaks test-set labels into every row's encoded value.
Q5. What is the correct order of operations for preprocessing inside a k-fold cross-validation loop?
- A) For each fold: fit transformers on training indices only, transform both sides with that fit, then train and evaluate — validation uses only training-derived parameters.
- B) Fit all preprocessing transformers on the full training set once before the CV loop begins, then just transform each fold's train and validation portions inside the loop and evaluate normally.
- C) Inside each fold, first evaluate the model on raw, unprocessed validation data to get a baseline score, then fit transformers on the training fold and report the improvement over that baseline.
- D) Fit all preprocessing transformers exactly once on the entire combined dataset — train plus test — before any splitting happens, to keep feature distributions consistent across every fold.
Q6. You aggressively tune hyperparameters and select features by repeatedly checking performance on the same held-out set, then report that set's score as your final number. Which TWO of the following are true?
- A) It's not optimistic at all — a held-out set is completely immune to overfitting no matter how many times or how aggressively you evaluate models and tune choices against it.
- B) Repeatedly making choices — architectures, thresholds, feature sets — based on the same held-out set leaks your decisions into it, so its reported score gradually drifts above true performance.
- C) The fix is to touch the final test set only once, and when tuning aggressively, use nested cross-validation — an inner loop for tuning, an outer loop for the honest estimate.
- D) The fix is to report training accuracy instead of any held-out metric, since training accuracy is structurally never affected by how many times the test set gets reused.
Q7. You augment your image training set (rotations, crops) and also deduplicate, but do both before the train/validation split. Why can this still leak, and what's the correct order?
- A) It genuinely can't leak here — augmentation and deduplication are both standard preprocessing steps that are always completely safe to run before any train/validation splitting occurs.
- B) Augmenting before the split lets copies of one image land on both sides, so validation gets near-duplicates. Correct: dedupe first, split by image, augment training only.
- C) The leak comes only from the random crops, not the rotations, so cropping specifically should be removed while rotation augmentation is safely kept before the split happens.
- D) There's honestly no real fix for this — image augmentation always leaks in some form, so the only safe option is to never augment image data at all, under any circumstances.
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →