Data Leakage: The Eleven Types and How to Detect Each

Data leakage is the single most common reason an offline 0.95 AUC becomes a production 0.65. Most people know two or three kinds of leakage. There are at least eleven. A senior MLE needs to recognise each on sight — because if you do not catch them before deployment, you will see them in production, and they will be much harder to diagnose then.

Data leakage is when information that would not be available at production prediction time bleeds into the training or validation data, making the model look better offline than it can possibly be online. The naive definition — "the target is in the features" — covers maybe one tenth of the cases. The actual taxonomy is broader and the production failures are more varied.

1. Target leakage

The classical case. A feature contains information about the target. For a fraud model, a feature like "is_disputed" is a target leak — disputes are filed only after fraud is confirmed. The model achieves near-perfect offline metrics by reading the dispute outcome. In production, at scoring time, no dispute has been filed, the feature is missing or default, and the model collapses. Detection: audit every top-importance feature for "was this value generated before or after the label was observed?" If you cannot answer with certainty, you cannot ship.

2. Temporal leakage

You have time-ordered data and you use random k-fold cross-validation. Now your training folds contain examples from after your validation fold. The model has effectively seen the future. Feature values that depend on aggregates ("user's average purchase amount over the last 30 days") now include data from beyond the validation timestamp. Detection: walk-forward cross-validation should give roughly the same number as your random k-fold; if walk-forward is much worse, you have temporal leakage.

3. Train-test contamination

You ran preprocessing on the entire dataset before splitting into train and test. Now the test set's distribution influenced the training pipeline (through scaling parameters, imputation values, target encoding, etc.). The model has implicitly seen the test set. Detection: re-fit all preprocessing inside the train fold only; if test metrics drop significantly, you had contamination.

4. Group / entity leakage

The same logical unit (a user, a customer, a household) appears in both train and validation, even though no individual row is duplicated. The model learns user-specific patterns from training rows of that user and applies them to validation rows from the same user. It looks like generalisation; it is memorisation of known users. Detection: group-aware cross-validation (group-k-fold by user_id, household_id, etc.); if group-aware metrics are much worse than random k-fold, you had entity leakage.

5. Aggregation leakage

You compute an aggregate feature (mean target by category, time-since-event by user) using the full dataset. The aggregate then leaks information from all rows into each row. Even after train-test split, every row's aggregate feature was computed using both halves. Detection: compute aggregates inside the training fold only, then use the training-fold aggregate to score validation/test rows; or use a leave-one-out aggregation that excludes the row itself.

6. Feature availability leakage

A feature is in your training data because someone backfilled it. At production prediction time, that feature is computed by an asynchronous pipeline that lags by hours. The model is trained on values it will never see in time at inference. Detection: trace each feature's serving-time availability; for time-sensitive predictions, audit whether the feature will exist at the prediction timestamp.

7. Preprocessing leakage (the fit-before-split bug)

You fit a StandardScaler, OneHotEncoder, or any other transformer on the full data, then split. The transformer's parameters were learned from data that includes the test set. The fix is sklearn's Pipeline with fit applied only inside the training fold of each cross-validation split. This is the bug that ships every junior ML engineer's first production model.

8. Label-window leakage

You define the positive class as "user churned within 30 days." For training, you take a user, look at the next 30 days, and label. For features, you also accidentally use data from those 30 days (the period during which churn was being measured). The model now has information from inside the label window. Detection: visually map the timeline — features must come from strictly before the prediction timestamp; the label must come from strictly after; the prediction timestamp must be in the middle.

9. Feature store leakage

The most modern leakage type. Your feature store has a single row per entity (user) that gets updated whenever the entity's underlying data changes. At training time, you fetch features "as they exist now." But the feature values reflect post-event updates — the user purchased something, the feature got updated to reflect the purchase, then you trained on that updated value to predict whether the user would purchase. Detection: point-in-time correct joins. Your feature store needs versioning by timestamp, not "current value." The training data must be assembled by asking "what was this feature's value at the prediction timestamp" — never "what is it now."

10. RAG / evaluation set leakage

In LLM applications, your retrieval index was built using the same documents that appear in your evaluation set. When the model is evaluated, it retrieves the exact answer document and rephrases it. The metric (BLEU, ROUGE, accuracy) is sky-high; the model has merely learned to use the index lookup. Detection: hold out a fully separate document corpus from the index and test only on questions whose answers are in that held-out corpus.

11. Selection-bias leakage (the survivorship trap)

The training data was filtered by a process that depends on the label. Customers who made a successful purchase enter the training set; customers who churned before purchasing do not. The model learns patterns specific to surviving customers, who are not the population it will be applied to. Detection: enumerate the data-collection steps; for each step, ask "does this step depend on the outcome we are trying to predict?" If yes, you have selection bias; the training distribution does not match the deployment distribution.

WARNING — Production tell: "Champion model from last month is still better." If your offline metric says the new model is significantly better, but the A/B test in production shows no improvement or worse, 80% of the time it is leakage. The leakage gave you the offline lift; production strips it away. The diagnostic sequence: (1) compute walk-forward CV — if much worse, you have temporal leakage. (2) Compute group-k-fold CV — if much worse, you have entity leakage. (3) Run a leak-check: audit every top feature for whether it could plausibly depend on the label. (4) Audit preprocessing — was anything fit on the full dataset? (5) Trace serving-time availability — is every feature actually present at prediction time? If all five pass, the lift is probably real and the A/B is detecting noise or a serving bug.

The single most useful discipline

Point-in-time correctness. For every feature in your training data, you must be able to state: this feature's value was computed using only information available before the prediction timestamp. If you cannot state this for every feature, you have leakage of some kind. Most large companies have moved to feature stores precisely because point-in-time correctness is too hard to enforce manually at scale.

Interview questions on this topic

"You discover that your top feature in a churn model is 'days_since_last_login.' Is this safe?" — Possibly not. It depends on when the feature is computed relative to the churn label. If churn is defined as "no login for 30 days" and the feature is "days since last login as of label time," then the feature has trivially perfect predictive power because it definitionally equals 30 for every churned user. You have label-window leakage. The fix is to compute the feature at a fixed lookback before the prediction window (e.g., "days since last login as of T-30, predicting churn in [T-30, T]"), not at T itself.

"Walk through point-in-time correct join semantics for a feature store." — At training time, for each training example, you fetch features as they existed at the prediction timestamp of that example. The feature store stores versioned values: for each entity (user), each feature has a series of (timestamp, value) tuples representing every update. The training join becomes an asof join: for entity X at time T, return the latest feature value with timestamp <= T. This is computationally expensive, which is why feature stores are non-trivial infrastructure. It is also the only correct way to assemble training data from a live feature store.

"You find evidence of leakage in a model in production. What is the right escalation?" — Pause the model. Compute the leak-free metrics on the existing validation set (fit preprocessing inside folds, compute aggregates inside folds, use walk-forward CV, audit point-in-time correctness). Report the corrected metrics to stakeholders. Decide whether the model is still net positive at the corrected metrics — sometimes leaked models are still better than the previous champion at the leak-free metric. If so, ship with disclosure. If not, roll back and revisit the feature design.

"Is cross-validation enough to prevent leakage?" — No. Cross-validation prevents the model from being fit on the test set, but it does not prevent target leakage (a feature that contains the answer), feature-availability leakage (a feature that will not exist at production scoring time), or feature-store leakage (a feature computed using post-event data). Cross-validation is necessary but not sufficient. Feature audit is the complement.

Try on Colab: download a public dataset with timestamps (e.g. the Brazilian e-commerce Olist dataset). Build a churn model. Deliberately introduce three of the eleven leakage types listed above — target leakage (add a feature that depends on the post-churn outcome), temporal leakage (random k-fold instead of walk-forward), aggregation leakage (compute user-level aggregates over the full dataset). Show how each type inflates offline AUC and how the diagnostic sequence above catches each one. This is the practical equivalent of memorising the taxonomy.