Group-Level Contamination: The Leakage Nobody Catches Until Production

Your train/test split is random. Your features are computed correctly. Your evaluation still lies. The reason: you split rows, but your data is grouped — multiple rows per user, per patient, per product. When the same entity appears in both train and test, the model memorises entity-level patterns and your validation AUC reflects that memory, not generalisation.

Group-level contamination is the leakage form that survives all the standard preprocessing hygiene checks. You split before engineering. You fit scalers on train only. You avoid target encoding on the full dataset. You still have leakage — because the split itself is wrong.

The mechanism

Your dataset has 50,000 rows from 8,000 users. Each user appears an average of 6 times. A random 80/20 split puts roughly 5 rows from user_id=12345 in training and 1 row in validation.

The model learns user-level patterns — average spend, historical churn signals, engagement velocity. On the validation row for user_id=12345, it essentially recognises the user. It does not generalise to new users; it memorises existing ones.

Your validation AUC is 0.89. In production, the model sees user_id=99999, a user it has never encountered. The AUC drops to 0.74. The gap is not model complexity or hyperparameters. It is the split.

Where this appears in recommender systems

In recommender systems, this is the standard failure mode of item-based collaborative filtering evaluation. A random split across user-item interactions means both train and test contain interactions from the same users and the same items. The model learns user embeddings and item embeddings that perfectly predict held-out interactions — because it has seen every user and every item before.

The correct evaluation: a leave-one-user-out or leave-one-item-out split. Users in validation are a disjoint set from users in training. Or: time-split, where validation contains interactions after a cutoff date for all users.

The correct fix: GroupShuffleSplit

sklearn provides `GroupShuffleSplit`, which accepts a `groups` parameter (your entity ID array) and guarantees that all rows from a given entity are in the same split. Usage:

```python from sklearn.model_selection import GroupShuffleSplit gss = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42) train_idx, val_idx = next(gss.split(X, y, groups=df['user_id'])) ```

After this split, every user in validation is a user the model has never seen during training. This is the honest evaluation.

When group leakage is acceptable

Sometimes you want the model to leverage known-entity signals. A fraud model that flags known bad actors is legitimate if your production use case also covers those same actors. In this case, group leakage in validation is intentional and your metric is: "how well does the model perform on entities it has seen before?"

But you still need a second evaluation: "how well does the model perform on new entities?" Both evaluations should be reported separately. Conflating them gives a metric that means neither thing precisely.

Cross-validation with groups

For cross-validation, use `GroupKFold`. Each fold assigns complete groups to train or test — no entity spans two folds.

```python from sklearn.model_selection import GroupKFold gkf = GroupKFold(n_splits=5) for train_idx, val_idx in gkf.split(X, y, groups=df['user_id']): # train and val share no user_ids ```

The model's cross-validated AUC now reflects generalisation to unseen entities — which is what production performance actually is.

The diagnostic question to ask of any dataset before splitting: Does this dataset have repeated observations from the same entity? If yes, which evaluation matters more — performance on known entities or performance on new entities? That answer determines your split strategy.