Late-Arriving Data and the Retroactive Feature Trap: How Pipeline Corrections Corrupt Training Sets

Your feature pipeline runs nightly and reprocesses yesterday's data to catch late-arriving events. This is good operational practice for data completeness. It is a silent catastrophe for model training. Every reprocessing run retroactively alters the features your historical training rows will see — and the model you train on Monday's data is systematically different from the model you train on Friday's reprocessed data.

Late-arriving data is a normal operational reality in data engineering. Transactions settle over 24–72 hours. User activity logs arrive with delay. External data sources batch-deliver with variable latency. The standard response: run a late-data correction job that reprocesses recent windows and backfills updated values.

This response is correct for data completeness. It is a training data integrity problem that most teams do not notice until a model retrain produces unexpectedly different results.

The retroactive corruption mechanism

You have a feature `transactions_last_7d` for each user, computed by a nightly batch job. On 2024-01-15 at 02:00 UTC, the job runs and computes features for 2024-01-14. It writes feature rows with timestamp 2024-01-14.

Some transactions from 2024-01-13 and 2024-01-14 arrive late — they were delayed in transit from a payment processor. On 2024-01-16, your late-data correction job reprocesses the 2024-01-14 window and rewrites the feature rows, now including the late transactions.

The feature row in your store for timestamp 2024-01-14 now reflects reality as of 2024-01-16 — not reality as of 2024-01-14 at 02:00 UTC when it was first computed.

Why training is inconsistent

When you train a model on Monday, you generate your training dataset by reading the feature store. You get features reflecting the state as of Monday — including all retroactive corrections up to Monday.

When you retrain the same model on Friday with "the same training window," you generate a new training dataset. The feature store now reflects corrections up to Friday. Some rows have different feature values than they did on Monday. The model trains on different data. Its weights are different. Its decision boundary is different.

You cannot reproduce Monday's model on Friday. The training data has been altered underneath you. This is not drift — the production distribution has not changed. The training data is literally different.

The compounding effect

Over a 12-month training window, late-data corrections may affect 15–30% of training rows. The corrections are not random — they disproportionately affect high-activity users (more transactions to late-arrive) and recent time windows (more recent corrections have had less time to finalise). Your model learns a systematically biased picture of those user segments.

In a credit scoring context: users with many transactions are high-activity customers. If their features are consistently overestimated (because late transactions always add to their totals), your model learns that high transaction volume is predictive of creditworthiness by a larger margin than it actually is. The bias is invisible in offline metrics because your validation set has the same correction bias as your training set.

The two patterns that prevent this

Pattern 1: Immutable feature rows with a write timestamp.

Never overwrite a feature row. Instead, append a new row with the corrected value and a new `availability_timestamp`. Your point-in-time join retrieves the row that was available at training time — the original, uncorrected value. Training is reproducible because it always reads the as-of-training-time snapshot, not the current snapshot.

Pattern 2: Correction awareness in feature computation.

Accept that features will be incomplete at first computation. Compute features with an explicit `completeness_timestamp`: the earliest time at which this feature value is considered final. Your training pipeline only uses rows where `event_timestamp + correction_window <= training_run_timestamp`. You exclude the most recent data from training until it is final.

How to detect retroactive corruption

Run your training pipeline twice with a 7-day gap between runs, using identical training window bounds. Compute the feature-level correlation between the two datasets for matching entity-timestamp pairs. If feature values differ for more than 2% of rows, your pipeline has retroactive corrections altering the training data.

Also: if retrained models show unexpected performance divergence from their predecessors with no apparent distribution shift in production, retroactive feature corruption is a primary suspect.