The Feature Store Time-Travel Bug That Quietly Corrupts Your Models
You have a feature store. Your training pipeline reads features from it. You believe your training data is point-in-time correct. It isn't. The time-travel bug is the most insidious failure mode in ML data infrastructure, and it exists in almost every feature store deployment that hasn't explicitly tested for it.
Point-in-time correctness means: when you build a training example for an event at timestamp T, every feature value reflects the state of the world at time T — not T+1, not T+24h, not "whenever the batch job ran."
Almost every feature store tutorial demonstrates point-in-time joins. Very few explain how they silently break in production.
How the bug appears
Your feature store materialises features via a daily batch job that runs at 02:00 UTC. It reads all events from the previous calendar day and computes feature values. The features are stored with a timestamp of "2024-01-15" — the calendar date of the source events.
Your training pipeline does a point-in-time join: for each training event at timestamp T, fetch the most recent feature row where feature_timestamp <= T.
Looks correct. The bug: your feature computation job runs at 02:00 UTC on 2024-01-16 to process 2024-01-15 data. The feature row is available in your store starting at 02:00 UTC on 2024-01-16. But the feature_timestamp stored is "2024-01-15 00:00:00 UTC."
For any training event that occurred on 2024-01-15 after 00:00 UTC but before 02:00 UTC on 2024-01-16, your point-in-time join correctly finds the 2024-01-15 feature row. But those features were computed including data from the full day of 2024-01-15 — including data after the event. If a user made a purchase at 08:00 UTC on 2024-01-15, their "purchases yesterday" feature will include that very purchase. Leakage.
The four variants
1. Calendar-day vs event-time mismatch. Features stamped at midnight of the source date but containing full-day data. Training events from early in the day see future-contaminated features.
2. Processing lag hiding behind event timestamp. The feature row is available at T+N hours due to pipeline latency, but stamped at T. Any event between T and T+N uses a feature row that "didn't exist yet."
3. Late-arriving data correction. Your pipeline reprocesses yesterday's data today to incorporate late-arriving events. The corrected feature row overwrites the original but keeps the original timestamp. Historical training data is retrospectively altered.
4. Wall-clock time vs event time aggregation. A "sessions in the last 7 days" feature computed at 2024-01-15 23:55 UTC will include session data that arrived between the training event at 2024-01-15 10:00 UTC and the time the feature was computed.
How to detect it
For any feature that should reflect "state at time T," compute it independently from raw events for a sample of training rows. Compare against what your feature store served. Systematic overestimation of features for events early in the day = temporal leakage.
Also: if a model degrades faster after deployment than expected, but shows no feature distribution shift in PSI monitoring, temporal leakage is a prime suspect. The model learned from future-contaminated data and production data is uncontaminated.
The correct fix
Store features with their availability_timestamp — the time the row was actually written to the store — not the event timestamp. Your point-in-time join must use availability_timestamp <= event_timestamp, not feature_date <= event_date.
This requires materialising feature rows with accurate write timestamps, which means changing how your pipeline records metadata. Feast 0.28+ supports this with the ttl parameter and feature_view materialisation logs. Without this, you're doing "point-in-time" joins that are not actually point-in-time.
The test you should run
For 100 training events at time T, fetch the features your training pipeline used. Then independently compute those features using only events with timestamp < T. If they differ systematically, you have the bug.