Feature Stores: Why They Exist and What Point-in-Time Correctness Means

Feature stores solve two problems that do not exist in academic ML: (1) features computed in batch for training must be computed in real time for serving, consistently; (2) historical features used to train a model must not "leak" future information. Point-in-time correctness is the second problem. It is subtle, it is widespread, and when violated, your model looks better in offline evaluation than it performs in production.

Feature engineering in production involves two distinct challenges. The first is operational: the same feature (user's average purchase value over the last 30 days) must be computed in batch for training and in milliseconds at serving time. The second is statistical: when training on historical data, the features used to predict an outcome at time T must reflect what was known at time T, not what was known later. Feature stores are the infrastructure that solves both.

The training-serving skew problem

Training data is computed by a data engineer in SQL running over the full historical dataset. Serving features are computed in real-time by a feature computation service. If these two systems use different code, different data sources, or different aggregation windows, the features at serving time will not match what the model was trained on. The model has learned from features it will never see in production.

Common causes: SQL uses a different timezone than the real-time system; batch aggregations use slightly different window boundaries; NULL handling differs between Pandas and the serving library; category encoding was fit on training data and not persisted to serving. Any of these causes silent model degradation — you only discover it by comparing feature distributions between training and serving.

The point-in-time correctness problem

When training on historical data, you construct (features, label) pairs for past events. The label at time T might be "did the user churn in the 30 days after T?" The features should reflect what was known at time T — not at T+30 or at T+60. If your feature pipeline accidentally pulls in data from after the label window, the model has access to information it could not have had in production. This is a form of data leakage that produces overoptimistic offline metrics.

Example: a user-level feature "total purchases in the user's history" is queried at the time of training (say, 2024-01-01). But the event you are labelling happened on 2022-06-01. The feature value at training time includes purchases made between June 2022 and January 2024. At serving time, you can only use purchases up to June 2022. The model trained with future information performs worse in production than offline evaluation predicted.

Point-in-time joins: the solution

A point-in-time join retrieves feature values as they existed at the time of each training event, not at the time of training generation. For a training event (user_id, event_timestamp), the join retrieves the feature value from the most recent feature snapshot before event_timestamp. This requires: storing historical feature values with timestamps, not just the current value; implementing an efficient as-of join that retrieves the correct historical value for each event.

Feature stores (Feast, Hopsworks, Tecton, Databricks Feature Store) implement point-in-time joins as a core primitive. Without a feature store, implementing correct historical lookups requires careful custom SQL and is a frequent source of training-serving skew.

Online vs offline stores

Offline store: a columnar data warehouse (Parquet on S3, Delta Lake, BigQuery) holding historical feature values indexed by entity_id and timestamp. Used for training data generation. Query latency: seconds to minutes. Example: historical feature snapshots written every hour.

Online store: a low-latency key-value store (Redis, DynamoDB, Bigtable) holding only the current feature values indexed by entity_id. Used for real-time serving. Query latency: < 5ms. Example: "user_123: {avg_purchase_30d: 42.5, session_count_7d: 3}".

The offline store enables point-in-time training. The online store enables real-time serving. The feature store ensures both stores are populated from the same computation logic.

The fresh feature problem

Some features are highly time-sensitive: user's last action, current cart contents, real-time search query. These cannot be precomputed — they must be computed on-the-fly at serving time and passed directly to the model. Feature stores handle precomputed features; real-time features are passed as request context. The model input at serving time is: precomputed offline-computed features from the online store + fresh request-context features. Managing this boundary correctly is a common engineering challenge.

Try on Colab: use Feast (open-source feature store) with a small synthetic dataset. Define a feature view for user purchase statistics (avg_30d, count_7d). Materialize historical features to an offline store (Parquet). Use get_historical_features with point-in-time joins to generate training data for events at different timestamps. Compare the resulting features to a naive join that ignores time — observe the data leakage.