Feature Store Architecture

Offline store, online store, registry, materialisation, latency SLAs

Five ML teams at one company all need the same thing: "user average spend in last 30 days." Each team builds its own pipeline. Five nightly SQL jobs. Five Redis keys. Five different implementations of the 30-day window, each with its own null handling, timezone quirks, and new-user edge cases. A user spends ten thousand dollars at midnight, and each team's copy of the feature updates at a slightly different time with a slightly different number. The fraud model and the recommendation model now disagree about this user's spending. Neither team can see the other's value. Neither is wrong by its own logic. Yet they are quietly inconsistent, and both models suffer for it.

A feature store fixes this by computing each feature *once,* correctly, and serving that one answer to everyone — through two storage backends built for two very different jobs.

The offline store: history, for building training sets

It keeps every past value of a feature, stamped with time. That is what lets you ask, "What was this user's 30-day average spend *as of* time T?" — the point-in-time query that prevents leakage. Without it, your training pipeline reaches for *today's* values when building rows for last month's events, and every rolling aggregate silently swallows data that didn't exist yet when the prediction would have been made.

The online store: the current value, fast

A live fraud request needs this user's spend average in under 5ms. The online store (Redis, DynamoDB, Cassandra) holds only the *latest* value per user and returns it at memory speed. No history — that's the offline store's job. It's sized for latency at peak traffic, not for storage.

Online-store latency failure modes. The online store's P50 latency can look fine while P99 spikes badly, and the usual causes are specific, not vague "network blips": a hot key (one popular entity — a viral post, a high-volume account — absorbing a disproportionate share of reads, so requests queue up behind it), memory pressure that forces the store to evict cached keys under load, or an oversized serialized feature vector that's slow to deserialize on every read. The fixes track the causes directly: hash or shard hot keys so no single partition takes outsized traffic, add capacity so eviction pressure eases, and put a circuit breaker in front of the store so a P99 spike degrades gracefully instead of cascading into the caller.

The registry and materialization: the parts a plain database lacks

The *registry* is the governance layer: it records each feature's definition, owner, freshness SLA, upstream dependencies, and which models consume it — so teams can find what already exists and know what breaks if a pipeline is retired. *Materialization* is the act of computing features from raw data and writing them to both stores: batch (Spark, Airflow, hourly/daily) when some staleness is fine, or streaming (Kafka → Flink → Redis) when 5-minute freshness matters, like fraud or live inventory. Each write to the online store is typically stamped with a TTL a little longer than the materialization interval -- insurance against a value going stale forever if a job stalls. But that insurance has a sharp edge: if materialization stops for good (a pipeline decommissioned, a job silently failing) the online store keeps serving its last computed value as if nothing were wrong, right up until that TTL lapses -- then the key vanishes and a lookup returns null. Serving code almost always treats a missing key as "impute the default," not "raise an error," so the model quietly starts scoring on a placeholder value with no exception anywhere in the stack. No error in the logs is not evidence a feature pipeline is healthy -- only an explicit freshness check is.

And that's the real answer to "isn't this just a database?" A database stores bytes. A feature store adds four things a database won't: point-in-time-correct history, one shared computation path across online and offline, a registry for discovery and lineage, and materialization with freshness monitoring. With all four, training-serving consistency becomes a property of the *system* — not a hope resting on individual engineers remembering to match each other.

Key points

• Establish point-in-time join correctness in your training pipeline before building anything else — this is the hardest invariant to get right and the primary reason feature stores exist. Everything else in a feature store is optimization. The offline store's value is precisely that it can answer "what were this user's features at timestamp T" using only data available before T. Without this guarantee, training datasets contain temporal leakage, offline metrics are inflated, and the gap between evaluation and production performance is structural.
• Trap: materialization lag creates freshness gaps that can be catastrophic for time-sensitive applications. If a feature is computed hourly and a user event happens 50 minutes into the cycle, the online store has 50-minute-old data. For fraud detection, a user who just moved 100K dollars is invisible for up to an hour. Design materialization frequency based on the feature's staleness tolerance — and monitor actual freshness, not scheduled frequency. The schedule and the actual update time are not the same thing.
• Diagnostic: compare feature values your model sees at training time versus at serving time for the same user events. Any systematic difference — even small — is training-serving skew that will silently degrade accuracy. The dual-write pattern makes this check explicit: run the old pipeline and the new feature store simultaneously on real traffic, compare outputs on a sample of entities, and assert identity before cutting over. Do this before the feature store is in the critical path, not after.
• Trap: online-store P99 latency spikes have specific, diagnosable causes. The three usual suspects are a hot key (one popular entity absorbing a disproportionate share of reads, so requests queue behind it), memory pressure that forces the store to evict cached keys under load, and an oversized serialized feature vector that's slow to deserialize on every read. Fix each directly: hash or shard hot keys, add capacity, and put a circuit breaker in front of the store so a latency spike degrades gracefully instead of cascading into the caller.
• Trap: a dead materialization pipeline fails silently, not loudly. Online-store writes are typically set with a TTL a little longer than the materialization interval. If materialization stops for good, the store keeps serving its last good value until that TTL lapses -- then the key vanishes, a lookup returns null, and serving code almost always imputes a default rather than raising an error. A model can degrade for weeks with nothing in the logs to flag it; the only real defense is monitoring a feature_last_updated_timestamp against the freshness SLA, not waiting for an exception that will never come.

Recap

• Problem: five teams compute "30d avg spend" five ways → silent inconsistency, all models suffer.
• Fix = compute once, serve everyone, through two backends for two jobs.
• Offline store = history, stamped by time → point-in-time "as of T" queries that prevent leakage.
• Online store = latest value only, fast (Redis/DynamoDB/Cassandra), <5ms, sized for latency not storage.
• Registry = governance: definition, owner, freshness SLA, lineage, consumers.
• Materialization = compute + write both stores: batch (Spark/Airflow) or streaming (Kafka→Flink→Redis) when 5-min freshness matters.
• Not "just a database": point-in-time history + one shared path + registry + monitored freshness = consistency as a *system* property.

Check your understanding