ML Platform Design
Feature store, training infra, model registry, serving, observability
Two teams build separate models and both need "user purchase history." One computes it in Spark, the other in Python, with slightly different null-handling and timezone logic. Their models already see *different* values for the "same" feature at training time — and at serving time the real-time versions drift further apart. A third team starts building the same feature for a third model. Three teams, three subtly different implementations, each debugging its own copy of one problem.
What a platform actually fixes. It makes the *feature* the shared thing: computed once, registered centrally, consumed by any model. Training pulls point-in-time-correct history; serving pulls the identical computation at low latency. One definition, one source of truth, no three-way skew. A model registry with versioning and lineage turns rollback from a manual artifact hunt into one API call — it connects a deployed model to its training run, data, and metrics.
But it only pays off at scale. For one model and two data scientists, hand-rolled infra is genuinely simpler — build the platform then and you've built abstraction with no users. For ten models across teams, the platform amortizes fast and the hand-rolled path becomes a coordination disaster. The skill is knowing which side of that line you're on; the common failure is building platform infrastructure before any model is in production.
Key points
- The feature store solves coordination: features computed once, registered, consumed with point-in-time correctness. Without it, each team reinvents features with subtle divergences (nulls, timezones, windows), and training-serving skew appears when models ship. Debugging skew across three implementations is a week of engineering per incident. Point-in-time correctness also blocks a specific failure mode: a training row for event time t must only be enriched with feature values computed by t — a feature computed after t (for example, a 7-day aggregate that includes days after the label) leaks future information into training and inflates offline metrics.
- A model registry with versioning and lineage is what makes rollback one API call. It links the deployed model to its training run, data, and eval metrics. The first production degradation without a registry turns into a multi-day investigation.
- Build the platform at ~5–10 models across teams, not before. For 1–2 models, MLflow/W&B + FastAPI + git-versioned SQL features + manual dashboards is simpler and faster. Premature platform-building is abstraction with no users to amortize it.
An ML platform is a coordination technology, not a modeling one: the feature store kills training-serving skew by making each feature a single point-in-time-correct definition, and the registry makes rollback one call — but both only amortize past roughly 5–10 models, so building them for the first model is premature abstraction.
Recap
- The problem a platform fixes is coordination, not modeling: N teams each reimplement "user purchase history" in Spark, Python, SQL with subtly different null-handling and timezone logic — so their models see *different* values for the "same" feature at training time, drift further apart at serving, and each team debugs its own copy of one problem.
- The feature store makes the feature the shared thing: computed once, registered centrally, consumed by any model. Training pulls point-in-time-correct history; serving pulls the identical computation at low latency — one definition, one source of truth, no three-way skew. Point-in-time joins specifically prevent future-leakage (a 7-day aggregate that includes days after the label).
- A model registry with versioning + lineage makes rollback one API call: it links the deployed model to its training run, data, and eval metrics. Without it, the first production degradation becomes a multi-day manual hunt for the last-good artifact.
- It only amortizes past ~5–10 models across teams: for 1–2 models and a couple of data scientists, MLflow/W&B + FastAPI + Docker + git-versioned SQL features + manual dashboards is genuinely simpler and faster.
- The classic failure is building the platform before any model ships — abstraction with no users to amortize it. The skill is knowing which side of the break-even line you're on.
Check your understanding
Q1. Select the two correct statements about whether a 3-data-scientist startup shipping its first model should build a full ML platform now.
- A) Not yet — MLflow/W&B tracking plus FastAPI + Docker serving and git-versioned SQL features cover this stage well.
- B) Platform infra amortizes past roughly 5–10 production models, or once a large share of engineering time goes to tooling.
- C) A feature store must be built before any model exists, because retrofitting it later is provably impossible.
- D) A model registry and full serving infra are mandatory for any production deployment, regardless of team size.
Q2. What specific failure does point-in-time correctness in a feature store prevent?
- A) Model overfitting caused specifically by having too many correlated features present in the training set.
- B) Label leakage from the future: a training row for event time t gets enriched with feature values computed after t, inflating offline metrics.
- C) Slow feature retrieval at serving time, caused specifically by unindexed lookups against a poorly-configured, legacy online feature store deployment.
- D) Embedding staleness inside the ANN retrieval index, unrelated to how the feature-store joins are performed.
Q3. A production model's quality silently dropped last night. With a model registry + lineage, what does the response look like versus without one?
- A) Identical either way — the registry only stores artifacts and doesn't help diagnose or roll back a live regression.
- B) With a registry you compare the served version to the previous one and roll back with one call; without it, you manually hunt for the last-good artifact.
- C) Without a registry it's actually faster overall, because engineers are forced to fix forward instead of wasting time rolling anything back.
- D) The registry only matters for compliance audits and internal paperwork trails, not for actual live production incident response or postmortem write-ups either.
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →