Model Registry & Versioning

Artifact storage, metadata, lineage, deployment gating, experiment tracking

Your fraud model was updated four weeks ago. A new fraud pattern shows up and the model is missing it. Now you need answers, fast: when was it updated? What data trained it? What metrics did it hit? Who approved it? Can you roll it back in the next ten minutes? Without a model registry, every one of those is archaeology — digging through Slack, squinting at S3 timestamps, hunting down the engineer who ran the job. And the whole time, the business is eating the cost of a degraded model.

What a registry actually is

It's the governance layer sitting between training and production. It holds the model artifact and its full lineage — the exact dataset (with a content hash), the code commit, the feature versions used, the hyperparameters. It holds deployment history: which version went where, who approved it, what gate it passed. And it holds lifecycle state: Experiment → Staging → Production → Archived.

Weights alone are not the artifact

This is the part teams underbuild. A model trained on scaler-normalized features, deployed *without* that fitted scaler, will get raw inputs, treat them as normalized, and output confident garbage — no error thrown. So the registry must store the *complete inference artifact:* weights plus the fitted preprocessing pipeline plus the feature schema. One load, everything you need.

Experiment tracking is the registry's other job, upstream of deployment

Before a model ever reaches Staging, two data scientists might independently train candidate models with different hyperparameters. Experiment tracking — the piece of the registry that tools like MLflow implement — is what lets them collaborate without a meeting: both log every run's hyperparameters, metrics, and dataset version to the same shared experiment instead of a personal notebook. That experiment shows up in a comparison UI, sortable by any metric (validation AUC, F1, whatever matters for the task), and because each run captured its hyperparameters, metrics, and dataset version together, any run in the list can be reproduced exactly. Pick the winning run by sorting, then register only that one.

A deployment gate is a specific automated check, not a metaphor

"What gate it passed" means a concrete test run at promotion time: a schema-compatibility check (does this version's expected input schema match what the target environment will send it?), a metric threshold (does validation AUC clear the bar the last production model set?), or a required sign-off recorded in the registry. A promotion with no gate is a file copy with delusions of process. A promotion behind a gate is blocked automatically the moment a check fails, before a human has to notice the problem in production.

Multi-environment deployment: config is not weights, and it needs its own versioning

Dev, staging, and prod often run different data schemas — a column added in staging before prod catches up, a mocked field in dev that's real downstream. The fix is to keep environment-specific config (schema mapping, feature-flag state, resource limits) versioned separately from the model weights, and associate the triple (model_version, environment, config_version) at the moment of promotion, not baked in at training time. Every promotion into an environment runs that environment's schema-compatibility gate first, so a mismatch fails loudly at promotion instead of silently at inference.

Rollback is the part that's time-critical

When a model goes bad, you promote the previous version back to Production in a single API call. But only if that artifact still exists — delete it and your "rollback" becomes rebuilding from scratch mid-incident, hours instead of minutes. So never delete production artifacts; a year of storage costs less than one hour of a live incident.

That's the whole difference from "just an S3 folder." Plain storage hands you a file. A registry gives you an approval trail, deployment lineage (what changed between v7 and v8?), instant rollback, and a record of which model made which decision when. A named folder that relies on people staying disciplined under deadline pressure is not a registry — it's a filesystem with good intentions, and it fails you exactly in the first ten minutes of an incident, when you need those answers instantly.

Key points

• Register every model artifact before deployment — even for teams of one — and store the complete inference artifact: weights plus preprocessing pipeline plus feature schema. When something goes wrong in production, the registry is the first place you look. A model registered without its scaler requires manual reconstruction of preprocessing parameters during an incident. A model registered without its dataset hash cannot be compared to the previous model to identify what changed. Register everything, atomically, before the model ever touches production traffic.
• Trap: storing only model weights without preprocessing artifacts will cause a silent production failure the first time the scaler version or schema ordering changes. A StandardScaler fit on training data with mean=120, std=45, deployed without the fitted scaler, will receive raw feature values and interpret them as if they were normalized. The model has never seen inputs in that range. Predictions will be wrong with full confidence and no error. Serialize the fitted scaler inside the model artifact as a single sklearn Pipeline — the scaler travels with the weights and is loaded atomically.
• Diagnostic: attempt to reproduce a model registered 3 months ago using only the registry metadata. If you cannot, the registry is incomplete. The gaps you find are exactly where the next incident investigation will stall. Add dataset version tracking and code commit hash to every registration as mandatory fields, not optional ones. Lineage that is optional gets skipped under deadline pressure — which is exactly when you need it most.
• Experiment tracking (e.g. MLflow) and deployment gates are both registry features, not afterthoughts — and multi-environment deployments need their own versioned config. Experiment tracking logs every training run's hyperparameters, metrics, and dataset version to a shared, sortable comparison UI, so two scientists can pick a winner without a meeting. A deployment gate is a concrete automated check — schema compatibility, a metric threshold, or a required sign-off — run at promotion time, not a vague notion of "passing something." And when the same model moves through dev, staging, and prod with different schemas, keep environment-specific config versioned separately from the weights, associate (model_version, environment, config_version) at promotion, and run that environment's schema-compatibility gate before every promotion.

Recap

• Registry = governance layer between training and production. Without it, every incident answer is archaeology through Slack and S3 timestamps.
• Holds three things: the artifact + full lineage (dataset hash, code commit, features, hyperparams), deployment history, lifecycle state (Experiment→Staging→Production→Archived).
• Weights alone are not the artifact: store the complete inference artifact = weights + fitted preprocessing pipeline + feature schema.
• Deploy scaler-normalized weights without the scaler → raw inputs treated as normalized → confident garbage, no error.
• Rollback is time-critical: promote previous version in one API call — but only if the artifact still exists.
• Never delete production artifacts — a year of storage costs less than one hour of a live incident.
• Not "just an S3 folder": approval trail + deployment lineage + instant rollback + a record of which model decided what, when.
• Experiment tracking (e.g. MLflow) logs hyperparameters + metrics + dataset version per run to a shared, sortable comparison UI — how two scientists pick a winner without a meeting.
• A deployment gate is a concrete automated check run at promotion time — schema compatibility, a metric threshold, or a required sign-off — not a vague notion of "passing something."
• Multi-environment deployment: keep environment-specific config versioned separately from weights, bind (model_version, environment, config_version) at promotion, and run a schema-compatibility gate before every promotion.

Check your understanding