Data Versioning and Pipelines

Models are only reproducible if both code and data are versioned — and production ML breaks when training and serving compute features differently.

A model you shipped six months ago starts misbehaving: performance on iOS users has fallen off a cliff since October. You go to investigate — and hit a wall. You have *no record* of what data that model was trained on, what the pipeline looked like before a September change, or which features that change touched. So begins two weeks of archaeology. Now imagine the alternative: you `git log` the pipeline, run one command to check out the *exact* dataset the model trained on, re-run training, and find the bug in two hours. That is what data versioning buys you.

The core idea is simple: a model is a function of three things — its training data, its code, and its hyperparameters — and to reproduce a model you must be able to recover all three. Code versioning (Git) is second nature. The piece teams forget is the *data*. Tools like DVC fix this by storing a tiny pointer file — essentially a fingerprint of the dataset — right next to your code in Git, so checking out any past commit gives you back the code and the data's fingerprint from that point; a separate `dvc checkout` (or `dvc pull`) then materializes the actual data to match it. A tool like MLflow captures the third leg, logging each run's code version, data fingerprint, hyperparameters, and results, so any past experiment can be rebuilt from its run ID instead of from memory.

The other silent killer: training-serving skew

Here is a failure that bites far more teams than expect it. The logic that computes your features usually gets written *twice* — once in Python for training, once in SQL or Java for the live serving system. Over time the two quietly drift apart: a timezone handled differently, a null treated differently, a rounding difference in an aggregate. Now the model is fed inputs at serving time that are subtly different from anything it trained on. Offline it scores 91%; in production it scores 77% — and *nothing errors*.

There is only one real fix, and it is structural: compute each feature in *one* canonical place that both training and serving use. This is exactly what a feature store (Feast, Tecton, Hopsworks) does — it keeps a single definition of each feature and serves it to both the training pipeline and the live system, so the two can never drift apart.

Is this overkill for a small team?

It is tempting to skip all this as heavyweight process. But the maths is stark. Adding DVC to a repo costs a couple of hours, once. The *first* time a silent data bug causes a production incident without it, you lose days digging — and you may not be able to confidently roll back at all, because you cannot reproduce the good state. A good test of whether you are actually versioned: *could a teammate who was never on the project reproduce this exact model from scratch in half an hour, given only the commit hash?* If not, you do not really have a versioned pipeline yet.

The fourth leg: environment and dependencies

Code + data + hyperparameters isn't quite the whole story — the *environment* is a fourth leg of reproducibility. A different scikit-learn version can change a default and shift results; a different CUDA/cuDNN or GPU can change floating-point outputs; an un-pinned dependency can silently upgrade under you. So version the environment too: lockfiles (Conda, Poetry, pip freeze), a Docker image (or documented base image + CUDA/library versions), and a note of the hardware/runtime. "It reproduced on my machine" isn't reproducibility until the machine itself is pinned.

The model registry and its lifecycle

Trained artifacts need governance, which is what a model registry (MLflow Registry, SageMaker, Vertex) provides. It tracks each model version through lifecycle stages — MLflow's actual stage names are None → Staging → Production → Archived — with approval gates between them, one-command rollback to a prior version, lineage back to the training run, and a model card documenting intended use, metrics, and limitations. This is what turns "which model is live and how do I revert it?" from an incident into an API call.

Pipeline orchestration

The steps (ingest → features → train → evaluate → deploy) run as an orchestrated DAG in tools like Airflow, Prefect, Dagster, or Kubeflow. What the orchestrator buys you: retries and failure alerts, scheduling and backfills (re-run a date range after a fix), and — critically — idempotency (re-running a step on the same input produces the same output with no duplicates). Without idempotency you can't safely retry, and without alerts a silent stage failure becomes next month's mystery.

Feature store: two stores, freshness, materialization

The feature store that cures training-serving skew has real internal structure worth knowing. It has an offline store (historical values, point-in-time-correct, for building training sets) and an online store (the latest value per entity, low-latency, for serving). Materialization is the job that computes features and writes them to both; each feature has a freshness SLA and often a TTL. The hard parts are keeping online latency low and getting backfill correctness right — recomputing historical features with point-in-time correctness so training and serving see the same values.

Point-in-time correctness, spelled out: it means a feature's value for an example labeled at time T must reflect what was true *as of T* — not a value recomputed later from data that didn't exist yet. Skip this and you get temporal leakage: label a purchase example from 6 months ago, but compute its '30-day purchase count' feature from *today's* data, and that count includes purchases the customer made *after* the label date — information the model could never have had at prediction time. The model trains on leaked future signal, looks great offline, then fails once that future data isn't available at serving time.

Data contracts and experiment tracking, spelled out

A data contract is the enforceable interface a producing team commits to: schema, types, null-rate ceiling, value ranges, cardinality, uniqueness, volume, freshness SLA, and clear ownership — so an upstream breaking change is caught at the boundary. Experiment tracking should capture everything needed to rebuild and compare a run: code commit, dataset hash, feature-set version, config file, random seeds, split IDs, hyperparameters, metrics, and per-slice evaluation reports. If a run can't be rebuilt from its logged record alone, the tracking is incomplete.

CI/CD for ML, and end-to-end lineage

ML needs its own CI/CD: unit tests for feature logic, data-validation tests on incoming batches, training smoke tests (does a tiny run complete?), model-performance gates (block deploy if a metric regresses), and canary rollouts with automated rollback. Tying it all together is lineage: for any production prediction you should be able to trace *backward* — prediction → model version → the exact feature values used → the feature-computation code → the raw data snapshot → the training run. That backward trace is what makes an incident debuggable in hours instead of weeks, and it's the ultimate payoff of versioning code, data, environment, and pipeline together.

Key points

• Add DVC to any project the moment you have a second training run — tracking data versions retroactively is harder than starting upfront. DVC setup is a `dvc init` plus a `dvc add`/`dvc.yaml` entry per dataset — not a Makefile — and it adds no overhead to training itself. For the iOS debugging scenario: `dvc checkout` restores the exact dataset used six months ago. Without DVC, the training table has been updated, overwritten, or partitioned differently since then. Reproducing the model state is impossible, not just hard.
• Trap: versioning model artifacts but not data. If you can reproduce the model checkpoint but not the training data, you cannot audit why the model behaves the way it does. Data versioning is more important than model versioning. MLflow saves the trained model weights. DVC saves the dataset hash. You need both. Model weights tell you what the model learned; the dataset hash tells you what it learned from. Without the dataset, you cannot audit for label errors, investigate training distribution, or reproduce a retraining run.
• Diagnostic: ask yourself "can I reproduce this model from scratch in under 30 minutes?" If the answer is no, you do not have a versioned pipeline. The test is concrete: given only the Git commit hash for a past training run, a colleague who was not on the project should be able to reproduce the model checkpoint within 30 minutes. If this is not possible — because data is untracked, pipeline stages are undocumented, or hyperparameters were set interactively — the pipeline is not versioned in any meaningful sense.
• Reproducibility has four legs, and production ML needs a registry, orchestration, and a feature store. Code + data + hyperparameters + environment — pin dependencies with lockfiles and a Docker image (CUDA/library versions included), since a package bump can silently change results. A model registry manages lifecycle (MLflow's stage names: None → Staging → Production → Archived) with approval gates, one-command rollback, lineage, and model cards. Orchestrators (Airflow/Prefect/Dagster/Kubeflow) give retries, alerts, backfills, and idempotency. The feature store has an offline store (point-in-time history for training) and online store (latest value, low latency for serving), joined by materialization with a freshness SLA.
• Formalise data contracts and experiment tracking, and wire ML CI/CD with full lineage. A data contract commits a producer to schema, types, null-rate, ranges, cardinality, uniqueness, volume, freshness SLA, and ownership. Experiment tracking must log code commit, dataset hash, feature-set version, config, seeds, split IDs, hyperparameters, metrics, and per-slice reports — enough to rebuild the run. ML CI/CD adds unit tests for feature logic, data-validation tests, training smoke tests, performance gates, and canary rollouts. The payoff is end-to-end lineage: trace any production prediction back through model version → feature values → feature code → raw data snapshot → training run.

Recap

• A model is a function of code, data, and hyperparameters together — versioning only code leaves the half that caused the incident missing.
• Version data, not just model artifacts: weights say *what* it learned; the dataset hash says what it learned *from*. You need both (DVC + MLflow).
• Add DVC at the second training run — `dvc checkout` restores the exact past dataset; retroactive versioning is near-impossible.
• The 30-minute test: given only a Git commit, a colleague reproduces the checkpoint in <30 min — else the pipeline isn't versioned.
• Reproducibility has four legs: code + data + hyperparameters + environment (pin lockfiles + Docker; a package bump silently changes results).
• Production ML needs a registry (lifecycle, rollback, model cards), orchestration (retries, backfills, idempotency), and a feature store (offline point-in-time + online low-latency).
• End-to-end lineage is the payoff: trace prediction → model version → feature values → feature code → raw snapshot → training run — hours, not weeks.

Check your understanding