ML Pipeline Architecture
Batch vs streaming ingestion, orchestration, idempotency, pipeline failures
A team has a training script, a deployment script, and a cron job. They call it "the pipeline." Six months later nobody can say which data the live model was trained on. The training script hard-codes a local file path that broke when they moved to the cloud. The last model update took a week of manual fiddling to reproduce. That isn't a pipeline — it's a pile of scripts held together by memory that is quietly fading as people forget and move on.
What a real pipeline is: eight stages, end to end
*Ingestion* pulls the data, checks the schema, and writes a versioned snapshot. *Feature engineering* runs reproducible transforms into versioned tables. *Training* is pinned to a data version, hyperparameters, and a random seed, producing a versioned artifact. *Evaluation* scores it on a fixed holdout and compares to the current production model. *A deployment gate* auto-promotes if it clears the bar, or routes to a human if not. *Serving* looks up the model, computes online features, logs predictions. *Monitoring* watches drift and performance. *A retraining trigger* — on a schedule or an event — kicks the whole loop off again.
Three properties separate infrastructure from debt
*Idempotency:* re-running a step on the same input gives the identical output, no duplicates — done with fixed seeds, content-hashed data, and atomic overwrites. *Fast-fail on bad data:* check availability and schema *before* any compute runs — for example, if an outage cuts the upstream feed off mid-afternoon and training starts anyway on just the first 60% of that day's rows, the model learns a skewed slice missing an entire segment of the day's traffic (evenings, say) and deploys with full confidence in that skew; a model trained on a truncated, unrepresentative slice like that is worse than staying on the current model, which was validated on a complete day. *Complete lineage:* every artifact traces back to its exact data version, code commit, and parameters — without it, debugging is archaeology, and you cannot roll production back to the last model trained before a bug was introduced, because you do not know which models that bug touched.
The orchestrator you pick (Airflow, Prefect, Kubeflow, Metaflow) matters far less than whether it enforces those three.
The real test. Training and deployment are just two of the eight stages; without the other six you have technical debt dressed up as a system. So ask one question: can a new person reproduce the current production model from scratch, deterministically, in under two hours, without asking anyone? If yes, you have a pipeline. If no, you have scripts — and a week-long reconstruction waiting for you at the worst possible moment.
Key points
- Build the evaluation step before optimizing model quality — an automated gate that compares new models to the current production model prevents regressions and cuts manual review down to only the cases it cannot auto-resolve. This is the highest-leverage infrastructure investment. Without an evaluation gate, every model update requires human judgment under time pressure with incomplete information. With one, the decision was made in advance under no pressure: "a challenger must match or exceed the champion on these specific metrics before promotion." That is the decision-making context you want.
- Trap: non-idempotent pipelines make debugging impossible — if rerunning a step produces different outputs, you can never reproduce a historical result or isolate a regression. Enforce idempotency from the start: fix random seeds, version data snapshots by content hash, use UPSERT instead of INSERT, and overwrite partitions atomically. The test is simple: run the same step twice on the same input and assert identical output. If the assertion fails, you have a non-idempotent step that will silently produce different models on different runs.
- Diagnostic: ask "can a new team member reproduce the current production model from scratch in under 2 hours using only documented steps?" If the answer is no, the pipeline has critical gaps. The gaps are exactly where the next incident will live: undocumented data transformations, implicit file path assumptions, preprocessing parameters stored in someone's local environment, or model artifacts without provenance. Name the gaps before the incident names them for you.
A pipeline that reports success tells you nothing about whether it produced correct data — the gap between "ran without errors" and "produced correct outputs" is exactly where silent bugs live, and only data quality assertions on pipeline outputs close it.
Recap
- "Training script + deploy script + cron" is not a pipeline — it's a pile of scripts held together by fading memory.
- Eight stages: ingestion → feature engineering → training → evaluation → deployment gate → serving → monitoring → retraining trigger.
- Three properties separate infra from debt: idempotency, fast-fail on bad data, complete lineage.
- Idempotency: re-run same input → identical output. Fixed seeds, content-hashed data, atomic overwrites, UPSERT not INSERT.
- Fast-fail: check availability + schema *before* compute — e.g. an outage cuts the feed off mid-afternoon and training runs anyway on the first 60% of the day's rows: the model learns a slice missing an entire segment of traffic and deploys with full confidence in that skew, worse than staying on the current, fully-validated model.
- Orchestrator choice (Airflow/Prefect/Kubeflow) matters less than whether it enforces those three.
- The real test: can a new person reproduce prod from scratch, deterministically, in <2 hours, without asking anyone?
Check your understanding
Q1. Your daily retraining pipeline fails on day 3 because the upstream data source was unavailable. Select the two correct design choices for handling this gracefully.
- A) Use a data availability sensor at pipeline start that fails fast and alerts if data hasn't arrived within N hours, while continuing to serve the last good model
- B) Configure the pipeline to train on whatever partial data happens to be available and deploy that model — partial data beats no update at all
- C) Ensure idempotent backfill with upsert semantics once data recovers, and use automatic retry with backoff for transient failures
- D) Increase the pipeline's memory allocation to 128GB so it can cache the previous day's raw data as an automatic fallback source
Q2. You need to ensure that if a feature computation step fails and is rerun, it does not create duplicate records in your feature store. How do you design this?
- A) Add a deduplication step after every write using a SELECT DISTINCT query scheduled hourly via a cron job on the warehouse's replica cluster
- B) Use UPSERT keyed on (entity_id, feature_name, computation_date) rather than INSERT; overwrite Parquet partitions atomically; verify idempotency in CI by running the step twice
- C) Use a distributed lock via ZooKeeper with a 30-second lease to prevent concurrent runs — the duplicate problem only occurs when two pipeline runs overlap in time
- D) Idempotency is only needed for streaming pipelines running on Kafka consumer groups with at-least-once delivery; batch pipelines can safely use plain INSERT because scheduler retries are statistically rare events
Q3. A model is retrained daily. You discover that 4 days ago, a bug was introduced in the feature computation that corrupted 3 features. What is the remediation process?
- A) Retrain with the buggy features using the standard weekly schedule and deploy immediately — the model, given enough training epochs, will reliably learn to compensate for the corruption on its own
- B) Halt training; audit all models trained in the past 4 days via lineage; rollback production to the last pre-bug model; fix the bug, backfill the corrupted partitions, and retrain
- C) Delete all 4 days of corrupted training data entirely and retrain from scratch using only historical data older than 6 months, discarding recent labels
- D) Deploy a hotfix directly to the serving pipeline to correct the 3 corrupted features at inference time using a static lookup table; no retraining is needed at all
Q4. What is the difference between a pipeline failure and a pipeline bug, and why does this distinction matter for ML systems?
- A) There is no meaningful distinction between the two failure classes as defined in the SRE playbook — both result in a model that underperforms and should be handled with the identical rollback runbook
- B) A failure is detectable — the pipeline throws and alerts fire. A bug is silent — the pipeline succeeds on wrong data, and degradation surfaces weeks later; only output assertions catch bugs
- C) A pipeline failure, per the orchestrator's retry policy, affects only the current training run; a bug silently affects all future runs until someone notices and fixes it
- D) Failures are always caused by infrastructure issues like disk-full errors; bugs are always caused by bad upstream data — both require the identical response of rolling back to the last good model
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →