Data Quality for ML
Schema drift, distribution shift in features, null rates, automated validation
It is 3am. An alert fires: the fraud model has stopped returning high-confidence positives. Nothing was deployed in the last two days. The model? Running fine. The feature pipeline? Reported success. The upstream transaction table? Zero rows for the last two hours — an ingestion outage upstream. No error was ever thrown. The pipeline happily ran on empty windows, produced all-zero feature vectors, and the model scored every transaction as low-risk. Thousands of fraudulent orders sailed through unflagged.
The one asymmetry that makes data quality hard: bad data doesn't raise exceptions
An empty table returns in 5 milliseconds. A feature that nulls out because a join failed hands you a number, not an error. A model fed all zeros returns a confident score. At every layer the system reports *success* while quietly producing garbage. That is why "did the pipeline run?" tells you almost nothing — you have to actively *check the data itself.*
So you check it at every stage
At *ingestion:* does the data even exist, in the expected volume, with the expected schema? At *feature computation:* are null rates in bounds, are distributions close to training? At *training:* does the label rate match history, has any feature's mean drifted more than 2σ? At *serving:* does the live feature schema still match training, are values in range?
The five checks that catch the most
*Freshness* — is data arriving on time (alert when nothing new for longer than expected). *Completeness* — null rates within bounds, per feature. *Validity* — values in range, categories from the known set. *Volume* — row count within ±30% of the rolling weekly average. *Schema consistency* — no surprise column renames or type changes from upstream. Tools like Great Expectations and TensorFlow Data Validation turn these into assertions that *fail the pipeline loudly* — not dashboards someone has to remember to open.
And the mindset that ties it together: data quality is never "done at launch." Upstream teams change schemas, inject nulls, and shift distributions all the time, and they will not tell you. The real question isn't "is our data clean today?" — it's "will our monitoring catch the problem before the model does?" A pipeline that screams on day one of a schema change beats one that silently retrains on corrupted features for six weeks.
Key points
- Add freshness and volume checks first — an empty or truncated upstream table is the most catastrophic failure mode, and it is the cheapest to catch: one row-count check and one recency check per table. A table that should have 10,000 rows but has 0 is an emergency. Catching it before the feature pipeline runs is the difference between a 30-minute incident and a 6-hour one. Alert when no new rows have arrived for more than the expected latency, and alert when record count falls below 70% of the rolling 7-day average. These two checks are the floor, not the ceiling.
- Trap: monitoring aggregate statistics but not per-slice quality hides the failures that matter most. A table with 10,000 rows and 0.1% overall null rate can have 100% null rate for iOS users — exactly the segment most affected by a specific data pipeline bug. Always monitor data quality stratified by the key business dimensions your model cares about: platform, geography, user segment, device type. Aggregate metrics pass; per-slice checks catch the real failures.
- Diagnostic: set up a daily data quality report that diffs current column statistics against a snapshot from training time. Any column whose mean shifts more than 2σ from the training distribution is a drift candidate requiring investigation. This check costs one SQL query per feature and catches the silent degradation pattern — upstream changes the data, the pipeline reports success, the model retrains on shifted features, and no one notices until a business metric moves six weeks later.
Bad data throws no exceptions — the only thing that distinguishes "pipeline ran" from "pipeline ran on data the model was trained to handle" is data quality assertions you wrote before the incident happened.
Recap
- The asymmetry: bad data raises no exceptions. Empty table returns in 5ms, failed join returns a number, all-zeros model returns a confident score.
- "Did the pipeline run?" tells you almost nothing — you must check the data itself.
- Check at every stage: ingestion (exists? schema?), features (nulls, distribution), training (label rate, 2σ drift), serving (schema, range).
- Five checks: freshness, completeness (null rate), validity (range/categories), volume (±30% of weekly avg), schema consistency.
- Tools (Great Expectations, TFDV) fail the pipeline loudly — assertions, not dashboards someone must remember to open.
- Watch per-slice, not just aggregate: 0.1% overall null can hide 100% null for iOS users — the exact segment a bug hit.
- The real question isn't "is data clean today?" but "will monitoring catch it before the model does?"
Check your understanding
Q1. Your training pipeline runs successfully every day, but model performance has been slowly degrading over 3 weeks with no code changes. Select the two correct diagnostic steps.
- A) Check per-feature mean-shift (>2σ from the training distribution) over time and null-rate trends across the 3-week window for a slow, spike-free drift pattern
- B) Redeploy the exact same model artifact to reset an internal staleness counter that Kubernetes maintains for long-running pods
- C) Check label distribution shifts, data volume changes, and upstream schema changes over the same period
- D) Gradual degradation without any spikes is always caused by concept drift specifically; retrain immediately on only the most recent 7 days
Q2. You add a new upstream data source to your feature pipeline. How do you validate data quality before using it in training?
- A) If the pipeline runs without exceptions using the default Airflow retry policy of 3 attempts, the data source is valid — schema errors would always fail the job at the Avro deserialization step
- B) Validate schema and non-null constraints, verify row counts by date for continuity, inspect distributions against domain expectations, and measure join quality with the main entity table
- C) Run a sample of 100 rows through the feature pipeline with a fixed random seed of 42; if the output looks reasonable on manual inspection, approve the data source for production
- D) Data quality validation is only needed for existing data sources that have a documented SLA breach history; new sources are assumed clean until proven otherwise by a downstream metric regression
Q3. A feature pipeline runs successfully but produces the wrong values — all "user_account_age" values are approximately 365 days regardless of actual account age. How does data validation catch this?
- A) Range validation would catch this because 365 falls exactly on the upper boundary of a [0, 365] range check that was actually configured for an unrelated tenure feature, not account age
- B) Schema validation catches this because the data type silently flips from int32 to float64 whenever the upstream Airflow DAG computes values incorrectly
- C) Variance/standard-deviation expectations catch this — a constant feature produces stdev near zero, failing an expect_column_stdev_to_be_between assertion
- D) This bug cannot be caught by any automated validation currently in place; it requires a manual weekly spot-check of 500 sampled records by an engineer
Q4. How do you implement data quality checks that catch issues before they affect model training, without slowing down the pipeline significantly?
- A) Run all data quality checks after model training completes, via a nightly Airflow DAG with a 6-hour SLA, so they never block the pipeline's critical path
- B) Use a stratified strategy: real-time schema validation at ingestion, fast statistical checks after each batch, and full distribution-drift (2σ mean-shift) checks daily; gate training on all checks passing
- C) Sample exactly 1% of records using reservoir sampling for quality checks and extrapolate the results, since full dataset validation is too slow for any daily pipeline
- D) Data quality checks should run in a fully separate pipeline on a different Kubernetes namespace that never blocks model training at all — alerts get addressed only after deployment
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →