Foundational 35 min read data qualityprofilingmissing valuesoutliers

Data Quality Audit

Understand what makes data "dirty" and why auditing before modeling is non-negotiable.

A vendor hands you a shiny 2-million-row dataset with 47 columns. Before you write a single line of modeling code, you spend 20 minutes just *looking* at the data — a data quality audit — and here is what you find. The `income` column is stored as text, not numbers. `date_of_birth` has values from 1800 to 2300. `transaction_amount` is 12% missing, and the missing ones are suspiciously the *large* transactions. And 30,000 rows are exact duplicates. Twenty minutes of looking just saved you a week of chasing a broken model in production.

Here is why this step is non-negotiable, and it is subtler than "garbage in, garbage out." A powerful model does not choke on bad data — it *learns from it, confidently*. Feed a gradient-boosted tree rows where `age = -3` and it will happily decide that negative ages predict something. Your training metrics look fine. The model ships. It assigns high confidence to wrong answers and never once flags that anything is off. The real slogan is "garbage in, *confident* garbage out," and the bill arrives weeks later when the true labels do.

What to actually check

A good audit sweeps seven things:

- Schema — are the column types right, and are all the expected columns present? - Missing values — how much is null per column, and is the missingness random or *patterned*? - Duplicates — exact repeated rows, or near-duplicates on a key? - Distribution — outliers, strange skew, two humps where you expected one? - Target — is the label balanced, and are there noisy or impossible label/feature combos? - Leakage — does any feature secretly use information you would not have at prediction time? - Coverage — does this data actually resemble the population you will deploy against?

Outliers are not the same as impossible values

These two get lumped together, but they need different treatment. An outlier is a data point that is extreme but potentially *valid* — a $50,000 transaction is unusual, not wrong. An impossible value violates a hard constraint and is *always* an error — the `age = -3` row from a moment ago is not "an unusual customer," it is broken data, and it gets nullified before anything else runs. Confusing the two is dangerous in both directions: nullifying a legitimate extreme value throws away real signal, while treating a constraint violation as "just an outlier" lets broken data quietly train the model.

And "outlier" itself is not automatically noise to be removed. In fraud or anomaly detection, the fraud rows *are* the outliers — rare and extreme by definition, relative to the mass of normal transactions. A blanket "remove the outliers" step, applied without asking what the outliers *are* in this dataset, strips out most of the positive class before the model ever sees it, leaving a detector that has learned to predict "not fraud" almost every time.

The sneakiest failure: rows that silently vanish

One failure deserves special mention because it hides so well. Join two tables on customer ID, and if some IDs in one table have no match in the other, those rows simply *disappear* — no error, the pipeline reports success, and 120,000 training examples are gone. And they are almost never a random 120,000: they tend to be a specific group (older customers, one region), which is now missing entirely from training. Your model quietly learns nothing about them.

Make the checks automatic

The fix is to turn the whole checklist into *executable assertions* that run on every new batch of data — "column income must be a float between 0 and 10 million with under 5% nulls." Tools like Great Expectations do exactly this. When an assertion fails you get a loud error on day one instead of a mysterious performance drop six weeks later. And re-run these at *every* retrain, not just once: a column that was 2% null in January and 18% null in June is a data-collection problem, and you will never notice it without looking again. A pipeline running without crashing is not the same thing as the data being good.

Missingness has a taxonomy — and it changes what you can do

"How much is missing" is only half the question; "*why* is it missing" decides the fix. MCAR (missing completely at random): the missingness is unrelated to anything, so dropping or simple imputation is safe. MAR (missing at random): the missingness depends on *other observed* features (income missing more often for a certain age group) — you can impute using those features. MNAR (missing not at random): the missingness depends on the *missing value itself* (high earners refuse to state income) — and this is the dangerous one, because the missing values carry signal or bias that no ordinary imputation recovers. During the audit, don't just count nulls; look at *what else is true* about the missing rows.

Drift has metrics, not just eyeballs

"Does this batch resemble training?" can be measured. For continuous features, PSI (population stability index) is the industry-standard drift score (below 0.1 stable, above 0.2 significant), the KS test measures the largest gap between two distributions, and Wasserstein distance captures how far mass moved. For distributions generally, KL and Jensen-Shannon divergence quantify how different two are (JS is symmetric and bounded). For categorical features, compare category frequencies (chi-squared, or PSI on the buckets) and watch for brand-new categories. Wire these into the audit so drift is a *number that crosses a threshold*, not a vibe.

Audit the labels, not just the features

Dirty labels quietly cap model quality more than dirty features. Check for noisy labels (wrong ground truth), conflicting labels (identical rows labeled differently), annotation disagreement (measure inter-annotator agreement), delayed labels (the truth arrives weeks later, so recent rows are under-labeled), label leakage (a feature encodes the label), and label-definition drift (what counts as "fraud" changed between last year and this year). A feature audit that ignores label quality misses the ceiling on how good any model can get.

Train-serving skew and feature freshness

Some quality problems only exist *between* environments. Train-serving skew is when a feature is computed one way offline (SQL over full history) and another way online (a real-time approximation) — the model then scores on values it never trained on. Related is staleness: a feature that's fresh offline may be hours old at serving time. So the audit must track *timestamps*: the event time (when it happened), the ingestion time (when it landed), and the feature-computation time — and enforce a maximum feature age. A value that's correct but stale is still wrong at decision time.

Data contracts and a severity policy

Mature pipelines formalise this with data contracts: the producing team commits to a schema, types, value ranges, cardinality, null-rate ceilings, and a freshness SLA, with clear ownership so a breaking change upstream is caught at the boundary, not six weeks downstream. And each failed check needs a severity/action policy decided in advance: does this failure block training, quarantine the batch, fall back to a previous model, fire a warning only, or route to human review? "The check failed" is useless without "…and therefore we do X."

Audit sample coverage, not just column values

Finally, check *who* is in the data. A dataset can be clean on every column yet systematically under-represent a segment — a geography, a device type, an acquisition channel, a cohort, or the rare high-value population you most care about. Break the audit down by these dimensions and confirm each important segment has enough coverage, because a model trained on a skewed sample is confidently wrong exactly where the data was thin.

Key points

Run a data quality audit before any EDA or modeling — 30 minutes of auditing routinely saves days of debugging model failures caused by silent data issues. The 2-million-row vendor dataset example is representative: type mismatches, impossible values, structured missingness, and 30,000 duplicates all coexist quietly until you look for them explicitly. Powerful models do not flag dirty data — they learn from it confidently.
Trap: auditing only the training set. Data quality gates must run on every new incoming batch in production — upstream systems change schemas, inject nulls, and shift distributions without notice. A column with 2% nulls in January training data and 18% nulls in June production data is a structural change in data collection. The model trained on the low-null version will impute using training-fit parameters that no longer fit the incoming distribution, degrading silently on the most information-rich rows.
Diagnostic: build a schema snapshot of the training data — column types, value ranges, null rates, cardinality — and diff every new batch against it. A diff that exceeds threshold is a data quality alert, not a model problem. Great Expectations implements this as code: assertions that fail loudly when violated, run in CI/CD on every batch. The alternative is discovering the schema drift 6 weeks later as a "mysterious performance regression."
Diagnose missingness by mechanism and measure drift with real metrics. Don't just count nulls — determine MCAR (unrelated, safe to impute), MAR (depends on other observed features, impute using them), or MNAR (depends on the missing value itself, so ordinary imputation injects bias). Turn "does this batch look like training?" into numbers: PSI (>0.2 significant) and KS/Wasserstein for continuous features, KL/Jensen-Shannon for distributions, category-frequency comparison for categoricals — with new-category detection. And audit label quality (noisy, conflicting, delayed, leaked, definition-drifted labels), which caps model quality more than dirty features.
Guard the boundaries: skew, freshness, contracts, and a severity policy. Train-serving skew (feature computed differently offline vs online) and staleness are cross-environment bugs, so track event/ingestion/computation timestamps and enforce a maximum feature age. Formalise producer-consumer data contracts (schema, types, ranges, cardinality, null-rate ceiling, freshness SLA, ownership) so upstream breaks are caught at the boundary. Give every failed check a pre-decided action — block training, quarantine batch, fall back to previous model, warn, or human-review — and audit sample coverage by segment/geography/device/channel/cohort so no important population is silently thin.

Takeaway

Data quality failures surface as production incidents, not training errors — because models train confidently on garbage and only reveal the problem when ground-truth labels arrive weeks later.

Recap

Garbage in, *confident* garbage out: powerful models learn from bad data silently — the bill arrives weeks later with the true labels.
Audit before modeling: 20-min sweep of schema, missing, duplicates, distribution, target, leakage, coverage catches type mismatches, impossible values, 30k dupes.
Silent row loss: a join with unmatched IDs drops rows with no error — and the dropped group (a region, a cohort) is never random.
Make checks executable assertions that run on *every* batch (Great Expectations) — 2% nulls in Jan vs 18% in June is a collection bug you only catch by re-checking.
Missingness has a mechanism: MCAR safe to impute, MAR impute from observed features, MNAR carries bias no ordinary imputation recovers.
Drift is a number, not a vibe: PSI (>0.2 significant), KS/Wasserstein for continuous, KL/JS for distributions, category-frequency + new-category for categoricals.
Guard the boundaries: train-serving skew, feature staleness, data contracts, per-check severity policy, and per-segment coverage audits.

Check your understanding

Q1. A colleague argues that outlier rows should simply be removed before training to keep the model clean. What breaks if you follow this advice blindly on a fraud detection dataset?

A) The model trains 30% faster since fewer rows remain, but loses calibration on high-value transactions specifically, requiring a full threshold recalibration sweep before deployment and launch.
B) Fraud transactions are inherently outliers — rare and anomalous by definition. Removing outliers would strip out most of the positive class, leaving a model that almost never predicts fraud.
C) Removing outliers reduces variance but introduces bias toward the mean transaction profile, which specifically causes the model to underperform on weekend and holiday transaction patterns.
D) The model becomes overconfident on the majority class, but this is fully corrected by applying SMOTE with a 5:1 oversampling ratio right after the outlier removal step.

Q2. You join two tables on a customer ID and your training set shrinks from 500,000 to 380,000 rows without any error. What likely happened and why does it matter?

A) A deduplication step silently ran during the join on the customer_id key, removing duplicate rows. This is expected, and the remaining 380,000 rows are still a representative random sample.
B) A filter on the signup-date column excluded customers who joined before 2019 during the join, but since the remaining sample is still 380,000 rows, the model generalizes without issue.
C) The join used an implicit DISTINCT clause, collapsing multi-purchase customers to one row per customer ID, which slightly underrepresents heavy buyers but leaves the label distribution unchanged.
D) Referential integrity failure: 120,000 rows were lost because their customer IDs had no match in the second table — likely a specific region or cohort, so the model trains on a biased subset.

Q3. What is the difference between an outlier and an impossible value, and why should they be handled differently?

A) An outlier is extreme but potentially valid (a 50,000-dollar transaction is unusual). An impossible value violates a hard constraint (age = -3) and is always an error, nullified before anything else.
B) An outlier is any value above the 99th percentile; an impossible value is any value above the 99.9th percentile. Both should be clipped to the 99th percentile before training to stabilize the model.
C) An outlier is a data entry error; an impossible value is a measurement instrument failure. Outliers should be removed entirely; impossible values should be imputed with the domain-specific minimum valid value.
D) Both terms describe the same phenomenon — any value that falls outside two standard deviations from the column mean. The distinction is purely semantic and never changes how either should be handled.

Q4. You run a data profile on training set in January and model performs well. You retrain in June without re-profiling and performance drops. What data quality issue is most likely responsible?

A) The model's hyperparameters are no longer optimal because the June dataset grew by 40%, so the January grid search must be fully rerun before the model can be trusted again.
B) Random seed differences between the January and June training runs caused the optimizer to converge to a different local minimum, which alone accounts for the performance drop.
C) Distribution shift between January and June that a re-profile would catch — a new null rate, an upstream encoding change, or a real-world shift the pipeline can't see without looking again.
D) The validation split was proportionally smaller in June because the training set grew, so the June evaluation is noisier and meaningfully less representative of true production performance overall.

Q5. A column has 55% null values. A colleague says to impute with the median. What should you do before accepting that advice?

A) Run a Shapiro-Wilk test to confirm the non-null 45% of values are normally distributed; if they are, switch to mean imputation instead of median and proceed without further checks.
B) First determine the mechanism of missingness. If MNAR, median imputation fills in systematically wrong values for exactly the highest-risk cases — verify this before accepting median as the fix.
C) Check whether the column has more than 10 unique values; if so, KNN imputation with k=5 is always superior to median imputation, regardless of the missingness mechanism at play.
D) Immediately drop the column entirely — any feature carrying more than 30% nulls introduces more downstream bias than predictive value, and per best practice should never be imputed under any circumstance.

Q6. Two features are each 20% missing. In feature A, the missingness is unrelated to anything; in feature B, high earners systematically decline to report the value. How do you classify each, and why does it change your handling?

A) Both are MCAR because the missing rate happens to be identical at 20%, so mean or median imputation is equally safe and statistically unbiased for both features, regardless of which rows are missing or why.
B) Feature A is MCAR — its missingness is unrelated to any value, so simple imputation is safe. Feature B is MNAR: high earners are the ones missing, biasing any ordinary imputation on high-value cases.
C) Feature A is actually MAR and feature B is MCAR here, so both should simply be handled identically with a k=5 nearest-neighbor imputer trained on all the other observed columns in the dataset.
D) The MCAR/MAR/MNAR classification is irrelevant in this case — with 20% missing on both features, the safest move is to simply drop both of them regardless of the underlying missingness mechanism.

Q7. Your feature pipeline is clean on every column, but the model underperforms badly for one region and one device type. Aggregate metrics look fine. Which TWO of the following are true about what went wrong and how to catch it earlier?

A) You skipped hyperparameter tuning — regional and device-level underperformance is always a model-capacity problem, reliably fixed by training a larger model with more trees or layers.
B) You skipped a sample-coverage audit: a dataset can be valid on every column yet under-represent a segment (a region, device, or channel), so the model is confidently wrong exactly where the data was thin.
C) Aggregate metrics look fine because they average away the exact groups that are failing — evaluating performance broken down by region, device, and channel would have surfaced this gap before production.
D) Nothing was skipped here — per-segment underperformance on an otherwise-clean dataset is simply random noise from small sample sizes, and requires no further investigation or action.

Try it interactively

ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.

Open ML Systems Lab →