Missing Value Handling

The mechanism of missingness — MCAR, MAR, MNAR — determines the right treatment, not the null rate alone.

You are building a model to predict which hospital patients will be readmitted. One column — a creatinine lab result — is missing for 32% of patients. The natural instinct is to ask "how do I fill in these blanks?" But that is the wrong first question. The right one is: why are they missing? Because the reason completely changes what you should do.

So you ask the clinical team, and you learn there are actually *two* different reasons. Some patients had mild symptoms, so the doctor never bothered ordering the test — here the missingness depends on things you *can* see (their other symptoms). Other patients were too critically ill to draw blood — here the missingness depends on the very thing you *cannot* see (how bad their creatinine would have been). These two look identical in the data — both just show up as "null" — but they demand completely different treatment.

Statisticians name three cases. MCAR (missing completely at random): the blanks are pure chance, unrelated to anything — a lab machine randomly dropped some readings. MAR (missing at random): the blanks depend on things you *did* observe — mild patients skip the test. MNAR (missing not at random): the blanks depend on the missing value *itself* — the sickest patients, with the worst readings, are exactly the ones missing them.

Why the mechanism decides everything

For MCAR and MAR, you can fill the blanks intelligently. Since MAR missingness is explained by other columns, a model can *predict* the missing creatinine from a patient's age, comorbidities, and other labs. That is what MICE does: for each column with gaps, it trains a little regression on all the other columns and predicts what belongs in the blank, looping until the estimates settle.

For MNAR, you are in trouble, and no clever statistics can rescue you. If the worst readings are precisely the ones missing, then filling in the *average* hands the sickest patients a reassuringly normal number. The model learns that these patients look fine, predicts low risk for them, and systematically fails exactly the people who most need help. You cannot impute your way out of MNAR without explicitly modeling *why* the data is missing.

The alternative to imputing: dropping the rows. Before reaching for any fill-in method, some teams simply drop every row that has a null anywhere — this is complete-case analysis. It's only safe under MCAR: if the blanks are pure chance, the rows left behind are still a random sample, so nothing is biased. But under MAR or MNAR, dropping the rows with nulls also drops a *systematically different* slice of the population — for example, dropping every patient without a creatinine reading disproportionately removes the critically ill patients whose test was skipped for exactly that reason. A model trained on what remains never learns that population, so it can look strong on a validation split drawn from the same biased subset, then degrade once it meets, in production, the very patients complete-case analysis quietly deleted.

The imputation ladder, simplest to fanciest

Mean/median: replace blanks with the column's average. Fast, but it crushes the column's variance and erases its relationships with other features. KNN: find the most similar complete rows and borrow their values — respects local structure, but slow and needs scaled features. MICE: the regression-based looping method above — most accurate for MAR data, but you have to carry the fitted models around at serving time.

And one rule that holds no matter the mechanism: for any column more than about 5% missing, add a little yes/no "was this missing?" column next to it. The mere *fact* that a value was absent is often predictive (a skipped test says something), and a model given both the filled-in value and the was-missing flag can learn from the pattern of absence. Impute alone and you throw that signal away forever.

(And the usual leakage warning: fit your imputer on the *training* data only. Compute the fill-in value using the test set too, and you have leaked the future into the past.)

Single versus multiple imputation: honesty about uncertainty

Every method above is single imputation — it fills each blank with *one* number and then the model treats that guess as if it were a certainty. But it wasn't a certainty; it was an estimate with error, and pretending otherwise makes the model overconfident and understates the uncertainty in anything downstream. Multiple imputation fixes this: generate *several* completed datasets (each with slightly different plausible fills drawn from the imputation model's uncertainty), train/analyse on each, and pool the results. The spread across the versions honestly reflects how much the missingness actually costs you. MICE is naturally a multiple-imputation method (run it a few times with different seeds); use it when calibrated uncertainty matters, not just a point prediction.

Handling MNAR properly

MNAR can't be imputed away, but it can be *managed*. Sensitivity analysis: impute under several assumptions about how the missing values differ from the observed ones, and see whether your conclusions hold across them — if they do, you're robust; if they flip, you've found a real vulnerability. More formal tools model the missingness itself: pattern-mixture models fit different distributions for the missing-vs-observed groups, and selection models explicitly model the probability of being missing. And sometimes the right move is domain escalation — go back to the people who generated the data and ask why it's missing, because that answer often changes the whole approach. What you must *not* do is quietly median-fill and move on.

Categorical missingness is its own case

For categorical columns, you often don't need to "impute" at all — you can make missing an explicit "Unknown/Missing" category. This is clean and lets the model learn whatever the absence signals, without inventing a fake category value. Combine it with rare-category grouping (fold sparse categories, including Unknown, into an "Other" bucket if they're too thin to estimate) and, as with numeric columns, a missingness indicator where the fact of absence is predictive.

Imputation at serving time

Imputation isn't just a training-time step — it has to run *identically* in production, and that's where it breaks. You must store the fitted imputer (the training means, the MICE regression models, the KNN reference set) and apply the *same* parameters at serving, never re-fit on live data. You also have to handle unseen missingness patterns (a feature that was never missing in training suddenly arrives null) with a defined fallback, and monitor null-rate drift — a feature whose null rate jumps from 2% to 30% in production means the imputer is now guessing far more than it was validated to, and quality degrades silently.

Trees can sometimes skip imputation entirely

One model-family nuance worth knowing: several tree implementations (LightGBM, XGBoost, and histogram-based gradient boosting) handle missing values natively — they learn, at each split, which direction a missing value should go, so you can feed them NaNs directly and often *shouldn't* impute. Linear models, neural networks, and distance-based methods (k-NN, SVM) have no such mechanism and require explicit imputation (and usually scaling) first. So "how do I handle missing values" partly depends on which model you're feeding — trees may want the raw gaps, everything else needs them filled.

Key points

• Always add a binary indicator variable alongside imputation for any column with more than 5% missing — the fact that a value is missing is often more predictive than the imputed value itself. For the creatinine column: a model trained with only imputed values learns from the imputation. A model trained with imputed values plus a `creatinine_was_null` indicator can learn that absence itself is a clinical signal. Never throw away the missingness signal by imputing alone.
• Trap: fitting the imputer on the entire dataset before splitting. This leaks test-set statistics into training. Always fit imputers only on training data, then apply to validation and test. Fitting a mean imputer on train + test computes a mean influenced by test-set values. The training imputation now reflects the test distribution — a form of leakage that produces optimistic metrics which collapse in production. Use an sklearn Pipeline to enforce fit-on-train-only structurally, not through discipline.
• Diagnostic: compare model performance trained on imputed-only versus imputed plus indicator columns. If adding the indicator improves AUC, the missingness is *informative* — treat absence as a feature. This test costs one additional training run and shows whether the *fact* of missingness carries signal. Be careful with the conclusion, though: an informative indicator proves the missingness is predictive, but it does not by itself prove MNAR — MAR missingness (driven by other observed features) can also make the indicator useful. So use the indicator either way, but don't read "indicator helped" as a definitive MNAR diagnosis; confirm the mechanism with domain knowledge.
• Single imputation hides uncertainty; MNAR needs management, not a median fill. Single imputation fills one value and treats the guess as certain, making the model overconfident — multiple imputation generates several plausible completed datasets and pools them to reflect the real uncertainty (run MICE with different seeds). MNAR can't be imputed away but can be managed with sensitivity analysis (do conclusions hold under different assumptions?), pattern-mixture/selection models, or domain escalation. For categoricals, prefer an explicit "Unknown" category over inventing a value, plus rare-category grouping.
• Imputation must run identically at serving, and trees may not need it at all. Store the fitted imputer (training means, MICE models, KNN reference set) and apply the same parameters in production — never re-fit on live data — handle unseen missingness patterns with a defined fallback, and monitor null-rate drift (2% → 30% means the imputer is guessing far more than it was validated for). Model family matters: LightGBM/XGBoost/histogram gradient boosting handle NaNs natively (learning a default split direction), so you often shouldn't impute for them, while linear/neural/distance methods require explicit imputation and scaling.

Recap

• Ask *why* it's missing, not *how* to fill it — the mechanism decides the treatment.
• MCAR/MAR are imputable (MAR via a model like `MICE` predicting the gap from other columns); MNAR can't be imputed away — mean-fill hands the sickest patients a normal number.
• Imputation ladder: mean/median (fast, crushes variance) → KNN (local, slow) → MICE (regression loop, most accurate for MAR).
• Add a `was_missing` indicator for any column >5% missing — the fact of absence is often predictive.
• Fit imputers on train only — computing fills on test leaks the future into the past.
• Single imputation hides uncertainty; multiple imputation pools several plausible fills to restore honest error bars.
• Trees may want the raw gaps: LightGBM/XGBoost/hist-GBM learn a default split direction for NaNs; linear/NN/distance methods require explicit imputation + scaling.

Check your understanding