Data Preprocessing: Scaling, Categorical Encoding, and the MCAR/MAR/MNAR Taxonomy

Preprocessing is where most production ML bugs live. Using the wrong scaler can neuter regularisation. Using target encoding without proper cross-fitting leaks labels into features. Imputing MAR data as if it is MCAR produces biased estimates. This post covers every major preprocessing decision with the underlying reason — not just what to do, but why.

Data preprocessing is treated as boilerplate in most ML courses. It is not. The wrong preprocessing choices introduce subtle biases, destroy the meaning of regularisation, and silently degrade model quality. Each decision has a principled reason.

The MCAR / MAR / MNAR taxonomy for missing data

Missing data is not all the same. The type of missingness determines what you can do about it without introducing bias. Missing Completely At Random (MCAR): whether a value is missing is independent of the value itself and all other variables. Example: a lab technician drops a blood sample randomly. Any missing-data strategy is valid — even complete-case analysis (delete rows with missing values) gives unbiased estimates. Missing At Random (MAR): missingness depends on observed variables but not on the missing value itself, given those observed variables. Example: younger patients are less likely to have blood pressure recorded, but conditional on age, whether BP is recorded doesn't depend on BP value. Imputation using observed covariates (regression imputation, MICE) is valid. Complete-case analysis is biased. Missing Not At Random (MNAR): missingness depends on the unobserved value itself. Example: patients with very high blood pressure are less likely to have it recorded (they avoid the doctor). No standard imputation is unbiased without additional assumptions or external data. Must model the missingness mechanism explicitly. MNAR is the hardest case and often ignored in practice — leading to biased models.

Imputation strategies

Mean/median imputation: replace missing values with the column mean or median. Valid only under MCAR — creates artificial concentration at the mean, reduces variance, distorts correlations. Always add a binary indicator feature is_missing_j alongside the imputed value so the model can learn that missingness itself is informative. Regression imputation: predict the missing value from other features using a regression model. Valid under MAR. Preserves correlations between features. MICE (Multiple Imputation by Chained Equations): iteratively fit a regression model for each feature with missingness, using all other features as predictors. Repeat for many cycles until convergence. Produces multiple completed datasets; average predictions across them to get correct uncertainty estimates. Gold standard for MAR data. KNN imputation: fill missing values with the weighted average of the k nearest non-missing neighbours in feature space. Simple, often competitive with MICE for tabular data.

Feature scaling: when it matters and when it doesn't

StandardScaler: z = (x - μ) / σ. Each feature has zero mean and unit variance. Appropriate for: gradient descent-based models (neural nets, logistic regression, linear SVM) where features on different scales create elongated loss surfaces and slow convergence. L2/L1 regularisation — without scaling, the regularisation penalty is much larger for small-scale features than large-scale ones, biasing feature selection toward large-scale features. Distance-based models (KNN, kernel SVM, k-Means) where Euclidean distance is meaningless across different scales. MinMaxScaler: x_scaled = (x - x_min) / (x_max - x_min). Scales to [0,1]. Sensitive to outliers (one extreme value compresses all other values). Use when you need a bounded range (image pixel values, neural network output bounded to [0,1]) and outliers are not present. RobustScaler: x_scaled = (x - median) / IQR. Uses median and interquartile range instead of mean and std. Robust to outliers. Best when data has significant outliers that you cannot remove. When scaling doesn't matter: tree-based models (Random Forests, XGBoost, LightGBM) are invariant to monotone feature transformations — the split threshold adapts to the scale. Naive Bayes with Gaussian likelihoods — the likelihood computation normalises by the variance anyway. Rule: scale for gradient-based and distance-based models; skip for tree-based models.

Categorical encoding

One-hot encoding: create a binary indicator column for each category value. Correct for models that treat features as numeric (linear models, neural nets). Problem: high-cardinality features (e.g., ZIP code with 40,000 values) create 40,000 new columns — curse of dimensionality. Ordinal encoding: map categories to integers 0, 1, 2, .... Only valid when the categories have a true ordinal relationship (cold < warm < hot). Imposing fake ordinality on nominal categories misleads gradient-based models. Target encoding: replace each category with the mean of the target variable for that category. Computationally efficient, handles high cardinality. Critical risk: if computed on the same data used for training, this leaks the target into features — the model sees the answer before the question. Fix: use cross-fitting (compute target encoding from a held-out fold for each training sample). Always use sklearn's TargetEncoder with cv parameter. Frequency encoding: replace each category with its count or frequency in the training set. A simpler high-cardinality solution with no leakage risk. Weight of Evidence (WOE): used in credit risk — log(P(good|category) / P(bad|category)). Naturally handles binary targets and missing values; often the best encoding for logistic regression in financial applications.

Pipelines and the train/test split rule

The cardinal rule: all preprocessing statistics (mean, std, quantiles, target encoding statistics, imputation models) must be fit on the training set only and then applied to the test set. Fitting on the combined dataset leaks test information. In sklearn: always use Pipeline objects so that fit_transform is called only on training data and transform is called on test data. Cross-validation: sklearn's cross_validate with a Pipeline correctly re-fits the preprocessor on each fold's training data. Manual splits within a CV loop are error-prone.

Interview questions on this topic

"You have a feature 'city' with 5,000 unique values. What encoding strategies would you consider and what are the trade-offs?" — One-hot: 5,000 columns, sparse, correct but creates curse of dimensionality. Target encoding: single column, captures city-level signal, but requires careful cross-fitting to avoid leakage. Frequency encoding: no leakage risk, captures popularity signal but loses class-conditioned information. Embedding layer (if neural net): learns a dense representation end-to-end. Choice depends on model type and city cardinality relative to dataset size.

"What is the difference between MCAR and MAR? Does the distinction matter for imputation?" — MCAR: missingness is independent of everything. MAR: missingness depends on other observed variables but not the missing value itself. For MCAR, simple mean imputation is unbiased. For MAR, you must condition on the observed covariates to get unbiased estimates — regression imputation or MICE is required. For MNAR, no standard method is unbiased. Yes, the distinction matters enormously for inference validity.

"You fit a StandardScaler on your full dataset (train+test) and then do cross-validation. What went wrong?" — Test set statistics contaminate the scaler's mean and std estimates. The model implicitly has access to test set information during training. In practice this inflates cross-validation scores, sometimes significantly on small datasets. Always fit the scaler inside the cross-validation loop on the training fold only.

"Why doesn't tree-based models need feature scaling?" — Decision trees split features at threshold values. Whether a feature is in dollars or thousands of dollars doesn't change the optimal split threshold — the tree just picks a different number. The model is invariant to monotone transformations of features. Scaling would change the specific threshold chosen but not the information content of the split.

Try on Colab: take a dataset with mixed feature types (use the Titanic dataset). Introduce 20% missing values in the 'Age' column as MAR (probability of missingness depends on 'Pclass'). Compare four imputation strategies (mean, median, KNN, MICE) by measuring the bias in the estimated mean age per class. Show that mean imputation is biased for MAR missingness; MICE recovers the true conditional mean.