Categorical Encoding

Convert categories into numbers without lying to your model about distance, order, or information from the target.

You are predicting customer churn, and one of your columns is `city` — with 5,000 different values. Models only eat numbers, so you have to turn those city names into numbers somehow. And *how* you do it turns out to matter a lot, because every method quietly makes a *claim* about the categories, and the wrong claim hurts you.

The obvious way, and why it strains at scale

The default is one-hot encoding: make one yes/no column per city (is it San Francisco? is it Austin? …). For a handful of categories this is perfect. But 5,000 cities means 5,000 near-empty columns, and with 100,000 rows many cities have only a few examples each — so their column is basically noise. The model can "memorise" that San Francisco churned a lot in your training window without learning anything that generalises.

Target encoding: one dense, informative column

A better move at high cardinality is target encoding: replace each city with the average churn rate *of that city*. San Francisco, with plenty of examples, gets a stable estimate; "Smalltown, OH" with three examples gets nudged toward the overall average so its tiny sample cannot run wild. Instead of 5,000 sparse columns you get *one* meaningful number per row — and for tree models this often beats one-hot outright.

But target encoding has a sharp, specific trap: leakage. If you compute a city's average churn using the whole dataset and then feed it back as a feature, each row's encoded value was partly computed from *its own label* — you have handed the model the answer. A city with a single row gets encoded with that row's exact outcome. The fix is to compute the averages *out-of-fold* — each row encoded using only *other* rows — which the `category_encoders` library does for you with one setting.

A quick decision guide

- Low cardinality (under ~15: payment method, device type): one-hot is fine — few columns, no leakage worry. - Medium (15–50): one-hot, unless there is a *real* order (education: high-school < college < grad), in which case ordinal encoding respects it. Do not impose an order where none exists — a linear model reads ordinal integers as literal numeric magnitude, so on an *unordered* feature like city, a category coded 49 is treated as having roughly 49x the weighted effect of a category coded 1, purely because of where it landed in an arbitrary label order, not because of any real signal. One-hot avoids this by giving every category its own independent coefficient. - High (50+: cities, merchants): target encoding (out-of-fold), or frequency encoding. - Very high, with a neural network (user IDs, product IDs): embeddings — the network learns a small dense vector per category, and categories that behave alike end up close together in that space.

One myth to retire: "LightGBM handles categoricals natively, so I can skip all this." Its built-in handling is fine for modest cardinality but degrades past a couple hundred values — for a 5,000-city column, explicit out-of-fold target encoding still wins, and it is one line of code.

Target encoding with smoothing: the actual formula

"Nudge rare categories toward the global average" has a precise form worth carrying. The smoothed encoding for a category is a *count-weighted blend* of its own mean and the global mean:

encoded = (n·category_mean + m·global_mean) / (n + m)

where n is how many rows that category has and m is a smoothing strength. When a category is common (n ≫ m) the encoding ≈ its own mean; when it's rare (n small) it's pulled toward the global mean, so "Smalltown with 3 rows" can't produce a wild estimate. Bigger m means more shrinkage. This smoothing is what makes target encoding safe on long-tailed categories.

The full CV-safe protocol

Getting target encoding right end-to-end has three stages, and mixing them up is the classic bug. For the training set, use out-of-fold encoding (each row encoded from the *other* folds) so no row sees its own label. For the test set (and inference), encode using statistics computed from the *entire* training set — the test rows never contribute to any mean. So: OOF within train, full-train stats applied to test. Fitting one encoder on all the data, or using OOF stats for the test set, both break it.

Frequency and count encoding

A cheaper cousin of target encoding: replace each category with how often it appears (its count or frequency). It's useful precisely when *popularity itself is predictive* — a rare merchant may be riskier than a common one — and it has a nice property target encoding lacks: it doesn't touch the label, so there's no leakage to guard against. It handles high cardinality in one dense column at almost no cost. It's weaker than target encoding when the outcome, not the frequency, is what matters — but it's a strong, safe default to try alongside.

Hashing: cheap and unbounded, but blind

The hashing trick maps categories into a fixed number of buckets via a hash function — so it handles *unbounded* vocabularies (new categories just hash into existing buckets) at fixed memory. The costs: collisions (different categories share a bucket, blurring them together — worse for rare categories that collide with common ones), total loss of interpretability (you can't read a bucket back to a category), and a real choice of bucket count (too few = heavy collisions, too many = sparse). Use it when the vocabulary is huge or streaming and you can tolerate some collision noise, not when you need to explain the model.

Embeddings: dimension, cold start, minimum frequency

Learned embeddings are powerful for neural nets but have their own knobs. Dimension: a common heuristic is min(50, cardinality^0.25 × 4) — bigger for higher-cardinality columns, but too large overfits. (The scaling constant varies by source; fastai's alternative rule of thumb is min(50, (cardinality+1)//2).) Minimum frequency: categories seen only a handful of times can't learn a good vector, so group rare ones into a shared "rare" token. Cold start: a brand-new category at inference has no trained embedding — you need a reserved "unknown" embedding to fall back on. And embeddings need enough data per category and some regularisation or they memorise. They're not free lunch; they're target encoding's expressive, data-hungry sibling.

Native categorical handling differs by library

"The tree library handles categoricals" is true but the *details* differ and matter. LightGBM has native categorical splits (good to a few hundred values). XGBoost added native categorical support more recently. CatBoost is the standout: it uses ordered target statistics — it draws a random permutation of the training rows, then encodes each row's category using only the target values of rows that come *before* it in that permutation, so a row's own label is structurally excluded from its own encoding (unlike plain target encoding, where every row's mean includes its own label unless you explicitly hold folds out) — which is why it often wins on categorical-heavy data with minimal preprocessing. So if your data is dominated by high-cardinality categoricals, CatBoost is worth trying specifically for this reason.

Rare categories and new categories in production

Two operational rules. Rare-category policy: fold categories below a minimum-count threshold into an explicit "Other" bucket rather than trusting three-row estimates. New-category monitoring: production will see categories that never appeared in training, and different encodings fail differently — one-hot has no column for an unseen city, so it silently encodes the row as all-zero; ordinal has no integer assigned to it at all; target encoding, hashing, and embeddings each need an explicit fallback (global mean for target encoding, a hash bucket, an "unknown" embedding). Define that fallback *and* monitor the rate of unseen categories — a rising unknown rate means your encoding is increasingly guessing, and it's an early signal that the category space has drifted and you should retrain.

Key points

• Use target encoding with out-of-fold isolation for any categorical feature with cardinality above 50 — it is the highest-signal encoding for tree models and takes one line of code with the category-encoders library. For the 5,000-city feature: target encoding produces a single dense column where each city's value reflects actual churn signal from training data. One-hot produces 5,000 sparse columns where most cities have fewer than 20 training examples — a regime that guarantees memorization rather than generalization.
• Trap: computing target encoding statistics before the train/test split. This leaks test-set label information into training features and is one of the most common sources of inflated offline metrics. The mechanism: mean churn rate per city is computed across the full dataset. Each row's city feature is now a function of that row's own label (plus its neighbors'). For cities with few rows, the encoded value is nearly the target itself. Fix: compute within folds using category-encoders' cross-val encoding or TargetEncoder with cv parameter.
• Diagnostic: if a target-encoded feature shows near-100% feature importance in a tree model, check for leakage — the encoding likely included the target row's own label in the mean. A legitimately useful encoding produces moderate, plausible importance. An encoding that accidentally includes row-level label information will dominate feature importance because it is effectively a noisy copy of the target. Check by comparing feature importance on train vs. validation — a leaking feature will show much higher importance on training data.
• Smooth target encoding by count, and know frequency/hashing/embedding trade-offs. Smoothed target encoding blends category mean with global mean by count: (n·cat_mean + m·global_mean)/(n+m), pulling rare categories toward the global average. The CV-safe protocol: out-of-fold encoding within train, full-train stats applied to test/inference. Frequency/count encoding is a leakage-free alternative when popularity is predictive. Hashing handles unbounded vocabularies but brings collisions and no interpretability. Embeddings need a chosen dimension, a minimum frequency (group rare into a shared token), and an "unknown" vector for cold start.
• Library categorical support differs, and production needs a rare/new-category policy. LightGBM and XGBoost have native categorical splits (good to a few hundred values); CatBoost's ordered target statistics structurally avoid the leakage plain target encoding suffers, so it often wins on categorical-heavy data with minimal preprocessing. Operationally: fold categories below a minimum count into "Other," define the unseen-category fallback (global mean / hash bucket / unknown embedding), and monitor the rate of unseen categories in production — a rising unknown rate means the category space is drifting and it's time to retrain.

Recap

• Every encoding asserts something about category structure — the wrong assertion is a false claim the model learns as true.
• High cardinality (>50): target encoding gives one dense signal-bearing column; one-hot on 5,000 cities is sparse and forces memorization.
• Target encoding leaks if computed before the split — the encoded value becomes a noisy copy of the row's own label. Compute out-of-fold.
• Leakage tell: a target-encoded feature at ~100% importance, much higher on train than validation.
• Smooth by count: `(n·cat_mean + m·global_mean)/(n+m)` pulls rare categories toward the global average.
• Menu: frequency/count (leakage-free when popularity predicts), hashing (unbounded vocab, collisions), embeddings (need dim + min-freq + unknown vector).
• Production needs a rare/new-category policy: fold below-min into Other, define the unseen fallback, monitor unseen rate — rising = drift = retrain.

Check your understanding