Feature Engineering
Transform raw columns into representations that expose the signal a model can actually learn from.
You want to predict credit default, and you have three raw columns: income in dollars, account age in days, and the date of the last transaction. You feed them straight into a model and it does poorly. The tempting move is to grab more data or a fancier model. Both are wrong. The problem is that the raw columns do not present the information in a shape the model can use — and reshaping them is called feature engineering.
The key thing to internalise: a model sees *nothing but numbers*. It has no idea that income is measured in dollars, that an account opened last week is a different animal from a decade-old one, or that income only means something *relative* to debt. Your job is to bake that knowledge into the numbers themselves. Let us do it column by column.
Income: fix the scale
To a raw model, the jump from 50,000 to 51,000 and the jump from 50,000 to 100,000 are just "1,000" and "50,000" apart — so it treats the second as fifty times more important. But for credit risk, doubling someone's income matters enormously while a 1,000 bump is noise. A log transform fixes this: on a log scale, 50K → 100K is a big step while 900K → 950K is almost nothing. It stretches out the low range where the real variation lives and squashes the giant tail. The model is not smarter afterward — the *shape* of the feature is finally right.
Account age: hand it the domain knowledge
The raw numbers 7 days versus 3,650 days do not capture that a brand-new account behaves nothing like a ten-year one. Bucketing into "new / established / long-term," or building a ratio like "transactions per account-year," gives the model the thing an expert already knows: the first year is a different world from year ten.
Last transaction date: turn a calendar into a signal
A raw date means nothing on its own. But "days since last transaction" is a direct *recency* signal, and "transactions in the last 30 days" is a *velocity* signal. Neither exists in the original data — you compute them against a reference time. These are temporal features, earned by realising the model wants a time gap, not a calendar entry. And when a time field is *cyclical* — hour of day, day of week — encoding it as a plain integer is a trap: hour 23 and hour 0 are one hour apart in reality but 23 apart as integers, so a sin/cos pair placing each hour on a circle is what keeps midnight next to 11pm.
Income and account age together: the interaction
Income alone does not tell you whether it is typical for how long someone has held the account, and account age alone does not either — but income relative to account age does: a $200,000 income on an account opened last month reads very differently from the same income on a decade-old account. This is an interaction feature: a joint signal that neither parent carries by itself. A tree model can sometimes discover it on its own; a linear model never will unless you hand it the ratio explicitly.
Doesn't deep learning make this obsolete?
For images and text, largely yes — raw pixels and words already carry rich structure a network can exploit. But for *tabular* data like this, no. The model just sees bare numbers with no idea what they mean, so even gradient boosting — which handles non-linearities well — routinely gains 5–20% from good ratios and time lags — though not from log transforms specifically, since trees split on thresholds and a monotonic transform never changes which threshold is optimal. Feature engineering is not busywork; it is the craft of encoding what you know into the geometry of the input, so that a solvable problem actually becomes solvable.
Categorical columns need encoding too
The columns above were numeric; categorical columns ("country," "device," "merchant") have to be turned into numbers, and the choice matters. Quick map: one-hot for low-cardinality categories (a column per value), ordinal when the categories have a real order, target encoding (replace a category with the average outcome for it) for high-cardinality columns, frequency/count encoding when popularity itself is predictive, the hashing trick for huge or unbounded vocabularies, and learned embeddings for neural nets. Each has a different leakage and cardinality profile — the dedicated encoding lesson goes deep; the point here is that "engineer the features" includes choosing a categorical encoding, not just transforming numbers.
Aggregation and window features: where the real signal often lives
For behavioural data, the strongest features are usually *aggregates over time*, not raw columns. The classic frame is RFM — Recency (days since last transaction), Frequency (how many in the last N days), Monetary (total/average spend) — and it generalises: rolling statistics (mean/std/count over a trailing window), expanding windows (cumulative to date), and lag features (the value k steps ago). These compress a user's history into a few predictive numbers. The non-negotiable rule attached to all of them is the point-in-time join: every aggregate must be computed using only data available *strictly before* the label timestamp, or you leak the future — the single most common way aggregate features go wrong.
High-cardinality columns need special care
A "merchant_id" or "zip_code" column with 50,000 values can't be one-hot encoded sanely (50,000 columns, most nearly always zero). Options: group rare categories into "Other" below a count threshold, the hashing trick (map categories into a fixed number of buckets, accepting some collisions), and smoothed target encoding — replace each category with a blend of its own outcome average and the global average, shrinking rare categories toward the global mean so a category seen twice doesn't get a wild estimate. And target encoding is a leakage magnet, so it must be done out-of-fold inside cross-validation (each row encoded using only the *other* folds), never on the full data.
Text, images, embeddings: where engineering becomes representation learning
For unstructured inputs the "features" are learned, not hand-built. Text becomes TF-IDF vectors (classic) or embeddings from a pretrained model (modern); images become activations from a pretrained CNN; and you often reduce dimension afterward. The line to notice: at this point feature engineering has become representation learning — the network *learns* the features instead of you crafting them, which is exactly why deep learning displaced hand-engineering for images and text but not for tabular data.
A feature has to exist at prediction time
An engineered feature is worthless if you can't compute it live with the same values. Serving parity is the discipline: the offline (training) computation and the online (serving) computation must produce identical results, the feature must be fresh enough at request time, and backfilling it for historical training rows must respect point-in-time correctness. This is what feature stores exist to enforce. A feature that's brilliant offline but arrives stale, or is computed differently online, produces train-serving skew — the model scores on values it never learned from.
Validate new features by ablation
Don't trust "accuracy went up after I added 40 features." Validate the way you'd validate a model change: start from a baseline with raw features, add one feature family at a time, and measure the lift on an untouched validation/test set — features selected or tuned against the same set you report on will look better than they are. Pair this with redundancy checks (drop features correlated with existing ones), permutation importance, and a look at whether the picks are stable across folds. A feature family that doesn't move held-out performance is overfitting risk, not signal.
Key points
- Use log and sqrt transforms when a continuous feature is right-skewed and your model is not a tree. Income, transaction counts, prices, and time durations almost always need this. The rule of thumb: if the 95th percentile is more than 10× the median, the raw scale is hurting you. Apply log(x + 1) to handle zeros. For tree-based models, skip it — the relative ordering is all that matters and log transforms change nothing about optimal split thresholds.
- The most common production trap: computing temporal features without a strict temporal join. A "7-day rolling transaction count" sounds clean until you realize it was computed using the label day itself. The feature includes the day you are trying to predict. In training this inflates performance; in production the feature is computed before the outcome is known. Every lag feature, rolling mean, or "days since" feature must be computed using only data available strictly before the label timestamp. Validate this by running your feature pipeline on a single row and verifying the computation cutoff date.
- Diagnose which features are earning their place with permutation importance, not training loss. Shuffle a feature's values across the validation set and measure the drop in performance. A genuinely useful feature causes a large drop when shuffled. A spurious feature or a duplicate of another feature causes no drop. Features that do not move permutation importance are adding noise and overfitting risk — remove them. Run this check after any batch of new features before shipping to production.
- Aggregate over time with point-in-time joins, and handle high-cardinality categoricals carefully. The strongest behavioural features are RFM-style aggregates — recency, frequency, monetary, plus rolling/expanding/lag windows — but every one must be computed strictly before the label timestamp (point-in-time join) or it leaks the future. For high-cardinality columns (merchant_id, zip), group rare categories into "Other," use the hashing trick, or smoothed target encoding done out-of-fold in CV (never on the full data, or you leak the label). Encoding choice is part of feature engineering: one-hot for low cardinality, target/frequency/hashing for high, embeddings for neural nets.
- A feature must survive to serving, and new features earn their place by ablation. Serving parity is non-negotiable: the offline and online computations must match, the value must be fresh at request time, and backfills must be point-in-time correct — a feature that's great offline but stale or differently-computed online creates train-serving skew. For unstructured data (text/images), features become learned representations (TF-IDF/embeddings), which is where engineering shades into representation learning. Validate additions by ablation — baseline, add one feature family at a time, measure lift on an untouched test set, check stability across folds — rather than trusting a single accuracy bump.
Raw features encode what was recorded; engineered features encode what the model needs to find the pattern. The right representation can replace millions of additional training rows — the wrong one makes the signal invisible regardless of model complexity.
Recap
- Raw features encode what was recorded; engineered features encode what the model needs — the right representation can replace millions of rows.
- Log/sqrt right-skewed inputs for non-tree models (95th pctile >10× median); use `log(x+1)` for zeros. Trees don't care — ordering is all that matters.
- Temporal features must use a strict point-in-time join: a 7-day rolling count that includes the label day leaks the future.
- Permutation importance, not training loss, decides which features earn their place — shuffle it; no drop means no signal.
- High-cardinality categoricals: group rare into Other, hashing trick, or out-of-fold smoothed target encoding (never on full data).
- Serving parity is non-negotiable: offline and online computations must match, values must be fresh, backfills point-in-time correct — else train-serving skew.
- Validate additions by ablation: baseline, add one feature family at a time, measure lift on an untouched test set, check stability across folds.
Check your understanding
Q1. You have a 'time_of_day' feature encoded as integer 0-23. Your model performs poorly on predictions for late-night events. What is the encoding problem and fix?
- A) The integer encoding creates a class imbalance between daytime and nighttime hours; the fix is to oversample late-night training examples using SMOTE with a 3:1 ratio before training begins.
- B) Integer encoding treats midnight as the arbitrary midpoint of the day rather than a true boundary; the fix is to shift all 24 values by 12 so that noon instead maps cleanly to 0.
- C) Raw integers place hour 23 and hour 0 at maximum distance, when they're really 1 hour apart. Fix: sin/cos encoding puts each hour on a circle, so 23 and 0 sit close together.
- D) Integer encoding assigns disproportionate weight to the hour feature relative to every other feature in the model; the fix is to standardize the hour column with a StandardScaler transform.
Q2. Why does log-transforming an income feature help a linear regression model but not a random forest?
- A) Linear regression assumes linearity, so skewed income makes it overweight extremes. Random forest splits on thresholds — log doesn't change which threshold is optimal, so it barely helps trees.
- B) Log transformation improves both model types equally in practice; the only difference is that random forests already regularize through bagging, which fully masks the benefit from view.
- C) Log transformation helps linear regression because it mathematically removes outliers entirely; random forests are unaffected since they naturally ignore outliers through majority-vote ensembling.
- D) Log transformation converts multiplicative relationships into additive ones, which matters for linear regression only when the true underlying relationship is multiplicative rather than additive.
Q3. You are building a fraud detection model and add interaction term: transaction_amount × is_international. What does this feature capture?
- A) It captures the total transaction volume for international merchants specifically, which turns out to be the exact same signal as simply summing all international amounts over a rolling window.
- B) It captures geographic risk entirely independent of amount — flagging every international transaction regardless of its size, which is mathematically equivalent to using is_international alone.
- C) It captures a combined effect: a high amount is suspicious specifically when international, not domestic. The product is large only when both hold — a linear model can't discover this on its own.
- D) It captures the variance in transaction amounts across international versus domestic transactions, which is actually better estimated by computing the ratio of international mean to domestic mean.
Q4. A data scientist creates 200 interaction features from 20-feature dataset and reports improved validation accuracy. What risk does this improvement mask?
- A) The risk is multicollinearity — 200 interaction features will be highly correlated with their 20 parent features, making every coefficient estimate unstable and essentially impossible to interpret.
- B) With 200 mostly-noise features added, the model gains far more capacity to overfit — if validation tuned or selected features, the reported gain is optimistic. Verify with an untouched test set.
- C) The risk is that the 200 interaction features may simply not be available at serving time, since the underlying raw features are computed in separate pipelines with different latency requirements.
- D) The validation accuracy improvement is real but temporary — the model will degrade within a few retraining cycles as the interaction features cause growing numerical instability in gradient descent.
Q5. You engineer a "transactions in the last 7 days" feature for a fraud model and validation AUC jumps to 0.97, but production performance is far worse. The feature itself is predictive. What's the likely bug?
- A) The feature is simply too predictive on its own, so the model overfits heavily to just this single signal; the correct fix is to remove the feature entirely before retraining.
- B) The rolling aggregate almost certainly skipped a point-in-time join — it included label-day transactions, leaking future data. Recompute using data strictly before the label timestamp.
- C) The 7-day window is simply too short for this fraud pattern; extending the rolling window to a full 30 days will make the training and production computations match exactly.
- D) Production simply sees fewer transactions per account than training did, so the feature is naturally noisier there — there is nothing to actually fix in the pipeline itself.
Q6. Your tabular model uses a "merchant_id" column with 40,000 distinct values. One-hot encoding is a poor choice here — which TWO of the following are genuinely better options and why?
- A) One-hot is actually the ideal choice here — all 40,000 binary columns together give the model the maximum possible information about each individual merchant's identity and behavior.
- B) One-hot would create ~40,000 mostly-zero columns — huge, sparse, and starved of data per column. Grouping rare merchants into "Other," or hashing ids into a fixed bucket count, avoids this.
- C) Smoothed target encoding — blending each merchant's outcome average toward the global mean — works well here too, but must be computed strictly out-of-fold within CV to avoid leaking the label.
- D) The only valid option is ordinal encoding — assigning each merchant an arbitrary integer from 1 to 40,000 — which compactly preserves all categorical information without any loss at all.
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →