Random Forests

Bagging, OOB error, feature importance, hyperparameter sensitivity

The last module ended on eight loan applicants and a hard number: flip 2 of 8 labels and the root question reverses, and a brand-new applicant gets opposite predictions from the two trees even though each is 100% accurate on its own data. That instability was called variance, and it was named as a bug, not fixed. Time to fix it.

In 1906 the scientist Francis Galton was at a country fair where a crowd was trying to guess the weight of an ox. Nearly eight hundred people wrote down a number. No single guess was right — some were wildly high, some wildly low. But when Galton averaged them all, the crowd's answer came out at 1197 pounds. The ox weighed 1198. The crowd as a whole beat almost every individual in it, cattle experts included. That is the wisdom of the crowd, and a random forest is exactly this trick applied to decision trees.

Recall the debt/income tree from last module: it is twitchy precisely because its very first split is chosen from the whole dataset at once, so a couple of changed rows can flip it. But look closer at that flaw. If different slices of data grow different trees that make *different* mistakes, then averaging a whole crowd of them should cancel those mistakes out — the errors point in random directions and wash away, while the real signal they mostly agree on survives. So: grow many trees, let them vote, and the group is far steadier than any single tree. The question is exactly how much steadier, and that turns out to be a number you can compute.

A crowd only helps if it disagrees — put a number on "helps."

Here is the crucial catch. The ox crowd worked because people guessed *independently* — their errors were unrelated. If everyone had copied their neighbour, the "crowd" would be one guess repeated eight hundred times, and averaging would do nothing. The same holds for trees: if income is the strongest predictor of default, every tree handed the full dataset will split on income first and come out nearly identical — all making the same mistakes.

Make that precise. Suppose each tree's prediction carries some error with variance σ² = 100 (in squared-price units — think an individual tree's typical miss is about √100 = 10k on a house-price task), and any two trees' errors share correlation ρ, because they overlap: bootstrap resampling and shared features mean two trees trained on the same dataset agree more than two trees trained on unrelated problems would. For n trees averaged together, each with variance σ² and pairwise correlation ρ, the variance of their average is:

Var(average) = σ²/n + ((n−1)/n)·ρσ²

Plug in n=100 trees at ρ=0.5 (typical for trees given the full feature set every time, so they keep finding the same top splits): Var = 100/100 + (99/100)·0.5·100 = 1 + 49.5 = 50.5. Compare that to a single tree's variance of 100 — averaging did cut it roughly in half, but nowhere near the 100× a naive "more trees = less noise" intuition might expect. Push n to 1000: Var = 100/1000 + (999/1000)·50 = 0.1 + 49.95 = 50.05. A tenfold increase in tree count bought a movement from 50.5 to 50.05 — essentially nothing. As n→∞, Var(average) → ρσ² = 0.5×100 = 50, a floor set entirely by ρ that no amount of extra trees can push below.

Now change the *other* dial. Keep n=100 but cut ρ to 0.1 (decorrelated trees): Var = 100/100 + (99/100)·0.1·100 = 1 + 9.9 = 10.9 — down from 50.5, and the *asymptotic* floors themselves (ρσ²=50 vs ρσ²=10) are exactly 5× apart, a bigger swing than ten times the tree count bought. That's the whole justification for forcing trees to disagree: the lever that matters is ρ, not n.

A random forest drives ρ down with two tricks.

First: instead of handing every tree the whole dataset, give each one a random resample — draw rows with replacement until you have a fresh training set of the same size, where some rows repeat and others are left out. That resampling trick has a name, bagging (short for bootstrap aggregating).

Second: at each split, do not let the tree look at all the features — show it only a random handful (a common choice is the square root of the total). Now even the trees that *would* have latched onto income are sometimes forced to find other patterns; this is random feature selection, the second decorrelation lever. It's the sharper move of the two — bagging alone only decorrelates trees mildly (bootstrap resamples still overlap heavily), while restricting features at every split is what actually pushes ρ from something like 0.5 down toward 0.1, because it stops every tree from making the same first cut. That difference — 50 versus 10 as the variance floor — is the entire reason "more trees" plateaus around a few hundred while "fewer features per split" keeps moving the needle.

A free validation set, for nothing

That same row-resampling step that decorrelates the trees also hands you a bonus, and it has an exact source. A bootstrap resample of n rows drawn with replacement from n rows leaves some rows out entirely — the probability any single row is *never* drawn across n draws is (1−1/n)ⁿ, which converges to 1/e ≈ 0.368 as n grows. So roughly 36.8% of rows — "about a third" — never appear in a given tree's training set, and are that tree's out-of-bag rows. For each row you can ask only the trees that did *not* train on it to predict it, and check them against the truth. That gives an honest estimate of test performance — the out-of-bag (OOB) error — for free, with no separate validation set set aside. If the OOB error and your real test error disagree badly, that is a red flag for a distribution shift or a data leak.

The one trap to remember

A random forest built for regression cannot predict outside the range it has seen. Every tree's answer is an average of training values in a leaf, and an average of a forest of averages is still boxed in by the training data. Return to that same house-price task from above: train on prices up to 800k and the forest will never output more than 800k, no matter how enormous the house. If your target drifts upward over time — prices rising year over year — the forest will quietly under-predict the future while looking perfectly healthy on past data. When the target trends, reach for a model that can extrapolate.

The hyperparameters worth knowing

A forest has more knobs than "how many trees" — and now you can say precisely why `n_estimators` isn't the one to lean on. `n_estimators` is the tree count (more never hurts accuracy, just compute — recall it moved the variance floor from 50.5 to only 50.05 going from 100 to 1000 trees). `max_features` is the real diversity dial — how many features each split may consider (√p is a common default; fewer means lower ρ, and the arithmetic above showed dropping ρ from 0.5 to 0.1 cut the variance floor 5×). `max_depth` and `min_samples_leaf` control how deep each tree grows. `bootstrap` and `max_samples` control the resampling (turn bootstrap off and you lose OOB). `class_weight` up-weights a rare class, and `criterion` picks Gini/entropy or the regression split rule. In an interview, name `n_estimators`, `max_features`, `max_depth`, `min_samples_leaf`, and `class_weight` as the ones you'd actually tune — and lead with `max_features` as the one that moves the needle.

What the forest fixes — and what it doesn't

Be precise about the bias-variance story. Averaging many de-correlated trees mainly reduces variance — that's the whole wisdom-of-the-crowd effect. It does *not* reduce bias much: if the individual trees are systematically wrong because the signal is weak or a key feature is missing, averaging a thousand of them just gives you a very stable version of the same wrong answer. So a forest of deep trees can still be biased. Variance is what the crowd kills; bias you fix by adding signal, not trees.

OOB is handy but not bulletproof

Out-of-bag error is a free estimate, but it assumes rows are independent and identically distributed. It quietly *lies* when they're not. With time-series data, OOB lets a tree "see the future" (rows from later dates train a tree that scores earlier ones), so it's optimistic — you need a time-based split instead. With grouped data (many rows per customer), OOB leaks across the group. And under distribution shift or leakage, OOB reflects the training distribution, not production. So use OOB as a cheap sanity check, not as a replacement for a properly designed validation scheme.

Reading importances, carefully

Two importance traps. The built-in impurity importance — a different use of "Gini" than the split criterion above; this one ranks features by how much they cut impurity across all their splits, after training, rather than choosing questions during it — is biased toward high-cardinality features, same issue as a single tree. Permutation importance is better but has its *own* correlated-feature trap: if two features carry the same information, shuffling one barely hurts accuracy because its twin still supplies the signal, so *both* look unimportant even though the information is vital. Don't read low permutation importance as "useless" when features are correlated. And a forest is far less interpretable than a single tree — for real explanation reach for permutation importance, partial-dependence/ICE plots, and SHAP, all read with the correlation caveat in mind.

When the forest loses to boosting

A random forest is a fantastic *baseline*, but on tabular-accuracy leaderboards gradient boosting usually wins. The reason ties back to bias-variance: forests reduce variance but leave bias on the table, while boosting attacks bias by building trees sequentially, each correcting the last. The trade: boosting needs more careful tuning and is more sensitive to noise, outliers, and leakage (it will happily fit a leak that a forest partly averages away). So: forest for a fast, robust baseline; boosting when you'll invest tuning effort to squeeze out the last few points.

Under imbalance. A forest inherits the single tree's problem — it chases the majority class and its vote proportions get unreliable. Use `class_weight='balanced'` (or `balanced_subsample`), stratified CV so folds keep the rare class, threshold moving on the predicted probabilities, and judge with PR-AUC, balanced accuracy, or recall@K rather than raw accuracy.

Key points

• Use a random forest when you want a strong, low-effort baseline on tabular data. It takes mixed feature types as they come, needs no scaling, shrugs off irrelevant features, and hands you free out-of-bag validation. Its defaults work well with almost no tuning, which makes it the reliable first thing to try on classification or regression. Reach for gradient boosting instead when you need to squeeze out the last couple of accuracy points and are willing to tune carefully — but for a fast, trustworthy baseline, the forest is hard to beat.
• The trap: thinking more trees is the lever. Past a couple hundred, adding trees barely moves anything. For n trees with per-tree error variance σ² and pairwise correlation ρ, Var(average) = σ²/n + ((n−1)/n)ρσ² — and as n→∞ that converges to a floor of ρσ², not zero. With σ²=100 and ρ=0.5, going from 100 to 1000 trees only moves the variance from 50.5 to 50.05, because the floor itself is 50. Cutting ρ to 0.1 instead — by restricting features per split — drops the floor to 10, a 5× win that ten times the tree count couldn't buy. Tune diversity (max_features), not quantity (n_estimators).
• The check: compare the out-of-bag error to your held-out test error. Out-of-bag error is a free, honest estimate of how the forest does on data like its training set. If OOB says 10% but your real test error is 25%, something is off — usually the test data comes from a different distribution than training, or a leak made training look too easy. When the two disagree, compare the feature distributions of train and test before trusting the model in production.
• Be precise: the forest reduces variance, not bias — and OOB isn't bulletproof. Averaging de-correlated trees kills variance (the wisdom-of-the-crowd effect) but barely touches bias, so a forest of weak or wrong trees is just a stable version of the same wrong answer — fix bias with signal, not more trees. And OOB assumes i.i.d. rows: it's optimistic on time-series (a tree sees the future), leaks across grouped data (many rows per customer), and reflects the training distribution under shift. Use OOB as a cheap check, not a substitute for a time- or group-aware validation split. Tune `max_features`, `max_depth`, `min_samples_leaf`, `class_weight` — not just `n_estimators`.
• Read importances with the correlation caveat, and know when boosting wins. Built-in impurity importance is biased toward high-cardinality features; permutation importance is better but has its own trap — with two correlated features, shuffling one barely hurts (the twin still carries the signal), so both look unimportant even when vital. For real interpretation use permutation importance, PDP/ICE, and SHAP, all read cautiously. And a forest is a strong baseline but gradient boosting usually wins on tabular accuracy: boosting attacks the bias a forest leaves behind, at the cost of more tuning and more sensitivity to noise, outliers, and leakage. Under imbalance, use `class_weight='balanced'`, stratified CV, threshold moving, and PR-AUC.

Recap

• Random forest = wisdom of the crowd on trees — grow many, let them vote, errors cancel.
• Only works if trees are diverse: bagging (random resamples) + random feature subset at each split, to drive down ρ.
• Var(average) = σ²/n + ((n−1)/n)ρσ² → ρσ² as n→∞ — a correlation floor, not zero.
• Worked: σ²=100, ρ=0.5 — n=100→1000 moves variance 50.5→50.05 (n_estimators plateaus); ρ=0.5→0.1 at n=100 moves it 50.5→10.9 (max_features is the real lever).
• Reduces variance, not bias — a forest of biased trees is a stable version of the same wrong answer.
• Free OOB validation — (1−1/n)ⁿ→1/e≈36.8% of rows left out of each bootstrap, validate that tree for free.
• OOB assumes i.i.d. rows — optimistic on time-series/grouped data; use a time- or group-aware split instead.
• Silent trap: a regression forest can never predict outside its training range (no extrapolation).
• Importances: impurity importance biased toward high-cardinality features; permutation importance's own trap is correlated features looking falsely unimportant.

Check your understanding