Random Forest: Average Many Different Trees
One tree can be sensitive to the sample
A Decision Tree can change substantially when a few training rows change. Random Forest reduces this instability by training many trees on varied samples and feature choices, then combining their predictions.
Key Points
- A forest is an ensemble of trees.
- Each tree sees a bootstrap sample or related variation.
- Each split considers a random subset of features.
- Classification commonly combines tree votes or probabilities.
Bagging reduces variance by averaging
Bootstrap aggregating, or bagging, trains a model on samples drawn with replacement and averages the results. If individual trees make partly different errors, the average can be more stable than one tree.
Key Points
- Averaging helps when errors are not perfectly correlated.
- More trees usually improve stability until returns level off.
- The forest can still inherit bad features, leakage, or biased training data.
- Ensembling reduces some variance, not every failure mode.
See a majority vote
1tree_predictions = ["safe", "spam", "safe", "safe", "spam"]
2votes = {label: tree_predictions.count(label) for label in sorted(set(tree_predictions))}
3forest_prediction = max(votes, key=votes.get)
4
5print("Votes:", votes)
6print("Forest prediction:", forest_prediction)Votes: {'safe': 3, 'spam': 2}
Forest prediction: safeA probability-like vote share would be 3 ÷ 5 = 0.6 safe for this toy forest. Real implementations may average class probabilities from trees rather than only hard votes.
Out-of-bag rows provide an internal check
A bootstrap sample often leaves some training rows out of a particular tree. Those out-of-bag rows can act as an internal evaluation sample for that tree, producing an out-of-bag score across the forest.
Key Points
- Out-of-bag evaluation uses rows not selected for a tree.
- It can estimate performance without a separate validation split in some workflows.
- It does not excuse checking time, group, or leakage boundaries.
- A final untouched test set is still valuable for the locked pipeline.
Forest settings shape the tradeoff
Key Points
- Number of trees affects stability and compute.
- Maximum depth affects each tree’s capacity.
- Minimum leaf size prevents fragile branches.
- Maximum features controls tree diversity.
- Class weights can address unequal error importance.
- Random seed controls reproducibility.
More trees do not always repair an overfit feature pipeline. Tune the forest settings with the same validation discipline as any other model.
Feature importance is a clue, not a verdict
Random forests can expose impurity-based or permutation-based importance measures. These are useful for investigation but can be misleading with correlated features, high-cardinality columns, and different feature scales or opportunities to split.
Key Points
- Importance is not causation.
- Correlated features can share or distort importance.
- Permutation importance should be measured on held-out data.
- Check whether an important feature is available before prediction.
Many trees are not always easier to explain
A single tree offers one path. A forest offers many paths and a combined result. You can explain individual predictions with path or contribution tools, but the explanation is more involved than reading one shallow tree.
Key Points
- Use a simple tree as a communication baseline.
- Use forest explanations for the actual ensemble cautiously.
- Report validation and test performance, not only importance charts.
- Prefer a simpler model when a tiny gain does not justify complexity.
Compare one tree with a forest
1models = [
2 {"name": "deep_tree", "train_f1": 1.00, "test_f1": 0.71},
3 {"name": "random_forest", "train_f1": 0.98, "test_f1": 0.82},
4]
5
6for model in models:
7 print(f"{model['name']}: train={model['train_f1']:.2f}, test={model['test_f1']:.2f}")deep_tree: train=1.00, test=0.71 random_forest: train=0.98, test=0.82
The example illustrates a common goal: a small amount of training fit can be traded for better held-out performance and stability. The numbers are not universal guarantees.
Where people get confused
Key Points
- “A forest is just one very large tree.” It combines many trees trained with randomness.
- “More trees always fix overfitting.” More trees stabilise the ensemble but do not fix leakage or weak features.
- “Feature importance proves what causes the target.” Importance describes use inside a predictive model.
- “Randomness makes results unreproducible.” Store the random seed and data version.
- “The forest probability is automatically calibrated.” Vote shares may need calibration checks.
Quick check
Why can a Random Forest be more stable than one tree?
It averages or votes across varied trees. Their partly different errors can cancel, reducing sensitivity to one training sample.
What does the random feature subset do?
It encourages trees to use different features at splits, increasing diversity so the forest does not make exactly the same errors in every tree.
What are out-of-bag rows?
They are training rows not selected in a particular bootstrap sample and can provide an internal check for that tree.
Why should feature importance be interpreted cautiously?
Correlated features, feature cardinality, leakage, and the difference between prediction and causation can make importance misleading.
Recap
Key Points
- Random Forest combines many varied Decision Trees.
- Bagging and feature randomness can reduce variance.
- Depth, leaf size, feature sampling, weights, and tree count are hyperparameters.
- Importance helps investigate but is not causal evidence.
- The next lesson builds models sequentially with Gradient Boosting.
Theia
Can you explain this concept?
Practise Random Forest: Average Many Different Trees with realistic interview questions. Choose spoken or typed answers, get feedback on your reasoning, then return to the material that needs work.
Sign in to practise this conceptThe session stays focused on this lesson. Your selected interview level sets the bar.