Start learning

Random Forest

Lesson 35. About 7 minutes.

A random forest is a large group of decision trees, each trained on a random sample of the data and allowed to look at only a random subset of features at each split. Their votes are combined into one prediction.

Your Path Through This Lesson

0 of 5 done

By the End of This Lesson You Can
  • Explain the two kinds of randomness in a random forest.
  • Train random forests for classification and regression.
  • Use out-of-bag scores and feature importances.

Bagging Plus One More Trick

A random forest is bagging with decision trees, plus one extra idea:

  1. Random rows: each tree trains on a bootstrap sample of the data.
  2. Random features: at every split, a tree may only choose from a random subset of the features.

Why the second step? If one feature is very strong, every bagged tree would split on it first, and the trees would all look alike. Similar trees make similar mistakes, and averaging them doesn't help much. Forcing each split to consider only some features makes the trees different from each other, so their errors cancel better.

Setting Meaning Default
n_estimators Number of trees 100
max_features Features considered at each split "sqrt" for classification, all (1.0) for regression
max_depth Depth limit per tree None (fully grown)
min_samples_leaf Minimum rows per leaf 1
oob_score Score each tree on the rows it didn't see False
n_jobs CPU cores to use (-1 = all) 1

Classification

python
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)

tree = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)
forest = RandomForestClassifier(n_estimators=300, oob_score=True, random_state=0, n_jobs=-1).fit(X_train, y_train)

print(f"single tree  : test accuracy {tree.score(X_test, y_test):.3f}")
print(f"random forest: test accuracy {forest.score(X_test, y_test):.3f}")
print(f"out-of-bag   : accuracy      {forest.oob_score_:.3f}")
Output
single tree  : test accuracy 0.906
random forest: test accuracy 0.953
out-of-bag   : accuracy      0.967

The out-of-bag (OOB) score comes free: each tree is scored on the training rows left out of its bootstrap sample. It's a quick estimate of test performance without a separate validation set.

Regression

In the last part, a single tree managed only about 0.1 R² on the diabetes data. A forest averages hundreds of trees, so its predictions are much smoother:

python
from sklearn.datasets import load_diabetes
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeRegressor

Xd, yd = load_diabetes(return_X_y=True)
Xd_train, Xd_test, yd_train, yd_test = train_test_split(Xd, yd, test_size=0.25, random_state=0)

one_tree = DecisionTreeRegressor(max_depth=2, random_state=0).fit(Xd_train, yd_train)
rf = RandomForestRegressor(n_estimators=300, min_samples_leaf=5, random_state=0, n_jobs=-1).fit(Xd_train, yd_train)
lin = LinearRegression().fit(Xd_train, yd_train)
print(f"best single tree : test R2 {one_tree.score(Xd_test, yd_test):.3f}")
print(f"random forest    : test R2 {rf.score(Xd_test, yd_test):.3f}")
print(f"linear regression: test R2 {lin.score(Xd_test, yd_test):.3f}")
Output
best single tree : test R2 0.105
random forest    : test R2 0.270
linear regression: test R2 0.359

The forest more than doubles the single tree's score. Linear regression still wins here, because this target depends on the features in a mostly straight-line way. No model is best everywhere: try a simple baseline before reaching for an ensemble.

How Many Trees?

More trees never cause overfitting in a random forest; they just stop helping after a point, and cost more time.

python
for n in [1, 5, 25, 100, 300]:
    m = RandomForestClassifier(n_estimators=n, random_state=0, n_jobs=-1).fit(X_train, y_train)
    print(f"{n:>3} trees: test accuracy {m.score(X_test, y_test):.3f}")
Output
  1 trees: test accuracy 0.889
  5 trees: test accuracy 0.942
 25 trees: test accuracy 0.936
100 trees: test accuracy 0.953
300 trees: test accuracy 0.953

Feature Importance

A forest's importances are averaged over all its trees, so they're far more stable than a single tree's:

python
import numpy as np
names = load_breast_cancer().feature_names
for i in np.argsort(-forest.feature_importances_)[:5]:
    print(f"{names[i]:>22}: {forest.feature_importances_[i]:.3f}")
Output
       worst perimeter: 0.134
          worst radius: 0.132
  worst concave points: 0.117
   mean concave points: 0.108
            worst area: 0.107
Watch Out

These importances (based on impurity decrease) can favour features with many distinct values. For a more trustworthy ranking, use sklearn.inspection.permutation_importance, which measures how much the score drops when each feature's values are shuffled.

Tip

A random forest is one of the best "first serious model" choices: it needs no scaling, handles mixed features, rarely overfits badly, and works well with default settings.

Try It Yourself
  1. Set max_features=None (every split sees all features) and compare test accuracy with the default "sqrt".
  2. Compute permutation_importance(forest, X_test, y_test, random_state=0) and compare its top five with the list above.
The Key Idea

Random forest = bootstrap samples + a random subset of features at each split + many trees voting.

Summary

  • Random forest = bagged trees + random feature subsets at each split.
  • Default settings work well: no scaling, few hyperparameters to worry about.
  • Use oob_score=True for a free performance estimate.
  • Importances are averaged over trees; use permutation importance for a fairer ranking.
  • Next: the first boosting algorithm, AdaBoost.