Decision Tree Regression and Pruning

A regression tree predicts a number by splitting the data into groups and predicting each group's average. Pruning keeps the tree from growing so deep that it memorises the training data.

By the End of This Lesson You Can

Explain how a regression tree splits and predicts.
Compute the variance reduction of a split by hand.
Control overfitting with pre-pruning and cost-complexity post-pruning.

Trees for Numbers

A classification tree ends in leaves that say "pass" or "fail". A regression tree ends in leaves that hold a number: the average target value of the training rows in that leaf.

So a regression tree's prediction is a step function: flat within each leaf, jumping between leaves.

How It Chooses Splits

Instead of entropy or Gini, a regression tree measures how spread out the target values are, using the mean squared error around the node's average (the variance). A good split creates children whose values are close to their own averages.

A Worked Example

Five students again, hours and marks: (1, 20), (2, 40), (3, 50), (4, 40), (5, 50). The mean mark is 40.

Parent MSE: $\frac{( 20 - 40 ) ^{2} + 0 + 1 0 ^{2} + 0 + 1 0 ^{2}}{5} = \frac{600}{5} = 120$ .

Try "hours ≤ 1.5": left = {20}, right = {40, 50, 40, 50} with mean 45.

Left MSE = 0. Right MSE $= \frac{25 + 25 + 25 + 25}{4} = 25$ .
Weighted: $\frac{1}{5} (0) + \frac{4}{5} (25) = 20$ . Reduction $= 120 - 20 = 100$ .

Try "hours ≤ 2.5": left = {20, 40} (mean 30), right = {50, 40, 50} (mean 46.7).

Left MSE = 100. Right MSE $\approx 22.2$ .
Weighted: $\frac{2}{5} (100) + \frac{3}{5} (22.2) = 53.3$ . Reduction $= 66.7$ .

The first split wins. It separates Noman's very low mark from everyone else.

python

import numpy as np
from sklearn.tree import DecisionTreeRegressor, export_text

hours = np.array([[1], [2], [3], [4], [5]])
marks = np.array([20, 40, 50, 40, 50])

stump = DecisionTreeRegressor(max_depth=1).fit(hours, marks)
print(export_text(stump, feature_names=["hours"]))
print("predictions:", stump.predict([[1], [3], [6]]))

Output

|--- hours <= 1.50
|   |--- value: [20.00]
|--- hours >  1.50
|   |--- value: [45.00]

predictions: [20. 45. 45.]

The tree split at 1.5 hours, as we found by hand, and predicts each side's average: 20 and 45. Note that 6 hours also gets 45, because a tree can't predict beyond the values it saw.

Deep Trees Overfit

Left alone, a tree keeps splitting until every leaf holds one training row, so it memorises the data exactly.

python

from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split

X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)

for depth in [2, 3, 5, 10, None]:
    t = DecisionTreeRegressor(max_depth=depth, random_state=0).fit(X_train, y_train)
    print(f"max_depth={str(depth):>4}: train R2 {t.score(X_train, y_train):.3f}   test R2 {t.score(X_test, y_test):.3f}")

Output

max_depth=   2: train R2 0.496   test R2 0.105
max_depth=   3: train R2 0.578   test R2 0.087
max_depth=   5: train R2 0.716   test R2 0.096
max_depth=  10: train R2 0.945   test R2 -0.126
max_depth=None: train R2 1.000   test R2 -0.182

A perfect 1.0 on training data and a poor test score: classic overfitting.

Pruning

Pruning limits how complex a tree can get.

Pre-Pruning: Stop Early

Set limits before training:

Setting	Stops the tree when…
`max_depth`	it reaches this many levels
`min_samples_split`	a node has fewer rows than this
`min_samples_leaf`	a split would leave a leaf with fewer rows than this
`max_leaf_nodes`	it already has this many leaves

Post-Pruning: Grow, Then Cut Back

Cost-complexity pruning grows the full tree, then removes branches that add little. It minimises:

error of the tree + α \times number of leaves

A larger $α$ (ccp_alpha) charges more per leaf, so the tree gets smaller. (cost_complexity_pruning_path can list every useful $α$ for you; here we just try a range.)

python

for a in [0, 10, 50, 100, 200, 400, 800]:
    t = DecisionTreeRegressor(ccp_alpha=a, random_state=0).fit(X_train, y_train)
    print(f"ccp_alpha={a:>4}: leaves {t.get_n_leaves():>3}   test R2 {t.score(X_test, y_test):.3f}")

Output

ccp_alpha=   0: leaves 318   test R2 -0.182
ccp_alpha=  10: leaves  81   test R2 -0.077
ccp_alpha=  50: leaves  15   test R2 0.080
ccp_alpha= 100: leaves   8   test R2 0.092
ccp_alpha= 200: leaves   5   test R2 0.091
ccp_alpha= 400: leaves   4   test R2 0.105
ccp_alpha= 800: leaves   2   test R2 0.074

As ccp_alpha grows, the tree shrinks from 318 leaves to a handful, and the test score climbs from below zero to about 0.1, before dropping again when the tree becomes too simple. Choose ccp_alpha (or max_depth) with cross-validation.

Notice that even the best single tree only reaches about 0.1 here, while multiple linear regression reaches about 0.36 on this same split. A single tree's step-shaped predictions suit this dataset poorly. Combining many trees, in the next part, closes much of that gap.

Try It Yourself

Find the best min_samples_leaf for the diabetes data with GridSearchCV, trying 1, 5, 10, 20 and 40.
Fit a depth-2 tree on our five students and print its rules. Can you predict the leaves' values before running it?

The Key Idea

Regression trees predict the average of each leaf and split to reduce squared error. Prune them, before or after growing, to stop overfitting.

Summary

Regression trees predict leaf averages and split to reduce squared error.
For our students, the best first split is hours ≤ 1.5, reducing MSE from 120 to 20.
Unlimited trees overfit; prune with max_depth, min_samples_leaf or ccp_alpha.
Next: a complete decision tree project in scikit-learn.

Your Path Through This Lesson

Trees for Numbers

How It Chooses Splits

A Worked Example

Deep Trees Overfit

Pruning

Pre-Pruning: Stop Early

Post-Pruning: Grow, Then Cut Back

Summary