Gradient Boosting

The Idea: Fix the Leftovers

Imagine guessing a student's marks in rounds:

1. Start with a simple guess: the average mark.
2. Look at the residuals: how far off each guess is.
3. Train a small tree to predict those residuals.
4. Add a fraction of that tree's prediction to your guess.
5. Repeat: new residuals, new tree, small correction.

Each tree fixes a bit of what's left. After a few hundred rounds, the corrections add up to an accurate model.

where each $f_m$ is a small tree and $\eta$ is the learning rate.

By Hand With Our Students

Marks: Noman 20, Lamia 40, Rafi 50, Tanvir 40, Arif 50 (hours 1–5). Learning rate $\eta =0.1$.

Step 0. Start with the mean: every prediction is 40.

Step 1. Residuals ($y-\hat{y}$): −20, 0, 10, 0, 10.

Step 2. Fit a one-split tree to the residuals. The best split (as in the regression tree lesson) is hours ≤ 1.5: the left leaf predicts −20, the right leaf predicts the average of 0, 10, 0, 10 = 5.

Step 3. Update with $\eta =0.1$:

Every prediction moved a little towards the truth. New residuals: −18, −0.5, 9.5, −0.5, 9.5. The next tree works on those.

import numpy as np
from sklearn.ensemble import GradientBoostingRegressor

hours = np.array([[1], [2], [3], [4], [5]])
marks = np.array([20, 40, 50, 40, 50])

gb = GradientBoostingRegressor(n_estimators=3, learning_rate=0.1, max_depth=1).fit(hours, marks)
for step, pred in enumerate(gb.staged_predict(hours), start=1):
    print(f"after tree {step}: {pred.round(2)}")

after tree 1: [38.  40.5 40.5 40.5 40.5]
after tree 2: [36.2  40.95 40.95 40.95 40.95]
after tree 3: [34.58 41.36 41.36 41.36 41.36]

After tree 1 the predictions are 38 and 40.5, exactly as calculated.

Why "Gradient"?

The residual $y-\hat{y}$ is the negative gradient of the squared-error loss with respect to the prediction. So each tree takes one step of gradient descent, not on the coefficients, but on the predictions themselves. Swap in a different loss (absolute error, log loss for classification) and the same machinery works: each tree fits that loss's negative gradient.

Learning Rate vs Number of Trees

The two settings trade off: halve the learning rate and you need roughly twice as many trees.

from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split

X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)

model = GradientBoostingRegressor(n_estimators=500, learning_rate=0.05, max_depth=2, subsample=0.8, random_state=0)
model.fit(X_train, y_train)

test_scores = [np.mean((y_test - p) ** 2) for p in model.staged_predict(X_test)]
best = int(np.argmin(test_scores)) + 1
print(f"best number of trees: {best}")
print(f"test R2 with all 500 trees: {model.score(X_test, y_test):.3f}")

best number of trees: 52
test R2 with all 500 trees: 0.210

After a point, extra trees start fitting noise and the test error creeps back up. In practice you find that point automatically with early stopping (n_iter_no_change) or cross-validation.

Key Settings

Summary

• Start from the mean; each tree fits the current residuals; add $\eta$ times its prediction.
• "Gradient" because residuals are the negative gradient of the squared-error loss.
• Learning rate and number of trees trade off; keep trees shallow.
• Use early stopping or cross-validation to choose the number of trees.
• Next: the optimised, regularised version used in competitions and industry, XGBoost.