Start learning

XGBoost

Lesson 38. About 6 minutes.

XGBoost (eXtreme Gradient Boosting) is a fast, regularised implementation of gradient boosting. It's one of the most successful algorithms for tabular data, for both classification and regression.

Your Path Through This Lesson

0 of 5 done

By the End of This Lesson You Can
  • Explain what XGBoost adds to plain gradient boosting.
  • Train an XGBClassifier and an XGBRegressor.
  • Tune the important settings and use early stopping.

What Makes It "Extreme"?

XGBoost follows the same recipe as gradient boosting: small trees added one at a time, each fixing the remaining errors. On top of that it adds:

Feature What it does
Regularisation Penalises complex trees: too many leaves (gamma) or large leaf values (reg_lambda, reg_alpha)
Second-order gradients Uses the curvature of the loss as well as its slope, for better, faster steps
Missing values Learns which way to send missing values at each split, so you don't need to fill them
Speed Parallel split finding, histogram binning, optional GPU support
Row and column sampling subsample and colsample_bytree add randomness, like a random forest
Early stopping Stops adding trees when a validation score stops improving

Install It

XGBoost is a separate library:

bash
pip install xgboost

It follows the scikit-learn pattern: fit, predict, score.

Classification

python
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from xgboost import XGBClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)

clf = XGBClassifier(n_estimators=300, learning_rate=0.05, max_depth=3, subsample=0.8, colsample_bytree=0.8, random_state=0)
clf.fit(X_train, y_train)
print(f"test accuracy: {clf.score(X_test, y_test):.3f}")
print("first 5 probabilities of class 1:", clf.predict_proba(X_test[:5])[:, 1].round(3))
Output
test accuracy: 0.947
first 5 probabilities of class 1: [0.001 0.998 0.001 0.989 0.001]

Regression

The regressor works the same way. Here it is on the diabetes data, where a single tree reached only about 0.1 R²:

python
from sklearn.datasets import load_diabetes
from xgboost import XGBRegressor

Xd, yd = load_diabetes(return_X_y=True)
Xd_train, Xd_test, yd_train, yd_test = train_test_split(Xd, yd, test_size=0.25, random_state=0)

reg = XGBRegressor(n_estimators=400, learning_rate=0.03, max_depth=2, subsample=0.8, random_state=0)
reg.fit(Xd_train, yd_train)
print(f"test R2: {reg.score(Xd_test, yd_test):.3f}")
Output
test R2: 0.269

Early Stopping

Instead of guessing n_estimators, give XGBoost a validation set and let it stop when the score hasn't improved for a number of rounds:

python
X_tr, X_val, y_tr, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=0, stratify=y_train)

clf = XGBClassifier(n_estimators=2000, learning_rate=0.05, max_depth=3, early_stopping_rounds=30,
                    eval_metric="logloss", random_state=0)
clf.fit(X_tr, y_tr, eval_set=[(X_val, y_val)], verbose=False)
print("trees actually used:", clf.best_iteration + 1)
print(f"test accuracy: {clf.score(X_test, y_test):.3f}")
Output
trees actually used: 107
test accuracy: 0.947

It was allowed 2,000 trees but stopped far earlier, once the validation log loss stopped improving.

Watch Out

The validation set used for early stopping is part of the model-building process. Keep a separate test set, as above, for the final honest score.

The Settings That Matter Most

Setting What it controls Good starting range
n_estimators Maximum number of trees 100–2000 (with early stopping)
learning_rate Step size per tree 0.01–0.1
max_depth Tree depth 3–6
subsample Fraction of rows per tree 0.6–1.0
colsample_bytree Fraction of features per tree 0.6–1.0
reg_lambda L2 penalty on leaf values 1 (default) to 10
min_child_weight Minimum "weight" needed in a leaf 1–10

A common approach: fix learning_rate at 0.05, use early stopping for n_estimators, then tune max_depth, subsample and colsample_bytree with cross-validation.

Tip

Other libraries do the same job with different strengths: LightGBM is often even faster on large data, and CatBoost handles categorical features with little preprocessing. scikit-learn's HistGradientBoosting models are a good choice when you don't want an extra dependency.

Try It Yourself
  1. Rerun the classifier with max_depth=6 and then max_depth=2. Which generalises better?
  2. Set a few values in X_train to np.nan and fit again. Does XGBoost complain?
The Key Idea

XGBoost = gradient boosting + regularisation + speed + built-in handling of missing values. Use early stopping to choose the number of trees.

Summary

  • XGBoost is regularised, fast gradient boosting with native missing-value support.
  • XGBClassifier and XGBRegressor follow the scikit-learn API.
  • Use early stopping with a validation set to pick the number of trees.
  • Tune max_depth, learning_rate, subsample and colsample_bytree.
  • Next part: a very different kind of model, support vector machines.