Start learning

Bagging and Boosting

Lesson 34. About 6 minutes.

Bagging and boosting are two ways of combining many simple models into one strong one. Bagging trains models side by side and averages them; boosting trains them one after another, each fixing the last one's mistakes.

Your Path Through This Lesson

0 of 5 done

By the End of This Lesson You Can
  • Explain why a group of models beats one model.
  • Describe how bagging and boosting each build their group.
  • Know which problem each one solves: variance or bias.

The Wisdom of Crowds

Ask one person to guess the number of sweets in a jar and they might be far off. Ask 200 people and take the average, and the answer is usually remarkably close. Individual errors point in different directions and cancel out.

An ensemble does the same with models. One decision tree is unstable and overfits; many trees, combined well, are among the most accurate models in machine learning.

Bagging: Many Models in Parallel

Bagging stands for bootstrap aggregating.

  1. Make many bootstrap samples: random samples of the training data, the same size as the original, drawn with replacement (so some rows appear twice and others not at all).
  2. Train one model, usually a deep tree, on each sample. They're independent, so they can train in parallel.
  3. Combine: majority vote for classification, average for regression.

Each tree overfits its own sample in its own way. Averaging cancels those individual quirks, so bagging mainly reduces variance.

Note

On average a bootstrap sample contains about 63% of the distinct original rows. The ~37% left out of each sample are called out-of-bag rows, and they give a free test set for that tree.

Boosting: Many Models in Sequence

Boosting builds the group one model at a time:

  1. Train a simple model, often a very shallow tree.
  2. Look at what it got wrong.
  3. Train the next model to focus on those mistakes.
  4. Repeat, then combine all models with weights.

Each new model fixes what the group still gets wrong, so boosting mainly reduces bias. It can turn very weak models, like one-question trees, into a strong one.

Side by Side

Bagging Boosting
Training order Parallel, independent Sequential, each depends on the last
Each model sees A bootstrap sample All data, re-weighted or re-targeted at the errors
Typical base model Deep tree (low bias, high variance) Shallow tree (high bias, low variance)
Mainly reduces Variance Bias
Risk Little overfitting from adding more trees Can overfit with too many rounds
Famous examples Random forest AdaBoost, gradient boosting, XGBoost

Comparing Them

python
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import AdaBoostClassifier, BaggingClassifier
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)

models = {
    "single deep tree": DecisionTreeClassifier(random_state=0),
    "single stump": DecisionTreeClassifier(max_depth=1, random_state=0),
    "bagging 100 deep trees": BaggingClassifier(DecisionTreeClassifier(), n_estimators=100, random_state=0),
    "boosting 100 stumps": AdaBoostClassifier(DecisionTreeClassifier(max_depth=1), n_estimators=100, random_state=0),
}
for name, model in models.items():
    print(f"{name:>23}: CV accuracy {cross_val_score(model, X, y, cv=5).mean():.3f}")
Output
       single deep tree: CV accuracy 0.917
           single stump: CV accuracy 0.900
 bagging 100 deep trees: CV accuracy 0.958
    boosting 100 stumps: CV accuracy 0.977

A single deep tree overfits; bagging many of them is clearly better. A single stump (one question) is too simple; boosting a hundred of them is better still.

Tip

In practice you'll rarely use BaggingClassifier directly. Its improved version, the random forest, is the next lesson.

Try It Yourself
  1. Change n_estimators for both ensembles to 10, then 300. Which one benefits more from extra models?
  2. Give the bagging ensemble stumps (max_depth=1) instead of deep trees. Why does it do worse?
The Key Idea

Bagging averages independent models to cut variance. Boosting chains models that fix each other's mistakes to cut bias.

Summary

  • An ensemble combines many models; their errors partly cancel.
  • Bagging: bootstrap samples, parallel models, vote or average. Cuts variance.
  • Boosting: sequential models, each fixing past errors. Cuts bias.
  • Next: the most popular bagging method, the random forest.