Start learning

Overfitting and Underfitting

Lesson 13. About 6 minutes, with a video walkthrough.

Overfitting is when a model learns the training data too well, including its noise, and fails on new data. Underfitting is when a model is too simple to learn the pattern at all.

Your Path Through This Lesson

0 of 5 done

By the End of This Lesson You Can
  • Recognise overfitting and underfitting from training and test scores.
  • Explain both with a simple everyday example.
  • List the usual fixes for each.

Two Kinds of Bad Student

Picture two students preparing for an exam.

  • Student A memorises every answer in last year's question paper, word for word. If the exam repeats those questions, perfect marks. If the questions change even a little, they're lost. That's overfitting.
  • Student B skims one chapter and decides "the answer is usually C". They do badly on last year's paper and this year's. That's underfitting.

What we want is Student C, who understands the ideas: good on practice papers and good on the real exam. That's a model that generalises.

Reading the Scores

You can't see overfitting by looking at training data alone, because an overfitted model looks brilliant there. You see it by comparing training and test performance:

Training score Test score What it means
Underfitting Low Low Model too simple to capture the pattern
Good fit High High, close to training The model learned the real pattern
Overfitting Very high Much lower The model memorised the training data

Watch all three fits in 20 seconds:

Seeing It Happen: Tree Depth

A decision tree can keep splitting until every training row sits alone in its own leaf. Let's watch training and test accuracy as we let it grow deeper, on a dataset of breast tumour measurements labelled benign or malignant:

python
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)

for depth in [1, 2, 3, 5, 8, None]:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=0).fit(X_train, y_train)
    print(f"max_depth={str(depth):>4}:  train {tree.score(X_train, y_train):.3f}   test {tree.score(X_test, y_test):.3f}")
Output
max_depth=   1:  train 0.930   test 0.895
max_depth=   2:  train 0.960   test 0.947
max_depth=   3:  train 0.967   test 0.947
max_depth=   5:  train 0.987   test 0.936
max_depth=   8:  train 1.000   test 0.912
max_depth=None:  train 1.000   test 0.912

With no depth limit (None), the tree scores 100% on the training data: it has memorised it. The test score doesn't follow. The gap between the two columns is the warning sign.

Why It Happens

A model overfits when it is flexible enough to fit noise and doesn't have enough data to tell noise from pattern. Every dataset has some randomness: a student who slept badly, a mistyped number. A very flexible model treats that randomness as a rule to learn.

A model underfits when it is too rigid: a straight line for a curved relationship, or too few features to explain the target.

How to Fix Each One

Fixes for overfitting Fixes for underfitting
Get more training data Use a more flexible model
Use a simpler model (lower degree, shallower tree) Add more or better features
Regularisation: Ridge, Lasso, ElasticNet Reduce regularisation
Remove features that are mostly noise Train for longer (for iterative models)
Use cross-validation to choose settings Check for bugs in the features
Combine many models (random forest, bagging)

You'll meet regularisation in the next few lessons. It's the main tool for keeping linear models from overfitting.

Watch Out

A tiny gap between training and test scores is normal, since the model has seen the training data. Worry when the gap is large, or when the test score falls as you make the model more complex.

Try It Yourself
  1. In the tree example, add min_samples_leaf=10 instead of a depth limit. Does the gap shrink?
  2. Rerun with test_size=0.5, so the tree trains on less data. Does overfitting get worse?
The Key Idea

Compare training and test scores: low and low means underfitting, high and much lower means overfitting.

Summary

  • Overfitting: great on training data, poor on new data. The model memorised noise.
  • Underfitting: poor on both. The model is too simple.
  • Spot them by comparing training and test scores.
  • Fix overfitting with more data, simpler models or regularisation; fix underfitting with more flexible models or better features.
  • Next: the same idea seen through two numbers, bias and variance.