Bias and Variance

The Dartboard

Imagine four players throwing darts at a bullseye. Each dart is one model, trained on a different random sample of data; the bullseye is the truth.

• Bias asks: on average, across many training sets, how far is the model from the truth?
• Variance asks: how much does the model change when the training set changes?

Connecting to Overfitting

A straight line always bends the same wrong way, whatever data you give it: high bias, low variance. A degree-14 polynomial passes through every training point, so it looks completely different on each new sample: low bias, high variance.

The Error Splits Three Ways

For regression with squared error, a model's expected error on new data splits into three parts:

The noise comes from the data itself and no model can remove it. You can only trade the other two against each other: making a model more flexible lowers bias but raises variance. This is the bias-variance trade-off.

Measuring It

We'll draw 200 different training sets from the same curve, fit a model to each, and look at the predictions at one point, $x=5$. The true value there is $10+14\times 5-5^2=55$.

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler

rng = np.random.default_rng(42)
true_f = lambda x: 10 + 14 * x - x ** 2

def predictions_at_5(degree, n_sets=200):
    preds = []
    for _ in range(n_sets):
        x = rng.uniform(0, 10, 20).reshape(-1, 1)
        y = true_f(x.ravel()) + rng.normal(0, 6, 20)
        model = make_pipeline(PolynomialFeatures(degree, include_bias=False), StandardScaler(), LinearRegression())
        preds.append(model.fit(x, y).predict([[5]])[0])
    return np.array(preds)

for degree in [1, 2, 12]:
    p = predictions_at_5(degree)
    bias_sq = (p.mean() - 55) ** 2
    print(f"degree {degree:>2}:  average prediction {p.mean():6.1f}   bias^2 {bias_sq:7.1f}   variance {p.var():7.1f}")

degree  1:  average prediction   46.8   bias^2    66.7   variance     5.0
degree  2:  average prediction   55.1   bias^2     0.0   variance     4.4
degree 12:  average prediction   54.6   bias^2     0.1   variance    28.7

• Degree 1: the average prediction misses 55 by a lot, so bias is high. Predictions don't vary much between training sets.
• Degree 2: the average lands almost exactly on 55 and the spread is small. The sweet spot.
• Degree 12: on average it's close to right, so bias is low, but individual predictions swing wildly from one training set to the next, so variance is high.

Summary

• Bias: error from wrong assumptions (too simple). Variance: error from sensitivity to the training data (too flexible).
• Expected error $={\text{Bias}}^2+\text{Variance}+\text{Noise}$.
• More flexibility lowers bias and raises variance: choose the balance with cross-validation.
• Next: Ridge regression, which deliberately adds a little bias to cut variance.