Polynomial Regression

Polynomial regression fits a curve instead of a straight line, by adding squared and higher powers of a feature and then running ordinary linear regression on them.

By the End of This Lesson You Can

Explain why a straight line fails on curved data.
Turn one feature into polynomial features and fit them with linear regression.
Choose a sensible degree, and recognise when a high degree overfits.

When a Line Isn't Enough

Think about marks and hours of study again. The first few hours help a lot. After eight or nine hours you're tired, and extra hours help less, or even hurt. The relationship bends. A straight line can't bend, so it will always miss the shape.

Polynomial regression fixes this with a simple trick: give the model $x^{2}$ (and maybe $x^{3}$ ) as extra features.

y^= θ_{0} + θ_{1} x + θ_{2} x^{2}

This is still linear regression. "Linear" means linear in the coefficients $θ$ : the model still multiplies each feature by a number and adds them up. We've just invented a new feature, $x^{2}$ .

A Curve We Know

Let's make data from a known curve, marks $= 10 + 14 x - x^{2}$ , which rises, flattens and then falls after 7 hours.

python

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures

x = np.arange(0, 11).reshape(-1, 1)          # 0 to 10 hours
y = 10 + 14 * x.ravel() - x.ravel() ** 2

line = LinearRegression().fit(x, y)
print("straight line R2:", round(line.score(x, y), 3))

poly = PolynomialFeatures(degree=2, include_bias=False)
x2 = poly.fit_transform(x)                   # columns: x, x^2
print(x2[:4])

curve = LinearRegression().fit(x2, y)
print("curve R2:", round(curve.score(x2, y), 3))
print("intercept:", round(curve.intercept_, 2), " coefficients:", curve.coef_.round(2))

Output

straight line R2: 0.672
[[0. 0.]
 [1. 1.]
 [2. 4.]
 [3. 9.]]
curve R2: 1.0
intercept: 10.0  coefficients: [14. -1.]

The straight line catches the overall rise (R² 0.67) but misses the bend completely: it can't come back down after 7 hours. With $x^{2}$ added, the model fits perfectly and recovers the exact rule: 10, 14 and −1.

Doing It the Clean Way: A Pipeline

Creating the features by hand gets messy. A pipeline chains "make polynomial features" and "fit linear regression" into a single model:

python

from sklearn.pipeline import make_pipeline

model = make_pipeline(PolynomialFeatures(degree=2, include_bias=False), LinearRegression())
model.fit(x, y)
print(model.predict([[7], [12]]))

Output

[59. 34.]

At 7 hours the curve peaks at 59. At 12 hours, beyond the data, it predicts 34: the curve keeps falling. That may or may not be true in real life, which is a reminder that curves extrapolate even worse than lines.

How High Should the Degree Go?

Higher degrees can bend more. That sounds good, but with noisy data a very flexible curve starts chasing the noise instead of the pattern. Here's the same curve with random noise, split into training and test points:

python

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(0)
x = rng.uniform(0, 10, 30).reshape(-1, 1)
y = 10 + 14 * x.ravel() - x.ravel() ** 2 + rng.normal(0, 6, 30)
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.5, random_state=0)

for degree in [1, 2, 3, 9, 14]:
    # scaling keeps huge powers like x**14 numerically stable
    m = make_pipeline(PolynomialFeatures(degree, include_bias=False), StandardScaler(), LinearRegression())
    m.fit(x_train, y_train)
    print(f"degree {degree:>2}:  train R2 {m.score(x_train, y_train):.3f}   test R2 {m.score(x_test, y_test):.3f}")

Output

degree  1:  train R2 0.787   test R2 0.595
degree  2:  train R2 0.905   test R2 0.873
degree  3:  train R2 0.924   test R2 0.798
degree  9:  train R2 0.978   test R2 0.763
degree 14:  train R2 0.979   test R2 -15.171

Read the table from top to bottom:

Degree 1 is too stiff: the lowest training score, and it misses the bend on the test data. That's underfitting.
Degree 2 matches the true shape and has the best test score.
Degree 9 and 14 keep improving on the training data, but the test score drops, and at degree 14 it collapses below zero. That's overfitting, the subject of the next part.

Tip

Start with degree 2, then try 3. Pick the degree by its test (or cross-validation) score, never the training score, which always improves as the degree goes up.

Watch Out

With several features, PolynomialFeatures also creates every product between them ( $x_{1} x_{2}$ , $x_{1}^{2} x_{2}$ , …). The number of columns explodes: 10 features at degree 3 gives 285 columns. Keep the degree low when you have many features.

Try It Yourself

Change the noise level from 6 to 1 and rerun the degree table. Does degree 15 still fail on the test data?
Run PolynomialFeatures(2).fit_transform([[2, 3]]) and match each output column to a term like $x_{1}^{2}$ or $x_{1} x_{2}$ .

The Key Idea

Polynomial regression is linear regression on new features ( $x^{2}$ , $x^{3}$ , …). Choose the degree by test performance, not training performance.

Summary

Polynomial regression adds powers of a feature and fits ordinary linear regression.
Use PolynomialFeatures inside a pipeline.
Too low a degree underfits, too high overfits. Choose by test or cross-validation score.
Next: a fairer way to measure a model than one train/test split, cross-validation.

Your Path Through This Lesson

When a Line Isn't Enough

A Curve We Know

Doing It the Clean Way: A Pipeline

How High Should the Degree Go?

Summary