Start learning

Ridge Regression

Lesson 15. About 6 minutes.

Ridge regression is linear regression with a penalty on large coefficients. It keeps the model from leaning too hard on any one feature, which reduces overfitting.

Your Path Through This Lesson

0 of 5 done

By the End of This Lesson You Can
  • Write the Ridge cost function and explain what the penalty does.
  • See how the strength setting alpha shrinks coefficients.
  • Use Ridge to tame features that repeat each other.

The Idea: A Tax on Big Coefficients

Ordinary least squares picks coefficients that make the training error as small as possible, whatever the cost. If two features are almost copies of each other, or the data is noisy, it happily picks huge coefficients that cancel out, and those fall apart on new data.

Ridge adds a penalty: the sum of the squared coefficients, multiplied by a strength (called alpha in scikit-learn).

Now the model has to balance two goals. A big coefficient is only worth it if it lowers the error by more than it costs in penalty. This is called L2 regularisation, because it uses the squared size of the coefficients.

  • : plain linear regression.
  • Small : a light nudge towards smaller coefficients.
  • Large : coefficients squeezed towards zero (but never exactly zero).
Note

The intercept is not penalised. Only the slopes are.

Ridge on Features That Repeat Each Other

Let's give the model two nearly identical features: hours studied, and the same hours with a tiny bit of noise.

python
import numpy as np
from sklearn.linear_model import LinearRegression, Ridge

rng = np.random.default_rng(0)
hours = rng.uniform(1, 8, 30)
hours_copy = hours + rng.normal(0, 0.01, 30)        # almost the same feature
marks = 20 + 6 * hours + rng.normal(0, 4, 30)
X = np.column_stack([hours, hours_copy])

print("Linear regression:", LinearRegression().fit(X, marks).coef_.round(2))
print("Ridge, alpha=1    :", Ridge(alpha=1).fit(X, marks).coef_.round(2))
Output
Linear regression: [-53.93  60.04]
Ridge, alpha=1    : [2.97 3.12]

Linear regression splits the effect between the two copies in a wild, unstable way, a big positive and a big negative number. Ridge shares it sensibly: about half of the true slope of 6 to each copy. Predictions from both models are similar on this data, but Ridge's will hold up far better on new data.

Watching Alpha Shrink the Coefficients

python
from sklearn.datasets import load_diabetes
from sklearn.preprocessing import StandardScaler

X, y = load_diabetes(return_X_y=True)
X = StandardScaler().fit_transform(X)

for alpha in [0.01, 1, 10, 100, 1000]:
    coef = Ridge(alpha=alpha).fit(X, y).coef_
    print(f"alpha={alpha:>7}:  largest |coef| {np.abs(coef).max():6.1f}   sum of squares {np.sum(coef ** 2):9.0f}")
Output
alpha=   0.01:  largest |coef|   37.6   sum of squares      4281
alpha=      1:  largest |coef|   32.8   sum of squares      3309
alpha=     10:  largest |coef|   25.3   sum of squares      1812
alpha=    100:  largest |coef|   21.4   sum of squares      1208
alpha=   1000:  largest |coef|    9.9   sum of squares       301

As alpha grows, every coefficient gets pulled towards zero. Too little and you're back to plain linear regression; too much and the model can't fit anything. You choose alpha with cross-validation, which the GridSearchCV lesson automates.

Watch Out

Always scale features before Ridge. The penalty treats every coefficient the same, so a feature measured in large units (which needs a tiny coefficient) is barely penalised, while one in small units is penalised heavily. Put StandardScaler in front, ideally in a pipeline.

Ridge With Cross-Validation Built In

RidgeCV tries several alphas and keeps the best one:

python
from sklearn.linear_model import RidgeCV
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = RidgeCV(alphas=[0.1, 1, 10, 100]).fit(X_train, y_train)
print("chosen alpha:", model.alpha_)
print("test R2     :", round(model.score(X_test, y_test), 3))
Output
chosen alpha: 1.0
test R2     : 0.454
Try It Yourself
  1. In the repeated-feature example, try alpha=100. What happens to the two coefficients?
  2. Compare LinearRegression and Ridge(alpha=10) on the diabetes data with 5-fold cross_val_score. Is Ridge better, worse or the same?
The Key Idea

Ridge adds to the cost, shrinking all coefficients towards zero. It trades a little bias for a big cut in variance.

Summary

  • Ridge cost = squared error (L2 penalty).
  • Larger alpha means smaller coefficients: less variance, a little more bias.
  • It stabilises models with correlated features.
  • Scale features first, and choose alpha with cross-validation.
  • Next: Lasso, a penalty that can switch features off completely.