Start learning

Lasso Regression

Lesson 16. About 6 minutes.

Lasso regression penalises the absolute size of the coefficients. Unlike Ridge, it can push some coefficients to exactly zero, so it also picks which features to keep.

Your Path Through This Lesson

0 of 5 done

By the End of This Lesson You Can
  • Write the Lasso cost function and compare it with Ridge.
  • Explain why Lasso sets some coefficients to exactly zero.
  • Use Lasso to find the features that matter.

A Different Penalty

Lasso (Least Absolute Shrinkage and Selection Operator) uses the absolute value of each coefficient instead of its square:

This is L1 regularisation. It looks like a small change from Ridge, but it behaves very differently.

Ridge (L2) Lasso (L1)
Penalty
Coefficients Shrink, never exactly 0 Many become exactly 0
Feature selection No Yes
With correlated features Shares weight between them Tends to keep one, drop the others

Why Lasso Reaches Exactly Zero

Think about the "price" of a coefficient near zero.

  • With Ridge, the penalty is . Near zero it's almost flat: going from 0.1 to 0 saves only 0.01. So Ridge never bothers to push all the way to zero.
  • With Lasso, the penalty is . Going from 0.1 to 0 saves the full 0.1, the same rate as anywhere else. If a feature isn't earning its keep, the cheapest move is to drop it entirely.

So Lasso does two jobs at once: it fits the model and selects features.

Lasso on the Diabetes Data

python
import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.linear_model import Lasso
from sklearn.preprocessing import StandardScaler

data = load_diabetes()
X = StandardScaler().fit_transform(data.data)
y = data.target

for alpha in [0.1, 1, 5, 10, 30]:
    coef = Lasso(alpha=alpha).fit(X, y).coef_
    kept = [name for name, c in zip(data.feature_names, coef) if c != 0]
    print(f"alpha={alpha:>4}: {len(kept):>2} features kept  {kept}")
Output
alpha= 0.1:  9 features kept  ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's4', 's5', 's6']
alpha=   1:  7 features kept  ['sex', 'bmi', 'bp', 's1', 's3', 's5', 's6']
alpha=   5:  5 features kept  ['sex', 'bmi', 'bp', 's3', 's5']
alpha=  10:  4 features kept  ['bmi', 'bp', 's3', 's5']
alpha=  30:  2 features kept  ['bmi', 's5']

As alpha grows, Lasso switches off more and more features. The ones that survive the longest, like bmi and s5, are the strongest predictors.

Lasso With Cross-Validation

LassoCV picks alpha for you:

python
from sklearn.linear_model import LassoCV
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = LassoCV(cv=5, random_state=0).fit(X_train, y_train)
print("chosen alpha :", round(model.alpha_, 3))
print("features kept:", int(np.sum(model.coef_ != 0)), "of", X.shape[1])
print("test R2      :", round(model.score(X_test, y_test), 3))
Output
chosen alpha : 1.643
features kept: 7 of 10
test R2      : 0.471
Watch Out

When two features are strongly correlated, Lasso tends to keep one and drop the other almost at random. That doesn't mean the dropped one is useless, just that it added little once its twin was in the model. If you need to keep groups of correlated features together, use ElasticNet.

Tip

Like Ridge, Lasso needs scaled features. Use StandardScaler in a pipeline.

Try It Yourself
  1. Run Lasso(alpha=1) and Ridge(alpha=1) on the scaled diabetes data and print both coefficient lists side by side. How many zeros does each have?
  2. Add a column of random noise to X and refit LassoCV. Does Lasso give the noise column a coefficient of zero?
The Key Idea

Lasso's penalty drives weak features to exactly zero, so it regularises and selects features at the same time.

Summary

  • Lasso cost = squared error (L1 penalty).
  • It sets weak coefficients to exactly zero, so it also does feature selection.
  • With correlated features it keeps one and drops the rest.
  • Scale features first and pick alpha with LassoCV.
  • Next: ElasticNet, which mixes Ridge and Lasso.