Start learning

ElasticNet Regression

Lesson 17. About 6 minutes.

ElasticNet combines the Ridge and Lasso penalties. It can drop useless features like Lasso while keeping groups of related features together like Ridge.

Your Path Through This Lesson

0 of 5 done

By the End of This Lesson You Can
  • Write the ElasticNet cost and explain its two settings.
  • Know when ElasticNet beats Ridge or Lasso alone.
  • Tune it with ElasticNetCV.

Two Penalties at Once

ElasticNet adds both penalties to the squared error:

In scikit-learn the two strengths are set with two numbers:

Setting Meaning
alpha Overall strength of the penalty
l1_ratio How the strength is split: 1 means pure Lasso, 0 pure Ridge, 0.5 half and half

So ElasticNet is a dial between Ridge and Lasso, and Ridge and Lasso are its two ends.

When to Use It

Situation Best choice
Most features matter a little Ridge
Only a few features matter, and they aren't related Lasso
Many features, some useless, with groups of related features ElasticNet
More features than rows (common in genetics, text) ElasticNet

The problem it solves: Lasso drops all but one feature in a correlated group, almost at random. The Ridge part of ElasticNet encourages related features to share the weight, so the whole group stays in.

Seeing the Difference

Let's build data where three features are nearly copies of each other and all of them carry the signal, plus five pure-noise features.

python
import numpy as np
from sklearn.linear_model import Lasso, ElasticNet

rng = np.random.default_rng(1)
base = rng.normal(0, 1, 100)
group = np.column_stack([base + rng.normal(0, 0.05, 100) for _ in range(3)])   # 3 related features
noise = rng.normal(0, 1, (100, 5))                                              # 5 useless features
X = np.column_stack([group, noise])
y = 3 * base + rng.normal(0, 0.5, 100)

print("Lasso     :", Lasso(alpha=0.1).fit(X, y).coef_.round(2))
print("ElasticNet:", ElasticNet(alpha=0.1, l1_ratio=0.3).fit(X, y).coef_.round(2))
Output
Lasso     : [ 1.49  0.72  0.61 -0.    0.    0.    0.   -0.  ]
ElasticNet: [ 0.95  0.93  0.93 -0.    0.04  0.    0.   -0.03]

The first three numbers are the related group, the last five are noise. Lasso favours one member of the group, giving it about twice the weight of the others; with a stronger penalty it would drop them altogether. ElasticNet spreads the weight almost evenly across all three, and both models push the noise features to zero or close to it.

Choosing Both Settings With Cross-Validation

python
from sklearn.datasets import load_diabetes
from sklearn.linear_model import ElasticNetCV
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = make_pipeline(StandardScaler(), ElasticNetCV(l1_ratio=[0.1, 0.5, 0.9, 1.0], cv=5, random_state=0))
model.fit(X_train, y_train)
enet = model[-1]
print("chosen alpha   :", round(enet.alpha_, 3))
print("chosen l1_ratio:", enet.l1_ratio_)
print("test R2        :", round(model.score(X_test, y_test), 3))
Output
chosen alpha   : 0.189
chosen l1_ratio: 0.5
test R2        : 0.461

ElasticNetCV searched every combination of l1_ratio and a range of alpha values, and kept the best one by cross-validation score.

Tip

If you're not sure whether to use Ridge or Lasso, use ElasticNetCV with l1_ratio values from 0.1 to 1.0 and let cross-validation decide. If it picks 1.0, Lasso was best; if it picks a small value, Ridge-like behaviour won.

Try It Yourself
  1. In the related-features example, change l1_ratio to 0.9 and then 0.1. How does the weight on the three related features change?
  2. Increase alpha to 1 in the same example. Which coefficients reach zero first?
The Key Idea

ElasticNet mixes L1 and L2 penalties: it drops useless features but keeps correlated groups together.

Summary

  • ElasticNet penalty = Lasso part + Ridge part.
  • alpha sets the strength, l1_ratio the mix (1 = Lasso, 0 = Ridge).
  • Best when you have many features, some useless, with correlated groups.
  • Tune both settings with ElasticNetCV.
  • Next: a general tool for tuning any model's settings, GridSearchCV.