Start learning

Linear Regression Assumptions

Lesson 19. About 7 minutes.

Linear regression quietly assumes a few things about your data. When they hold, its coefficients and error estimates can be trusted; when they break, you need to know what to fix.

Your Path Through This Lesson

0 of 5 done

By the End of This Lesson You Can
  • Name the five classic assumptions of linear regression.
  • Check each one with a quick test or plot.
  • Know the usual fix when an assumption fails.

The Five Assumptions

# Assumption In plain words
1 Linearity The target really changes in a straight-line way with each feature
2 Independence One row's error tells you nothing about another row's error
3 Homoscedasticity The errors are about the same size everywhere, not bigger for big predictions
4 Normality of errors The errors roughly follow a bell curve
5 No multicollinearity Features don't repeat each other

Most of these are about the residuals, , the errors the model leaves behind. A good model's residuals should look like random noise: no pattern, same spread everywhere.

Note

For pure prediction, you can often get good results even when some assumptions are bent. They matter most when you want to interpret coefficients or trust confidence intervals and p-values.

Let's fit a model on the diabetes data and check each one.

python
import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression

data = load_diabetes()
X, y = data.data, data.target
model = LinearRegression().fit(X, y)
pred = model.predict(X)
resid = y - pred
print("mean residual:", round(resid.mean(), 6))
Output
mean residual: -0.0

The residuals average to zero. That's always true for least squares with an intercept, so it doesn't prove anything on its own. We need to look at their pattern.

1. Linearity

Check: plot residuals against predictions. Random scatter is good; a curve (a smile or a frown) means the relationship isn't linear.

Numeric check: see whether the residuals still correlate with the squared prediction, a sign of leftover curvature.

python
curve = np.corrcoef(resid, (pred - pred.mean()) ** 2)[0, 1]
print("correlation of residuals with squared predictions:", round(curve, 3))
Output
correlation of residuals with squared predictions: 0.106

A value near zero means little leftover curvature. Fix if broken: add polynomial features, transform a feature (log, square root), or use a non-linear model.

2. Independence

Check: this one is about how the data was collected. Rows measured over time (daily sales), or several rows from the same person, are often not independent: one day's error leaks into the next. The Durbin–Watson statistic tests this for ordered data; values near 2 mean no correlation between neighbouring errors.

python
dw = np.sum(np.diff(resid) ** 2) / np.sum(resid ** 2)
print("Durbin-Watson:", round(dw, 2))
Output
Durbin-Watson: 2.03

Fix if broken: use time-series models, or include the time or group as a feature.

3. Homoscedasticity (Equal Spread)

Check: in the residual plot, the cloud should be the same height from left to right. A funnel shape, small errors for small predictions and large errors for large ones, is called heteroscedasticity.

Compare the spread of residuals for the lower and upper half of predictions:

python
low = resid[pred < np.median(pred)]
high = resid[pred >= np.median(pred)]
print("residual std, low predictions :", round(low.std(), 1))
print("residual std, high predictions:", round(high.std(), 1))
Output
residual std, low predictions : 46.7
residual std, high predictions: 59.4

If the two numbers are very different, the spread isn't constant. Fix if broken: transform the target, such as , or use a model that doesn't assume equal spread.

4. Normality of Errors

Check: a histogram or Q–Q plot of residuals should look like a bell curve. The Shapiro–Wilk test gives a p-value; below 0.05 suggests the residuals aren't normal.

python
from scipy.stats import shapiro
stat, p = shapiro(resid)
print("Shapiro-Wilk p-value:", round(p, 3))
Output
Shapiro-Wilk p-value: 0.616

Fix if broken: usually it doesn't matter much for predictions, especially with lots of data. For inference, transform the target or remove clear data errors.

5. No Multicollinearity

Check: the variance inflation factor (VIF). For each feature, predict it from all the other features and compute . A VIF above about 5–10 means that feature is largely a copy of others.

python
def vif(X):
    out = []
    for j in range(X.shape[1]):
        others = np.delete(X, j, axis=1)
        r2 = LinearRegression().fit(others, X[:, j]).score(others, X[:, j])
        out.append(1 / (1 - r2))
    return np.array(out)

for name, v in sorted(zip(data.feature_names, vif(X)), key=lambda p: -p[1])[:4]:
    print(f"{name:>4}: VIF {v:6.1f}")
Output
  s1: VIF   59.2
  s2: VIF   39.2
  s3: VIF   15.4
  s5: VIF   10.1

The blood tests s1 and s2 (two cholesterol measures) have high VIFs: they largely repeat each other and s3/s5. Fix if broken: drop one of the related features, combine them, or use Ridge.

Quick Reference

Assumption Check Common fix
Linearity Residuals vs predictions: no curve Polynomial features, transforms
Independence Durbin–Watson ≈ 2; how data was collected Time-series methods
Equal spread Residual cloud same height Log-transform the target
Normal errors Histogram, Shapiro–Wilk Transform target; often ignorable
No multicollinearity VIF below ~5–10 Drop or combine features, Ridge
Try It Yourself
  1. Plot resid against pred with matplotlib (plt.scatter(pred, resid)). Do you see a pattern?
  2. Remove s1 from the features and recompute the VIFs. What happens to s2?
The Key Idea

Most assumptions are about residuals: they should look like patternless noise with the same spread everywhere.

Summary

  • Five assumptions: linearity, independence, equal spread, normal errors, no multicollinearity.
  • Check them mostly through the residuals, plus VIF for multicollinearity.
  • They matter most when you interpret coefficients; predictions are more forgiving.
  • Next part: classification, starting with logistic regression.