Multiple Linear Regression

Multiple linear regression predicts a number from several features at once, by giving each feature its own slope and adding them up.

By the End of This Lesson You Can

Write the multiple linear regression equation and read its coefficients.
Fit it in scikit-learn on a real dataset.
Spot two classic traps: unscaled features and features that repeat each other.

One Feature Is Rarely Enough

Hours studied explained 60% of the variation in our students' marks. The other 40% comes from other things: sleep, attendance, how hard the paper was. Multiple linear regression lets the model use all of them.

With two features, hours $x_{1}$ and hours of sleep $x_{2}$ :

y^= θ_{0} + θ_{1} x_{1} + θ_{2} x_{2}

With $n$ features:

y^= θ_{0} + θ_{1} x_{1} + θ_{2} x_{2} + \dots + θ_{n} x_{n}

Instead of a line, the model is now a flat plane (two features) or a "hyperplane" (more). The idea, the cost function and gradient descent are exactly the same as in simple linear regression.

Reading the Coefficients

Each $θ_{j}$ answers one question: if this feature goes up by 1 and every other feature stays the same, how much does the prediction change?

Let's make a tiny dataset where we know the true rule, marks $= 10 + 6 \times hours + 2 \times sleep$ , and check that the model finds it.

Student	Hours	Sleep	Marks
Noman	1	8	32
Lamia	2	5	32
Rafi	3	7	42
Tanvir	4	6	46
Arif	5	8	56
Mitu	6	7	60

python

import numpy as np
from sklearn.linear_model import LinearRegression

X = np.array([[1, 8], [2, 5], [3, 7], [4, 6], [5, 8], [6, 7]])   # hours, sleep
y = np.array([32, 32, 42, 46, 56, 60])

model = LinearRegression().fit(X, y)
print("intercept:", round(model.intercept_, 2))
print("slopes   :", model.coef_.round(2))
print("6 hours study, 8 hours sleep ->", model.predict([[6, 8]]).round(1))

Output

intercept: 10.0
slopes   : [6. 2.]
6 hours study, 8 hours sleep -> [62.]

It found 10, 6 and 2. Read them as: each extra hour of study adds 6 marks while sleep stays the same, and each extra hour of sleep adds 2 marks while study stays the same.

The Same Thing as a Matrix

With many features, it's neater to write everything as matrices. Put a column of 1s in front of the features (for the intercept) and the whole dataset becomes one equation:

y^= X θ

Least squares then has a closed-form answer, called the normal equation:

θ = (X^{⊤} X)^{- 1} X^{⊤} y

python

X1 = np.column_stack([np.ones(len(X)), X])      # add the column of 1s
theta = np.linalg.inv(X1.T @ X1) @ X1.T @ y
print(theta.round(2))

Output

[10.  6.  2.]

The same three numbers. scikit-learn uses a more numerically stable method than inverting a matrix, but it solves this same equation.

A Real Dataset

scikit-learn ships with a medical dataset: 442 diabetes patients, 10 measurements each (age, body mass index, blood pressure and six blood tests), and a number measuring how far the disease progressed one year later.

python

from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score, root_mean_squared_error

data = load_diabetes()
X_train, X_test, y_train, y_test = train_test_split(data.data, data.target, test_size=0.2, random_state=42)

model = LinearRegression().fit(X_train, y_train)
pred = model.predict(X_test)
print("test R2  :", round(r2_score(y_test, pred), 3))
print("test RMSE:", round(root_mean_squared_error(y_test, pred), 1))

for name, coef in sorted(zip(data.feature_names, model.coef_), key=lambda p: -abs(p[1]))[:3]:
    print(f"{name:>4}: {coef:8.1f}")

Output

test R2  : 0.453
test RMSE: 53.9
  s1:   -931.5
  s5:    736.2
 bmi:    542.4

The model explains roughly half of the variation on patients it never saw. The three printed features have the biggest coefficients.

Watch Out

Only compare coefficient sizes when the features are on the same scale. The diabetes features come pre-scaled, so comparing them is fair. In raw data, a feature measured in taka will get a tiny coefficient and one measured in crore a huge one, even if they matter equally. Scale first (you'll see StandardScaler in later lessons) before reading importance from coefficients.

When Features Repeat Each Other

If two features carry nearly the same information, say "hours studied" and "minutes studied", the model can't tell which one deserves the credit. The coefficients become unstable: tiny changes in the data swing them wildly, even though predictions stay fine. This is called multicollinearity.

Signs to look for: coefficients with surprising signs or sizes, and coefficients that change a lot when you add or remove one row. The fix is to drop one of the repeated features, or use Ridge regression, which you'll meet soon.

Try It Yourself

In the six-student example, add a third column that is exactly 2 * hours. Fit again and print the coefficients. What happens?
On the diabetes data, fit a model using only the bmi column (data.data[:, [2]]). How does its test R² compare with all 10 features?

The Key Idea

Each coefficient is the change in the prediction when its feature rises by 1 and all the others are held still.

Summary

$y^= θ_{0} + θ_{1} x_{1} + \dots + θ_{n} x_{n}$ : one slope per feature.
Each slope is the effect of its feature with the others held constant.
The normal equation $θ = (X^{⊤} X)^{- 1} X^{⊤} y$ solves it in one step.
Compare coefficients only after scaling, and watch for features that repeat each other.
Next: what to do when the relationship is curved, with polynomial regression.

Your Path Through This Lesson

One Feature Is Rarely Enough

Reading the Coefficients

The Same Thing as a Matrix

A Real Dataset

When Features Repeat Each Other

Summary