Simple Linear Regression

The Idea: Draw the Best Line

Let's go back to our five students:

Plot them and you'll see the marks go up with hours, but the points don't sit on a perfect line. Real data never does. So the question becomes: which straight line passes as close as possible to all the points? That line is called the best-fit line, and finding it is all simple linear regression does.

Writing the Line Three Ways

You learned the equation of a line in school. Machine learning writes the same line with different letters, and you'll see all three versions in books, so let's line them up:

We'll use the machine learning version:

• $h_{\theta }(x)$ is the hypothesis: the model's prediction for input $x$. It's the same thing as $\hat{y}$.
• ${\theta }_0$ (theta zero) is the intercept: the prediction when $x=0$, where the line crosses the y-axis.
• ${\theta }_1$ (theta one) is the slope: how much the prediction goes up when $x$ goes up by 1.

Training the model means finding the best values of ${\theta }_0$ and ${\theta }_1$. That's all.

What "Best" Means: Residuals

For each student, the gap between the real mark and the line's prediction is called a residual (or error):

Some residuals are positive (the point is above the line) and some negative (below). The best-fit line is the one that makes the residuals as small as possible overall. Specifically, it minimises the sum of squared residuals. Squaring makes every gap positive, and it punishes big misses far more than small ones.

This method is called ordinary least squares (OLS). You'll see in the cost function lesson how this becomes a number we can minimise.

Solving It by Hand

For one feature, least squares has a neat closed-form answer:

Here $\bar{x}$ and $\bar{y}$ are the averages. Let's work it through.

Step 1: averages. $\bar{x}=(1+2+3+4+5)/5=3$ and $\bar{y}=(20+40+50+40+50)/5=40$.

Step 2: a table of deviations.

Step 3: slope and intercept.

So the best-fit line is:

What the Numbers Mean

• Slope ${\theta }_1=6$: every extra hour of study adds about 6 marks on average.
• Intercept ${\theta }_0=22$: a student who studies 0 hours is predicted to get 22.

Fitting It With scikit-learn

Now let's check our hand calculation in code.

The Data as Arrays

import numpy as np
from sklearn.linear_model import LinearRegression

hours = np.array([1, 2, 3, 4, 5])
marks = np.array([20, 40, 50, 40, 50])

Passing a 1D Array by Mistake

Almost everyone does this the first time:

model = LinearRegression()
model.fit(hours, marks)

ValueError: Expected 2D array, got 1D array instead:
array=[1 2 3 4 5].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

scikit-learn wants $X$ as a table: one row per example, one column per feature. hours is a flat list of 5 numbers, so we reshape it into 5 rows × 1 column. The -1 means "work out the number of rows for me".

X = hours.reshape(-1, 1)
print(X.shape)

(5, 1)

Training and Reading θ0 and θ1

model = LinearRegression()
model.fit(X, marks)

print("theta0 (intercept):", round(model.intercept_, 2))
print("theta1 (slope):    ", round(model.coef_[0], 2))

theta0 (intercept): 22.0
theta1 (slope):     6.0

The same 22 and 6 we got by hand. Note that coef_ is an array, with one slope per feature. We have one feature, so we take coef_[0].

Predicting From the Line

print(model.predict([[6]]))
print(model.predict(X))

[58.]
[28. 34. 40. 46. 52.]

Check the first one by hand: $22+6\times 6=58$. And here are the residuals for our five students:

The residuals add up to exactly 0. That always happens with a least-squares line that has an intercept: the line sits right in the "middle" of the points.

Plotting It

import matplotlib.pyplot as plt

plt.scatter(hours, marks, color="black", label="students")
plt.plot(hours, model.predict(X), color="red", label="ŷ = 22 + 6x")
plt.xlabel("Hours studied")
plt.ylabel("Marks")
plt.legend()
plt.show()

Simple vs Multiple Linear Regression

The idea, the training and the scikit-learn code are the same. Only the number of columns in $X$ changes.

Where It's Used

• A first, quick model for any number you want to predict. Always try the simple thing first.
• Understanding a relationship: "how many extra sales per extra 1,000 taka of advertising?"
• Trend lines in business, economics and science.

Summary

• Simple linear regression fits $\hat{y}={\theta }_0+{\theta }_{1}x$.
• The best line minimises the sum of squared residuals (ordinary least squares).
• By hand: ${\theta }_1=\sum (x-\bar{x})(y-\bar{y})/\sum (x-\bar{x}{)}^2$, then ${\theta }_0=\bar{y}-{\theta }_1\bar{x}$.
• In scikit-learn, $X$ must be 2D: use reshape(-1, 1).
• Next: the cost function, the single number that tells us how bad a line is.