Cost Function

Why We Need One Number

In the last lesson we said the best line has the smallest residuals "overall". But a computer can't look at a picture and judge which line looks best. It needs a single number to compare: line A scores 46, line B scores 24, so B is better.

That number is the cost, written $J$. Lower cost means a better line.

Building It Step by Step

Start with the errors, ${\hat{y}}_i-y_i$, for every example. Then:

1. Square each error, so negatives don't cancel positives and big misses count more.
2. Add them up across all $m$ examples.
3. Divide by $m$ to get an average, so the cost doesn't grow just because you have more data.
4. Divide by 2 as well. This is only a convenience: it cancels the 2 that appears when we take the derivative later.

The result is the mean squared error cost function:

where $h_{\theta }(x_i)={\theta }_0+{\theta }_1{x}_i$ is the prediction for example $i$, and $m$ is the number of examples.

Trying Different Lines by Hand

To make it easy to picture, let's keep the intercept fixed at ${\theta }_0=22$ and try different slopes ${\theta }_1$. Our data is still the five students (hours 1 to 5, marks 20, 40, 50, 40, 50), so $m=5$.

Slope ${\theta }_1=4$. Predictions $22+4x$ are 26, 30, 34, 38, 42.

Do the same for other slopes and you get this:

The cost falls, bottoms out at ${\theta }_1=6$, then rises again. The lowest cost is exactly the best-fit slope we found with least squares. That's no coincidence: least squares is "minimise this cost".

The Bowl

Plot $J$ against ${\theta }_1$ and you get a U-shaped curve, a bowl. Every point on the bowl is one possible line. The bottom of the bowl is the best one.

For our data the bowl has a neat formula (with ${\theta }_0=22$):

Plug in ${\theta }_1=4$: $24+5.5\times 4=46$. Matches the table.

When both ${\theta }_0$ and ${\theta }_1$ can change, the picture becomes a 3D bowl: a smooth valley with one lowest point. For linear regression the bowl is always this nice shape, with one single minimum and no false valleys. This shape is called convex, and it's why linear regression is so reliable to train.

Computing the Cost in Code

import numpy as np

x = np.array([1, 2, 3, 4, 5])
y = np.array([20, 40, 50, 40, 50])
m = len(x)

def cost(theta0, theta1):
    y_hat = theta0 + theta1 * x
    return np.sum((y_hat - y) ** 2) / (2 * m)

for theta1 in [0, 4, 6, 8, 10]:
    print(f"theta1 = {theta1:>2}:  J = {cost(22, theta1):.1f}")

theta1 =  0:  J = 222.0
theta1 =  4:  J = 46.0
theta1 =  6:  J = 24.0
theta1 =  8:  J = 46.0
theta1 = 10:  J = 112.0

Loss vs Cost

You'll see both words, often used loosely:

Summary

• The cost function turns "how wrong" into one number $J$.
• For linear regression: $J=\frac{1}{2m}\sum (h_{\theta }(x_i)-y_i{)}^2$.
• Plotted against the parameters, it's a convex bowl with one minimum: the best-fit line.
• Next: we can't always solve for the minimum with a formula. Gradient descent walks down the bowl step by step.