Logistic Regression

A Yes/no Question

Let's change our question about students from "how many marks?" to "will they pass?" Eight students, their hours of study, and whether they passed (1) or failed (0):

The answer is a class, not a number. This is classification.

Why Not Just Use a Line?

You could fit a straight line to the 0s and 1s and say "predict pass if the line is above 0.5". Two problems:

1. A line keeps going. For 15 hours it predicts something like 1.8, and for 0 hours something negative. Those aren't probabilities.
2. One extreme student, say someone who studied 40 hours, drags the whole line flatter and moves the 0.5 point, even though that student changes nothing about where the boundary really is.

We need something that always stays between 0 and 1.

Watch the sigmoid fix the line:

The Sigmoid Function

The sigmoid (or logistic) function squeezes any number into the range 0 to 1:

Large positive $z$ gives a value near 1, large negative near 0, and $z=0$ gives exactly 0.5.

Logistic regression computes a straight line first, then passes it through the sigmoid:

$p$ is the predicted probability of class 1 (pass).

A Worked Example

Suppose training found ${\theta }_0=-4.5$ and ${\theta }_1=1$. For a student who studied 6 hours:

An 82% chance of passing. For 3 hours, $z=-1.5$ and $p\approx 0.18$.

The Decision Boundary

To turn a probability into a class, use a threshold, usually 0.5: predict pass if $p\ge 0.5$.

Since $\sigma (z)=0.5$ exactly when $z=0$, the boundary is where ${\theta }_0+{\theta }_{1}x=0$:

Study more than 4.5 hours and the model predicts a pass.

Fitting It in scikit-learn

import numpy as np
from sklearn.linear_model import LogisticRegression

hours = np.array([1, 2, 3, 4, 5, 6, 7, 8]).reshape(-1, 1)
passed = np.array([0, 0, 0, 1, 0, 1, 1, 1])

model = LogisticRegression().fit(hours, passed)
t0, t1 = model.intercept_[0], model.coef_[0, 0]
print(f"theta0 = {t0:.2f}, theta1 = {t1:.2f}")
print(f"decision boundary at {-t0 / t1:.2f} hours")

for h in [2, 4, 4.5, 6]:
    p = model.predict_proba([[h]])[0, 1]
    print(f"{h} hours -> P(pass) = {p:.2f}, predicted class {int(p >= 0.5)}")

theta0 = -3.91, theta1 = 0.87
decision boundary at 4.50 hours
2 hours -> P(pass) = 0.10, predicted class 0
4 hours -> P(pass) = 0.39, predicted class 0
4.5 hours -> P(pass) = 0.50, predicted class 0
6 hours -> P(pass) = 0.79, predicted class 1

predict_proba returns two columns, P(class 0) and P(class 1); we take column 1. predict returns the class using the 0.5 threshold.

Reading the Coefficient: Odds

The sigmoid can be rearranged into something easier to interpret:

The left side is the log-odds. So ${\theta }_1$ is how much the log-odds rise per extra hour, and $e^{{\theta }_1}$ is how much the odds are multiplied by. With ${\theta }_1=1$, each extra hour multiplies the odds of passing by $e^1\approx 2.7$.

Summary

• Logistic regression predicts $p=\sigma ({\theta }_0+{\theta }_{1}x)$, a probability between 0 and 1.
• The sigmoid squeezes any number into (0, 1); $\sigma (0)=0.5$.
• With a 0.5 threshold, the decision boundary is where ${\theta }_0+{\theta }_{1}x=0$.
• $e^{{\theta }_1}$ is how much the odds multiply per unit increase in $x$.
• Next: the cost function logistic regression minimises, log loss.