Start learning

Log Loss

Lesson 21. About 6 minutes.

Log loss is the cost function for logistic regression. It punishes predictions that are confident and wrong much more than ones that are unsure.

Your Path Through This Lesson

0 of 5 done

By the End of This Lesson You Can
  • Compute log loss for single predictions by hand.
  • Explain why confident mistakes cost so much.
  • Know why logistic regression doesn't use mean squared error.

Scoring a Probability

Logistic regression outputs a probability, like "82% chance of passing". How do we score that? If the student passed, 0.82 was good, 0.99 better, 0.51 barely right and 0.10 badly wrong. We want a cost that grows as the probability we gave to the true class shrinks.

The Formula

For one example with true label (0 or 1) and predicted probability of class 1:

Only one half is ever "switched on":

  • If : loss .
  • If : loss .

In both cases it's minus the log of the probability you gave to what actually happened. Averaging over examples gives the cost:

This is also called binary cross-entropy.

By Hand

A student actually passed (). Here's the loss for different predictions (natural log):

Predicted (pass) Loss Verdict
0.99 0.010 Confident and right: almost no cost
0.90 0.105 Right
0.60 0.511 Right, but unsure
0.50 0.693 Coin flip
0.10 2.303 Confident and wrong: big cost
0.01 4.605 Very confident and wrong: huge cost

The loss doesn't grow evenly. Going from 0.5 to 0.1 costs more than everything between 0.99 and 0.5. Being confidently wrong is heavily punished. As approaches 0, the loss heads towards infinity.

Why Not Mean Squared Error?

We could square the difference between and , as in linear regression. Two reasons we don't:

  1. Weak penalty for confident mistakes. With MSE, predicting 0.01 when the answer is 1 costs at most . Log loss charges 4.6, which pushes the model much harder to fix it.
  2. A bumpy bowl. Once the sigmoid is inside, MSE gives a cost surface with several dips, so gradient descent can get stuck. Log loss gives a smooth, convex bowl with a single bottom, like the cost function for linear regression.

In scikit-learn

python
import numpy as np
from sklearn.metrics import log_loss

y_true = [1, 1, 0, 0]
good   = [0.9, 0.8, 0.2, 0.1]     # confident and right
unsure = [0.6, 0.6, 0.4, 0.4]     # right, but hesitant
wrong  = [0.1, 0.9, 0.2, 0.1]     # one confident mistake

for name, p in [("good", good), ("unsure", unsure), ("wrong", wrong)]:
    print(f"{name:>6}: log loss = {log_loss(y_true, p):.3f}")
Output
  good: log loss = 0.164
unsure: log loss = 0.511
 wrong: log loss = 0.684

The "wrong" model gets three of four right, just like "good", but its single confident mistake makes its log loss the worst of the three.

python
# the same number by hand for "good"
p = np.array(good)
y = np.array(y_true)
print(round(-np.mean(y * np.log(p) + (1 - y) * np.log(1 - p)), 3))
Output
0.164
Note

Log loss is also a useful evaluation metric: it rewards models whose probabilities are honest. Two models with the same accuracy can have very different log loss if one of them is overconfident.

Try It Yourself
  1. Compute the loss by hand for a student who failed () when the model said .
  2. Change one prediction in wrong from 0.1 to 0.01. How much does the log loss rise?
The Key Idea

Log loss is minus the log of the probability given to the true outcome. Confident mistakes cost the most.

Summary

  • Log loss , averaged over examples.
  • It's minus the log of the probability given to what actually happened.
  • Confident mistakes cost far more than unsure ones.
  • It's convex for logistic regression, so gradient descent finds the single best answer.
  • Next: counting the kinds of right and wrong answers, the confusion matrix.