Start learning

Confusion Matrix

Lesson 22. About 6 minutes.

A confusion matrix is a small table that counts a classifier's correct and incorrect predictions, split by type of mistake. It shows what accuracy alone hides.

Your Path Through This Lesson

0 of 5 done

By the End of This Lesson You Can
  • Fill in a confusion matrix: TP, FP, FN and TN.
  • Compute accuracy from it, and see why accuracy can mislead.
  • Build one in scikit-learn.

Four Possible Outcomes

For a yes/no question, say "does this patient have the disease?", every prediction lands in one of four boxes:

Predicted: yes Predicted: no
Actually yes True positive (TP) False negative (FN)
Actually no False positive (FP) True negative (TN)
  • True positive: sick, and the model said sick. Correct.
  • True negative: healthy, and the model said healthy. Correct.
  • False positive: healthy, but the model said sick. A false alarm, also called a type I error.
  • False negative: sick, but the model said healthy. A missed case, also called a type II error.
Tip

Read the names in two parts. The second word is what the model said (positive or negative). The first word says whether it was right (true) or wrong (false).

A Worked Example

A test screens 100 patients. 10 really have the disease. The model flags 12 people as sick, and 8 of them really are.

Predicted sick Predicted healthy Total
Actually sick TP = 8 FN = 2 10
Actually healthy FP = 4 TN = 86 90
Total 12 88 100

Accuracy is the share of all predictions that were right:

94% sounds great. But the matrix also tells us 2 sick patients were sent home. Whether that's acceptable depends on the disease, and accuracy alone would never show it.

The Accuracy Trap

Now imagine a lazy model that says "healthy" for everyone:

Predicted sick Predicted healthy
Actually sick TP = 0 FN = 10
Actually healthy FP = 0 TN = 90

Accuracy . It finds none of the sick patients, yet scores 90%. When one class is much rarer than the other (fraud, disease, spam), accuracy can look high for a useless model. That's why we need precision and recall.

In scikit-learn

python
from sklearn.metrics import confusion_matrix, accuracy_score

y_true = [1, 1, 1, 0, 0, 0, 0, 0, 1, 0]     # 1 = sick
y_pred = [1, 0, 1, 0, 1, 0, 0, 0, 1, 0]

cm = confusion_matrix(y_true, y_pred)
print(cm)
tn, fp, fn, tp = cm.ravel()
print(f"TP={tp} FP={fp} FN={fn} TN={tn}")
print("accuracy:", accuracy_score(y_true, y_pred))
Output
[[5 1]
 [1 3]]
TP=3 FP=1 FN=1 TN=5
accuracy: 0.8
Watch Out

scikit-learn orders the matrix by label: row and column 0 first, then 1. So its layout is [[TN, FP], [FN, TP]], with true negatives in the top left, the opposite of the table above. Use cm.ravel() as shown to unpack the four numbers without confusion.

A Real Classifier

python
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)          # 0 = malignant, 1 = benign
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)

model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)
print(confusion_matrix(y_test, model.predict(X_test)))
print("accuracy:", round(model.score(X_test, y_test), 3))
Output
[[ 61   3]
 [  4 103]]
accuracy: 0.959

The off-diagonal numbers are the mistakes. In a cancer test, the box that counts malignant tumours predicted as benign is the one doctors care about most.

More Than Two Classes

With three classes the matrix is 3 × 3: rows are the true class, columns the predicted class. The diagonal holds correct predictions; anything off the diagonal shows exactly which classes the model mixes up.

Try It Yourself
  1. In the 100-patient example, what are the four numbers if the model flags 20 people and catches all 10 sick ones?
  2. Pass normalize="true" to confusion_matrix on the breast-cancer predictions. What do the numbers mean now?
The Key Idea

The confusion matrix splits right and wrong answers into TP, FP, FN and TN. Accuracy hides which kind of mistake you're making.

Summary

  • Four outcomes: TP, TN, FP (false alarm) and FN (missed case).
  • Accuracy , but it can be high for a useless model when classes are imbalanced.
  • scikit-learn's layout is [[TN, FP], [FN, TP]].
  • Next: precision, recall and F-beta, which focus on the kinds of mistakes that matter.