Precision, Recall and F-Beta

Back to Our 100 Patients

From the confusion matrix lesson: TP = 8, FP = 4, FN = 2, TN = 86.

Precision

Of everyone the model flagged as sick, how many really were?

Low precision means many false alarms.

Recall

Of everyone who really was sick, how many did the model find?

Low recall means many missed cases. Recall is also called sensitivity or the true positive rate.

Which One Matters More?

It depends on what a mistake costs.

F1: One Number for Both

When both matter, combine them with the harmonic mean:

Why the harmonic mean and not the ordinary average? It punishes imbalance. A model with precision 1.0 and recall 0.02 (it flags one patient and happens to be right) has an ordinary average of 0.51 but an F1 of only about 0.04. You can't game F1 by being good at just one of the two.

F-Beta: Choosing the Weight

F-beta lets you decide how much more recall matters than precision:

In scikit-learn

from sklearn.metrics import precision_score, recall_score, f1_score, fbeta_score

# 100 patients: 8 TP, 2 FN, 4 FP, 86 TN
y_true = [1] * 10 + [0] * 90
y_pred = [1] * 8 + [0] * 2 + [1] * 4 + [0] * 86

print("precision:", round(precision_score(y_true, y_pred), 3))
print("recall   :", round(recall_score(y_true, y_pred), 3))
print("F1       :", round(f1_score(y_true, y_pred), 3))
print("F2       :", round(fbeta_score(y_true, y_pred, beta=2), 3))
print("F0.5     :", round(fbeta_score(y_true, y_pred, beta=0.5), 3))

precision: 0.667
recall   : 0.8
F1       : 0.727
F2       : 0.769
F0.5     : 0.69

F2 sits closer to recall (0.80) and F0.5 closer to precision (0.67), just as the weights say.

The Threshold Trade-Off

A classifier outputs a probability, and we turn it into "yes" when it's above a threshold, 0.5 by default. Moving the threshold trades precision against recall:

• Lower threshold → more yeses → finds more real cases (recall ↑) but more false alarms (precision ↓).
• Higher threshold → fewer, surer yeses → precision ↑, recall ↓.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
y = 1 - y                                   # make "malignant" the positive class (1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)
proba = model.predict_proba(X_test)[:, 1]

for t in [0.2, 0.5, 0.8]:
    pred = (proba >= t).astype(int)
    print(f"threshold {t}: precision {precision_score(y_test, pred):.3f}  recall {recall_score(y_test, pred):.3f}")

threshold 0.2: precision 0.824  recall 0.953
threshold 0.5: precision 0.938  recall 0.938
threshold 0.8: precision 1.000  recall 0.891

For cancer screening you'd lean towards a lower threshold: accept a few more false alarms to miss fewer tumours.

Summary

• Precision $=TP/(TP+FP)$: trust in the yeses. Recall $=TP/(TP+FN)$: coverage of real cases.
• F1 is their harmonic mean; F-beta weights recall $\beta$ times as much.
• Lowering the threshold raises recall and lowers precision.
• Pick the metric by the cost of each mistake.
• Next: a complete logistic regression workflow in scikit-learn.