One-vs-Rest Logistic Regression

One-vs-rest turns a problem with many classes into several yes/no problems: one classifier per class, each asking "is it this class or not?", and the most confident one wins.

By the End of This Lesson You Can

Explain how one-vs-rest handles more than two classes.
Compare it with multinomial (softmax) logistic regression.
Use both in scikit-learn.

The One-vs-Rest Idea

Take the iris dataset with three species: setosa, versicolor and virginica. One-vs-rest trains three ordinary binary classifiers:

Classifier	Question it answers
1	Setosa, or not setosa?
2	Versicolor, or not versicolor?
3	Virginica, or not virginica?

For a new flower, each classifier gives a probability. Suppose they say 0.10, 0.70 and 0.35. The prediction is versicolor, the class whose classifier is most confident.

With $K$ classes you train $K$ classifiers. Note the three probabilities don't add up to 1, because each classifier was trained on its own question.

The Alternative: Multinomial (Softmax)

Instead of $K$ separate models, multinomial logistic regression trains one model that outputs a score for every class and turns the scores into probabilities with the softmax function:

P (class k) = \frac{e ^{z_{k}}}{\sum _{j = 1}^{K} e ^{z_{j}}}

These probabilities always add up to 1. Since scikit-learn 1.5, LogisticRegression uses multinomial by default for multi-class problems.

	One-vs-rest	Multinomial (softmax)
Models trained	$K$ binary models	One joint model
Probabilities add to 1	Not automatically	Yes
Works with any binary classifier	Yes	Only some models
Usually more accurate	Sometimes	Often slightly

Both in scikit-learn

python

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.multiclass import OneVsRestClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, test_size=0.3, random_state=1, stratify=iris.target)

ovr = make_pipeline(StandardScaler(), OneVsRestClassifier(LogisticRegression()))
multi = make_pipeline(StandardScaler(), LogisticRegression())

for name, model in [("one-vs-rest", ovr), ("multinomial", multi)]:
    model.fit(X_train, y_train)
    print(f"{name:>12}: test accuracy {model.score(X_test, y_test):.3f}")

Output

 one-vs-rest: test accuracy 0.933
 multinomial: test accuracy 0.978

Looking Inside One-vs-Rest

python

inner = ovr[-1]
print("number of binary classifiers:", len(inner.estimators_))

flower = X_test[:1]
scaled = ovr[0].transform(flower)
for species, clf in zip(iris.target_names, inner.estimators_):
    print(f"P({species} vs rest) = {clf.predict_proba(scaled)[0, 1]:.3f}")
print("prediction:", iris.target_names[ovr.predict(flower)[0]])

Output

number of binary classifiers: 3
P(setosa vs rest) = 0.004
P(versicolor vs rest) = 0.125
P(virginica vs rest) = 0.873
prediction: virginica

Three separate classifiers, three separate probabilities, and the prediction is the highest one.

Tip

OneVsRestClassifier works with any binary classifier, not just logistic regression. It's the standard way to make a yes/no-only model handle many classes.

Note

There's also one-vs-one, which trains a classifier for every pair of classes, $K (K - 1) /2$ of them, and lets them vote. Support vector machines use it internally.

Try It Yourself

Print multi.predict_proba(X_test[:3]) and ovr.predict_proba(X_test[:3]). Do the rows add up to 1 for both? (scikit-learn rescales the one-vs-rest probabilities so they do.)
For the 10-digit dataset load_digits, how many classifiers would one-vs-rest train? How many would one-vs-one train?

The Key Idea

One-vs-rest trains one "this class or not" model per class and picks the most confident. Multinomial trains one model with softmax probabilities.

Summary

One-vs-rest: one binary classifier per class; the most confident wins.
Multinomial: one model with softmax probabilities that add to 1; scikit-learn's default.
OneVsRestClassifier wraps any binary classifier.
Next part: two simple and surprisingly strong classifiers, Naive Bayes and KNN.

Your Path Through This Lesson

More Than Two Answers

The One-vs-Rest Idea

The Alternative: Multinomial (Softmax)

Both in scikit-learn

Looking Inside One-vs-Rest

Summary