Naive Bayes

Naive Bayes is a fast classifier built on Bayes' theorem. It asks "given these features, which class is most likely?" and makes one bold, simplifying assumption: that the features don't affect each other.

By the End of This Lesson You Can

Read Bayes' theorem in plain words.
Explain the "naive" independence assumption and why it still works.
Choose between Gaussian, Multinomial and Bernoulli Naive Bayes.

Bayes' Theorem

Bayes' theorem tells you how to update a belief when you see evidence:

P (class ∣ features) = \frac{P ( features ∣ class ) \times P ( class )}{P ( features )}

Term	Name	Meaning
$P (class ∣ features)$	Posterior	What we want: the chance of the class after seeing the features
$P (features ∣ class)$	Likelihood	How typical these features are for that class
$P (class)$	Prior	How common the class is before looking at anything
$P (features)$	Evidence	The same for every class, so we can ignore it when comparing

An Everyday Example

About 20% of the emails you get are spam. The word "lottery" appears in 30% of spam emails but only 1% of real ones. An email contains "lottery". How likely is it to be spam?

Compare the two classes (we can drop $P (features)$ , since it's the same for both):

spam: 0.30 \times 0.20 = 0.060 not spam: 0.01 \times 0.80 = 0.008

Normalise so they add up to 1:

P (spam ∣ "lottery") = \frac{0.060}{0.060 + 0.008} \approx 0.88

One word moved our belief from 20% to 88%.

The "Naive" Part

Real emails have many words. Working out $P ("lottery", "winner", "claim" ∣ spam)$ directly would need examples of every combination of words, far more data than anyone has.

Naive Bayes assumes the features are independent given the class, so it can just multiply:

P (x_{1}, x_{2}, \dots, x_{n} ∣ class) \approx P (x_{1} ∣ class) \times P (x_{2} ∣ class) \times \dots \times P (x_{n} ∣ class)

That's rarely exactly true ("lottery" and "winner" tend to appear together), which is why it's called naive. But for picking the most likely class, it works surprisingly well, especially with text.

Three Flavours

Type	Feature type	Typical use
GaussianNB	Continuous numbers, assumed bell-shaped per class	Measurements like height, sensor readings
MultinomialNB	Counts	Word counts in documents, spam filtering
BernoulliNB	Yes/no features	Whether each word appears or not

Gaussian Naive Bayes on Flowers

python

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)

model = GaussianNB().fit(X_train, y_train)
print("test accuracy:", round(model.score(X_test, y_test), 3))
print("class priors :", model.class_prior_.round(3))

Output

test accuracy: 0.978
class priors : [0.333 0.333 0.333]

Training is almost instant: it only computes the mean and spread of each feature within each class.

Multinomial Naive Bayes on Text

python

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB

emails = [
    "win a free lottery prize now", "claim your free prize winner",
    "cheap loans win money now", "lottery winner claim cash",
    "meeting moved to monday", "please review the lecture notes",
    "lunch with the team tomorrow", "notes from monday meeting",
]
labels = [1, 1, 1, 1, 0, 0, 0, 0]        # 1 = spam

vec = CountVectorizer()
X_text = vec.fit_transform(emails)
nb = MultinomialNB().fit(X_text, labels)

new = ["free lottery for the team", "monday lecture notes"]
for text, p in zip(new, nb.predict_proba(vec.transform(new))[:, 1]):
    print(f"{text!r:>28}: P(spam) = {p:.2f}")

Output

 'free lottery for the team': P(spam) = 0.58
      'monday lecture notes': P(spam) = 0.05

CountVectorizer turns each email into word counts; Naive Bayes learns how often each word appears in spam and in real email.

Watch Out

If a word never appeared in, say, real emails during training, its probability for that class would be 0, and multiplying by 0 wipes out every other clue. Laplace smoothing fixes this by adding a small count (1 by default, the alpha setting) to every word. You'll see it at work in the worked example.

Tip

Naive Bayes is a great first model for text: fast to train, works with little data, and hard to beat as a baseline for spam and topic classification.

Try It Yourself

In the spam example, change the base rate: what is $P (spam ∣ "lottery")$ if only 5% of emails are spam?
Add a new spam email to the training list and see how the probabilities for the two test emails change.

The Key Idea

Naive Bayes multiplies the prior by each feature's likelihood, assuming the features are independent, and picks the class with the highest result.

Summary

Posterior ∝ likelihood × prior; pick the class with the largest posterior.
"Naive" = features assumed independent given the class, so likelihoods simply multiply.
Gaussian for numbers, Multinomial for counts, Bernoulli for yes/no features.
Fast and strong for text; use smoothing to avoid zero probabilities.
Next: a full Naive Bayes calculation by hand.

Your Path Through This Lesson

Bayes' Theorem

An Everyday Example

The "Naive" Part

Three Flavours

Gaussian Naive Bayes on Flowers

Multinomial Naive Bayes on Text

Summary