Start learning

Naive Bayes Worked Example

Lesson 27. About 7 minutes.

In this lesson we classify one new day completely by hand with Naive Bayes, step by step, then check every number with scikit-learn.

Your Path Through This Lesson

0 of 5 done

By the End of This Lesson You Can
  • Build the frequency tables Naive Bayes learns.
  • Compute the posterior for each class by hand.
  • See why Laplace smoothing is needed, and confirm the answer in code.

The Data: Will We Play Cricket?

A school cricket club recorded the weather on 14 afternoons and whether they played.

Day Outlook Temperature Humidity Wind Play
1 Sunny Hot High Weak No
2 Sunny Hot High Strong No
3 Overcast Hot High Weak Yes
4 Rain Mild High Weak Yes
5 Rain Cool Normal Weak Yes
6 Rain Cool Normal Strong No
7 Overcast Cool Normal Strong Yes
8 Sunny Mild High Weak No
9 Sunny Cool Normal Weak Yes
10 Rain Mild Normal Weak Yes
11 Sunny Mild Normal Strong Yes
12 Overcast Mild High Strong Yes
13 Overcast Hot Normal Weak Yes
14 Rain Mild High Strong No

The question: today is Sunny, Cool, High humidity, Strong wind. Will they play?

Step 1: The Priors

Out of 14 days, they played on 9 and didn't on 5:

Step 2: The Likelihood Tables

Count how often each value appears within each class. We only need today's values, but here are the full tables:

Outlook Yes (of 9) No (of 5)
Sunny 2 3
Overcast 4 0
Rain 3 2
Temperature Yes No
Hot 2 2
Mild 4 2
Cool 3 1
Humidity Yes No
High 3 4
Normal 6 1
Wind Yes No
Weak 6 2
Strong 3 3

Step 3: Multiply for Each Class

For Yes:

For No:

Step 4: Normalise

These two numbers aren't probabilities yet, because we skipped dividing by . Dividing each by their sum fixes that:

Prediction: No, they won't play (about 80% sure). Sunny, high humidity and strong wind all point towards "No" in the history.

The Zero Problem

Now try a day that's Overcast, Cool, High, Strong. For No, , so the whole product for "No" becomes 0, no matter what the other three features say. One unseen value vetoes everything.

Laplace smoothing adds 1 to every count (and the number of possible values to each total). For Outlook, which has 3 possible values:

Small, but no longer zero. scikit-learn's alpha setting controls this, with alpha=1 meaning add-one smoothing.

Checking With scikit-learn

CategoricalNB handles features that are categories. We encode each category as a number, and use a tiny alpha first so the answer matches our hand calculation without smoothing:

python
import numpy as np
from sklearn.naive_bayes import CategoricalNB
from sklearn.preprocessing import OrdinalEncoder

days = [
    ["Sunny", "Hot", "High", "Weak"], ["Sunny", "Hot", "High", "Strong"], ["Overcast", "Hot", "High", "Weak"],
    ["Rain", "Mild", "High", "Weak"], ["Rain", "Cool", "Normal", "Weak"], ["Rain", "Cool", "Normal", "Strong"],
    ["Overcast", "Cool", "Normal", "Strong"], ["Sunny", "Mild", "High", "Weak"], ["Sunny", "Cool", "Normal", "Weak"],
    ["Rain", "Mild", "Normal", "Weak"], ["Sunny", "Mild", "Normal", "Strong"], ["Overcast", "Mild", "High", "Strong"],
    ["Overcast", "Hot", "Normal", "Weak"], ["Rain", "Mild", "High", "Strong"],
]
play = ["No", "No", "Yes", "Yes", "Yes", "No", "Yes", "No", "Yes", "Yes", "Yes", "Yes", "Yes", "No"]

enc = OrdinalEncoder()
X = enc.fit_transform(days)
today = enc.transform([["Sunny", "Cool", "High", "Strong"]])

exact = CategoricalNB(alpha=1e-10).fit(X, play)
print("classes        :", exact.classes_)
print("no smoothing   :", exact.predict_proba(today).round(3))

smooth = CategoricalNB(alpha=1).fit(X, play)
print("alpha=1        :", smooth.predict_proba(today).round(3))
print("prediction     :", smooth.predict(today))
Output
classes        : ['No' 'Yes']
no smoothing   : [[0.795 0.205]]
alpha=1        : [[0.72 0.28]]
prediction     : ['No']

Without smoothing, scikit-learn gives exactly our 0.795 for "No". With add-one smoothing the numbers shift a little, but the decision stays the same.

Try It Yourself
  1. Work out by hand the prediction for Overcast, Mild, Normal, Weak. Then check it with smooth.predict.
  2. Compute with add-one smoothing by hand. (Outlook has 3 values.)
The Key Idea

Naive Bayes by hand: prior × the likelihood of each feature value, for every class, then normalise. Smoothing stops one unseen value from zeroing out a class.

Summary

  • Priors: 9/14 for Yes, 5/14 for No.
  • For Sunny, Cool, High, Strong: Yes ∝ 0.00529, No ∝ 0.02057, so P(No) ≈ 0.795.
  • An unseen value makes a probability 0; Laplace smoothing (alpha=1) prevents that.
  • CategoricalNB reproduces the hand calculation.
  • Next: a classifier with no training at all, K nearest neighbours.