Start learning

Information Gain

Lesson 31. About 6 minutes.

Information gain measures how much a split reduces impurity. A decision tree tries every possible question and picks the one with the highest information gain.

Your Path Through This Lesson

0 of 5 done

By the End of This Lesson You Can
  • Compute the weighted impurity of a split.
  • Calculate information gain by hand for each feature.
  • Watch a tree choose its root question.

The Idea

Before a split, a node has some impurity. After the split, each child has its own impurity. If the children are purer, the split was useful. Information gain is the drop:

Each child's impurity is weighted by its size: a tiny pure child shouldn't count as much as a big mixed one.

Splitting the Cricket Data on Outlook

The full data: 9 Yes, 5 No, entropy 0.940 (from the last lesson).

Splitting on Outlook gives three children:

Outlook Yes No Days Entropy
Sunny 2 3 5 0.971
Overcast 4 0 4 0 (pure)
Rain 3 2 5 0.971

Weighted entropy of the children:

(Keep a few extra decimals in the middle steps; rounding too early would give 0.246.)

Comparing Every Feature

Doing the same for the other three features:

Feature Weighted child entropy Information gain
Outlook 0.694 0.247
Humidity 0.788 0.152
Wind 0.892 0.048
Temperature 0.911 0.029

Outlook wins, so it becomes the root question. The Overcast branch is already pure ("Yes" every time), so it becomes a leaf. The tree then repeats the whole process separately inside the Sunny and Rain branches.

Computing It in Python

python
import numpy as np
import pandas as pd

df = pd.DataFrame({
    "outlook": ["Sunny", "Sunny", "Overcast", "Rain", "Rain", "Rain", "Overcast", "Sunny", "Sunny", "Rain", "Sunny", "Overcast", "Overcast", "Rain"],
    "temp": ["Hot", "Hot", "Hot", "Mild", "Cool", "Cool", "Cool", "Mild", "Cool", "Mild", "Mild", "Mild", "Hot", "Mild"],
    "humidity": ["High", "High", "High", "High", "Normal", "Normal", "Normal", "High", "Normal", "Normal", "Normal", "High", "Normal", "High"],
    "wind": ["Weak", "Strong", "Weak", "Weak", "Weak", "Strong", "Strong", "Weak", "Weak", "Weak", "Strong", "Strong", "Weak", "Strong"],
    "play": ["No", "No", "Yes", "Yes", "Yes", "No", "Yes", "No", "Yes", "Yes", "Yes", "Yes", "Yes", "No"],
})

def entropy(labels):
    p = labels.value_counts(normalize=True).values
    return -np.sum(p * np.log2(p))

parent = entropy(df["play"])
print(f"parent entropy: {parent:.3f}")
for feature in ["outlook", "humidity", "wind", "temp"]:
    weighted = sum(len(g) / len(df) * entropy(g["play"]) for _, g in df.groupby(feature))
    print(f"{feature:>8}: information gain {parent - weighted:.3f}")
Output
parent entropy: 0.940
 outlook: information gain 0.247
humidity: information gain 0.152
    wind: information gain 0.048
    temp: information gain 0.029

With Gini Instead

The same idea works with Gini impurity: parent Gini minus the weighted Gini of the children. scikit-learn calls this the "impurity decrease". With criterion="gini" the tree maximises the Gini decrease; with criterion="entropy" it maximises information gain.

Watch Out

Information gain favours features with many distinct values. A feature like "day number" (1 to 14) would split the data into 14 pure single-day children and get the maximum possible gain, but it's useless for predicting a new day. Drop ID-like columns before training a tree.

Note

scikit-learn's trees always make binary splits ("Outlook = Sunny?" yes or no), rather than one branch per category as in our hand example. The principle is the same: try every candidate split and keep the one with the biggest impurity decrease.

Try It Yourself
  1. Compute by hand the information gain for Humidity: High has 3 Yes / 4 No, Normal has 6 Yes / 1 No.
  2. Inside the Sunny branch only (5 days), which feature has the highest information gain? Use df[df.outlook == "Sunny"].
The Key Idea

Information gain = parent impurity − size-weighted child impurity. The tree splits on the feature with the biggest gain.

Summary

  • .
  • For the cricket data, Outlook gives the highest gain (0.247), so it's the root.
  • The tree repeats the search inside each branch.
  • Beware ID-like features with many values; they fake high gain.
  • Next: trees for numbers, and how to stop them overfitting with pruning.