Start learning

Entropy and Gini Impurity

Lesson 30. About 6 minutes.

Entropy and Gini impurity are two ways of measuring how mixed a group is. A decision tree uses them to judge which question sorts the data best.

Your Path Through This Lesson

0 of 5 done

By the End of This Lesson You Can
  • Compute entropy and Gini impurity for a node by hand.
  • Know their ranges for two classes: pure, mixed, and fifty-fifty.
  • Choose between them in scikit-learn.

Pure and Mixed

Picture a box of 10 cricket balls.

  • 10 red, 0 white: perfectly pure. If you draw one, you know its colour.
  • 5 red, 5 white: as mixed as it gets. A coin flip.
  • 8 red, 2 white: mostly pure.

A good decision-tree split creates boxes that are as pure as possible. We need a number for "how mixed".

Entropy

where is the share of class in the node. It comes from information theory: it measures how surprised you'd be, on average, by a random draw.

For two classes with shares and :

Mix Entropy
10 / 0 (pure)
8 / 2
5 / 5

So for two classes, entropy runs from 0 (pure) to 1 (fifty-fifty).

Note

By convention , so a class that's absent adds nothing.

Gini Impurity

Gini is the chance of labelling a random item wrongly if you labelled it at random according to the node's mix.

Mix Gini
10 / 0 (pure)
8 / 2
5 / 5

For two classes, Gini runs from 0 (pure) to 0.5 (fifty-fifty).

A Worked Example: The Cricket Data

In the Naive Bayes example, the club played on 9 of 14 days and didn't on 5. That whole dataset, the tree's starting node, has:

Quite mixed. The tree's job is to find questions that bring these numbers down.

Checking in Python

python
import numpy as np

def entropy(counts):
    p = np.array(counts) / sum(counts)
    p = p[p > 0]                       # 0 * log(0) counts as 0
    return abs(np.sum(p * np.log2(p)))  # abs avoids printing "-0.000" for a pure node

def gini(counts):
    p = np.array(counts) / sum(counts)
    return 1 - np.sum(p ** 2)

for counts in [[10, 0], [8, 2], [5, 5], [9, 5]]:
    print(f"{str(counts):>8}: entropy {entropy(counts):.3f}   gini {gini(counts):.3f}")
Output
 [10, 0]: entropy 0.000   gini 0.000
  [8, 2]: entropy 0.722   gini 0.320
  [5, 5]: entropy 1.000   gini 0.500
  [9, 5]: entropy 0.940   gini 0.459

Entropy or Gini?

Entropy Gini
Range (2 classes) 0 to 1 0 to 0.5
Uses logarithms Yes, slightly slower No, slightly faster
scikit-learn setting criterion="entropy" criterion="gini" (default)
Results Almost always pick the same or very similar splits

In practice they agree on most splits. Gini is the default because it's a little faster.

python
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
for crit in ["gini", "entropy"]:
    score = cross_val_score(DecisionTreeClassifier(criterion=crit, max_depth=4, random_state=0), X, y, cv=5).mean()
    print(f"{crit:>7}: CV accuracy {score:.3f}")
Output
   gini: CV accuracy 0.916
entropy: CV accuracy 0.940
Try It Yourself
  1. Compute entropy and Gini by hand for a node with 3 classes in equal shares (say 4, 4, 4). Is entropy still at most 1?
  2. Plot entropy and Gini for from 0 to 1. Where do both peak?
The Key Idea

Entropy and Gini both measure how mixed a node is: 0 means pure; higher means more mixed.

Summary

  • Entropy : 0 (pure) to 1 (fifty-fifty, two classes).
  • Gini : 0 (pure) to 0.5 (fifty-fifty, two classes).
  • The cricket data starts at entropy 0.940 and Gini 0.459.
  • Both usually choose the same splits; Gini is scikit-learn's default.
  • Next: using these to score a split, with information gain.