Start learning

Decision Tree Classifier

Lesson 29. About 6 minutes, with a video walkthrough.

A decision tree classifies data by asking a series of yes/no questions about the features, like a flowchart, until it reaches an answer.

Your Path Through This Lesson

0 of 5 done

By the End of This Lesson You Can
  • Read a decision tree and follow it to a prediction.
  • Name the parts of a tree: root, internal nodes, leaves, depth.
  • Fit a tree in scikit-learn and print its rules.

You Already Think in Trees

A doctor deciding whether you need an X-ray might think: Did you fall? If yes, does it hurt to move? If yes, X-ray. Each question narrows things down until there's a clear answer. A decision tree learns those questions from data.

A Small Example

Should a student be predicted to pass? A tree learned from student data might look like this:

text
Hours studied ≤ 4.5?
├── Yes → Attendance ≤ 5?
│         ├── Yes → FAIL
│         └── No  → PASS
└── No  → PASS

To classify a student who studied 3 hours with attendance 7: hours ≤ 4.5 is yes, so go left; attendance ≤ 5 is no, so go right: pass.

Watch a tree being built and used:

The Parts of a Tree

Part What it is In the example
Root node The first question, at the top Hours ≤ 4.5?
Internal node A question further down Attendance ≤ 5?
Branch The answer to a question (yes/no) The arrows
Leaf A final answer, with no more questions PASS or FAIL
Depth The number of questions on the longest path 2

How Does the Tree Pick Its Questions?

At each node, the tree tries every feature and every possible threshold, and picks the question that best separates the classes: after the split, each side should be as "pure" as possible, ideally all pass or all fail.

Purity is measured with Gini impurity or entropy, and the improvement a split brings is called information gain. Those are the next two lessons. For now, the idea is simple: ask the question that sorts the data best, then repeat on each side.

The tree stops splitting when a node is pure, when it reaches a maximum depth you set, or when a node has too few rows to split.

In scikit-learn

python
import numpy as np
from sklearn.tree import DecisionTreeClassifier, export_text

# hours studied, attendance (out of 10)
X = np.array([[1, 3], [2, 6], [3, 2], [3, 7], [4, 4], [5, 3], [6, 8], [7, 6], [8, 5], [2, 9]])
y = np.array([0, 0, 0, 1, 0, 1, 1, 1, 1, 1])      # 1 = pass

tree = DecisionTreeClassifier(random_state=0).fit(X, y)
print(export_text(tree, feature_names=["hours", "attendance"]))
print("depth:", tree.get_depth(), " leaves:", tree.get_n_leaves())
print("3 hours, attendance 7 ->", tree.predict([[3, 7]]))
Output
|--- hours <= 4.50
|   |--- attendance <= 6.50
|   |   |--- class: 0
|   |--- attendance >  6.50
|   |   |--- class: 1
|--- hours >  4.50
|   |--- class: 1

depth: 2  leaves: 3
3 hours, attendance 7 -> [1]

export_text prints the learned questions. Each class: line is a leaf.

Strengths and Weaknesses

Strengths Weaknesses
Easy to read and explain Overfits easily if allowed to grow deep
No feature scaling needed Small changes in data can change the whole tree
Handles numbers and categories Boundaries are always straight and axis-aligned (step-shaped)
Captures non-linear patterns and interactions One tree alone is rarely the most accurate model

The last weakness is why random forests and boosting combine many trees.

Tip

Trees don't care about feature scale, because they only ask "is this value above or below a threshold?". You can skip StandardScaler for tree models.

Try It Yourself
  1. Refit with max_depth=1. What single question does the tree choose, and how many training points does it get wrong?
  2. Follow the printed rules by hand for a student with 5 hours and attendance 9.
The Key Idea

A decision tree is a learned flowchart: at each node it asks the yes/no question that best separates the classes.

Summary

  • A decision tree is a flowchart of yes/no questions ending in leaves.
  • Parts: root, internal nodes, branches, leaves, depth.
  • Each split is chosen to make the groups as pure as possible.
  • Easy to explain and needs no scaling, but overfits if grown too deep.
  • Next: how purity is measured, with entropy and Gini impurity.