Start learning

Decision Tree in scikit-learn

Lesson 33. About 6 minutes.

This lesson brings the decision tree part together: a full project that trains, tunes, explains and evaluates a tree in scikit-learn.

Your Path Through This Lesson

0 of 5 done

By the End of This Lesson You Can
  • Train and tune a DecisionTreeClassifier with cross-validation.
  • Read its rules and feature importances.
  • Evaluate it fairly and know its key settings.

The Settings That Matter

Setting What it controls Default
criterion Split quality: "gini", "entropy" or "log_loss"; for regression "squared_error" "gini"
max_depth Maximum number of levels None (no limit)
min_samples_split Minimum rows needed to split a node 2
min_samples_leaf Minimum rows in each leaf 1
max_features Features considered at each split all
ccp_alpha Cost-complexity pruning strength 0
class_weight Extra weight for rare classes None
Note

Older tutorials use criterion="mse" for regression trees. It's now called "squared_error".

Step 1: Data and a Baseline

We'll use the wine dataset that ships with scikit-learn: 178 wines, 13 chemical measurements each, and three grape cultivars to predict.

python
import numpy as np
from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

wine = load_wine()
X_train, X_test, y_train, y_test = train_test_split(wine.data, wine.target, test_size=0.3, random_state=1, stratify=wine.target)

base = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)
print("depth:", base.get_depth(), " leaves:", base.get_n_leaves())
print(f"train {base.score(X_train, y_train):.3f}   test {base.score(X_test, y_test):.3f}")
Output
depth: 3  leaves: 5
train 1.000   test 0.944

No scaling step: trees don't need it.

Step 2: Tune With Cross-Validation

python
from sklearn.model_selection import GridSearchCV

grid = {
    "max_depth": [2, 3, 4, 5, None],
    "min_samples_leaf": [1, 3, 5],
    "criterion": ["gini", "entropy"],
}
search = GridSearchCV(DecisionTreeClassifier(random_state=0), grid, cv=5)
search.fit(X_train, y_train)
best = search.best_estimator_

print("best settings:", search.best_params_)
print(f"CV accuracy {search.best_score_:.3f}   test accuracy {best.score(X_test, y_test):.3f}")
Output
best settings: {'criterion': 'entropy', 'max_depth': 4, 'min_samples_leaf': 1}
CV accuracy 0.943   test accuracy 0.926

The tuned tree scored slightly lower on the test set than the baseline (0.926 vs 0.944). With only 54 test wines, that's a single wine's difference. The cross-validation score is averaged over five different splits of 124 training wines, so it's the more trustworthy number for choosing settings. Small test sets are noisy; don't over-read one number.

Step 3: Read the Rules

One of the best things about a tree: you can print exactly how it decides.

python
from sklearn.tree import export_text
print(export_text(best, feature_names=list(wine.feature_names)))
Output
|--- proline <= 730.00
|   |--- color_intensity <= 4.02
|   |   |--- class: 1
|   |--- color_intensity >  4.02
|   |   |--- flavanoids <= 1.40
|   |   |   |--- class: 2
|   |   |--- flavanoids >  1.40
|   |   |   |--- hue <= 0.71
|   |   |   |   |--- class: 2
|   |   |   |--- hue >  0.71
|   |   |   |   |--- class: 1
|--- proline >  730.00
|   |--- total_phenols <= 2.12
|   |   |--- malic_acid <= 2.08
|   |   |   |--- class: 1
|   |   |--- malic_acid >  2.08
|   |   |   |--- class: 2
|   |--- total_phenols >  2.12
|   |   |--- magnesium <= 135.50
|   |   |   |--- class: 0
|   |   |--- magnesium >  135.50
|   |   |   |--- class: 1

Step 4: Feature Importance

Each feature's importance is the total impurity decrease it produced across all its splits, scaled so all importances add up to 1.

python
order = np.argsort(-best.feature_importances_)
for i in order[:5]:
    print(f"{wine.feature_names[i]:>30}: {best.feature_importances_[i]:.3f}")
Output
                       proline: 0.421
               color_intensity: 0.229
                 total_phenols: 0.179
                    flavanoids: 0.090
                     magnesium: 0.035

Features that never appear in a split get an importance of 0.

Step 5: Evaluate

python
from sklearn.metrics import confusion_matrix, classification_report

pred = best.predict(X_test)
print(confusion_matrix(y_test, pred))
print(classification_report(y_test, pred, target_names=wine.target_names, digits=3))
Output
[[17  1  0]
 [ 2 19  0]
 [ 0  1 14]]
              precision    recall  f1-score   support

     class_0      0.895     0.944     0.919        18
     class_1      0.905     0.905     0.905        21
     class_2      1.000     0.933     0.966        15

    accuracy                          0.926        54
   macro avg      0.933     0.928     0.930        54
weighted avg      0.928     0.926     0.926        54
Tip

To draw the tree, use from sklearn.tree import plot_tree with matplotlib: plot_tree(best, feature_names=wine.feature_names, filled=True). It's a great picture for explaining a model to someone who doesn't code.

Watch Out

Feature importance from a single tree can be unstable: a slightly different training set may pick different features. For more reliable importances, average over many trees with a random forest.

Try It Yourself
  1. Use DecisionTreeRegressor on load_diabetes with the same tuning steps. Which feature is most important there?
  2. Remove the most important wine feature from the data and retrain. How much does the test accuracy drop?
The Key Idea

Tune a tree's depth and leaf size with cross-validation, then use its printed rules and feature importances to explain it.

Summary

  • Key settings: criterion, max_depth, min_samples_leaf, ccp_alpha.
  • Tune with GridSearchCV; no scaling needed.
  • Explain with export_text, plot_tree and feature_importances_.
  • Next part: combining many trees, starting with bagging and boosting.