Start learning

Hyperparameter Tuning With GridSearchCV

Lesson 18. About 6 minutes.

GridSearchCV tries every combination of settings you give it, scores each one with cross-validation, and hands you the best model.

Your Path Through This Lesson

0 of 5 done

By the End of This Lesson You Can
  • Tell parameters from hyperparameters.
  • Tune a model with GridSearchCV and read its results.
  • Know when to switch to RandomizedSearchCV.

Parameters vs Hyperparameters

Parameters Hyperparameters
Who sets them The model, during fit You, before fit
Examples Coefficients in linear regression, split points in a tree alpha in Ridge, max_depth in a tree, n_neighbors in KNN
How they're found From the training data By trying values and comparing scores

Hyperparameters are the knobs on the outside of the model. Choosing them well often matters as much as choosing the algorithm.

The Grid Search Idea

  1. List the values you want to try for each hyperparameter: the grid.
  2. For every combination, run cross-validation and record the average score.
  3. Pick the combination with the best score.
  4. Retrain on all the training data with those settings.

With 4 values of alpha and 5 folds, that's fits. With two hyperparameters of 4 and 3 values, fits. Grids grow fast.

Tuning Ridge

python
from sklearn.datasets import load_diabetes
from sklearn.linear_model import Ridge
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

pipe = Pipeline([("scale", StandardScaler()), ("ridge", Ridge())])
grid = {"ridge__alpha": [0.01, 0.1, 1, 10, 100]}

search = GridSearchCV(pipe, grid, cv=5, scoring="r2")
search.fit(X_train, y_train)

print("best settings :", search.best_params_)
print("best CV score :", round(search.best_score_, 3))
print("test score    :", round(search.score(X_test, y_test), 3))
Output
best settings : {'ridge__alpha': 10}
best CV score : 0.454
test score    : 0.457

Notice the name ridge__alpha: the step's name in the pipeline, two underscores, then the hyperparameter. That's how you reach a setting inside a pipeline.

search itself is now the best model, already retrained on all of X_train (that's refit=True, the default), so you can call search.predict straight away.

Two Hyperparameters at Once

A decision tree has several knobs. Let's tune two of them on the breast-cancer data:

python
from sklearn.datasets import load_breast_cancer
from sklearn.tree import DecisionTreeClassifier

Xc, yc = load_breast_cancer(return_X_y=True)
Xc_train, Xc_test, yc_train, yc_test = train_test_split(Xc, yc, test_size=0.3, random_state=0)

grid = {"max_depth": [2, 3, 4, 6, None], "min_samples_leaf": [1, 5, 10]}
search = GridSearchCV(DecisionTreeClassifier(random_state=0), grid, cv=5, scoring="accuracy")
search.fit(Xc_train, yc_train)

print("combinations tried:", len(search.cv_results_["params"]))
print("best settings     :", search.best_params_)
print("best CV accuracy  :", round(search.best_score_, 3))
print("test accuracy     :", round(search.score(Xc_test, yc_test), 3))
Output
combinations tried: 15
best settings     : {'max_depth': 4, 'min_samples_leaf': 1}
best CV accuracy  : 0.927
test accuracy     : 0.947

When the Grid Gets Too Big

If you have five hyperparameters with five values each, that's 3,125 combinations times 5 folds. RandomizedSearchCV instead tries a fixed number of random combinations:

python
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint

dist = {"max_depth": randint(2, 12), "min_samples_leaf": randint(1, 20)}
rsearch = RandomizedSearchCV(DecisionTreeClassifier(random_state=0), dist, n_iter=20, cv=5, random_state=0)
rsearch.fit(Xc_train, yc_train)
print("best settings:", rsearch.best_params_, " CV accuracy:", round(rsearch.best_score_, 3))
Output
best settings: {'max_depth': 5, 'min_samples_leaf': 4}  CV accuracy: 0.917

Here it searched a much wider range (depths 2–11, leaves 1–19, 190 combinations) with only 20 tries, and landed close to the grid's best (0.917 vs 0.927). On a small grid like ours a full search is fine; random search pays off when the full grid would have hundreds or thousands of combinations.

Watch Out

Never tune hyperparameters on the test set. GridSearchCV only looks at the training data (through cross-validation). The test set stays untouched until the very end, so its score is still an honest estimate.

Try It Yourself
  1. Print search.cv_results_["mean_test_score"] for the tree search. How different are the best and worst combinations?
  2. Add "criterion": ["gini", "entropy"] to the tree grid. How many combinations are tried now?
The Key Idea

GridSearchCV = try every combination × cross-validate × keep the best. Use RandomizedSearchCV when the grid is large.

Summary

  • Hyperparameters are set by you; parameters are learned during fit.
  • GridSearchCV cross-validates every combination and refits the best one.
  • Read results from best_params_, best_score_ and cv_results_.
  • Use RandomizedSearchCV for big search spaces, and keep the test set out of tuning.
  • Next: the assumptions behind linear regression, and how to check them.