Cross-Validation

Cross-validation tests a model several times on different slices of the data and averages the results, giving a more reliable score than a single train and test split.

By the End of This Lesson You Can

Explain why one train/test split can give a lucky or unlucky score.
Run K-fold cross-validation by hand and with scikit-learn.
Choose between K-fold, stratified K-fold, leave-one-out and time-series splits.

One Split Can Be Lucky

With a train and test split, your score depends on which rows landed in the test set. Change random_state and the score moves:

python

from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

X, y = load_diabetes(return_X_y=True)
for seed in [0, 1, 2, 3, 4]:
    X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=seed)
    print(f"random_state={seed}: test R2 = {LinearRegression().fit(X_tr, y_tr).score(X_te, y_te):.3f}")

Output

random_state=0: test R2 = 0.332
random_state=1: test R2 = 0.438
random_state=2: test R2 = 0.440
random_state=3: test R2 = 0.416
random_state=4: test R2 = 0.461

Same model, same data, very different scores. If you compared two models using one split, the "winner" might just have had an easier test set.

K-Fold Cross-Validation

The fix: test on every row once.

Split the data into $K$ equal parts, called folds (5 is the usual choice).
Train on $K - 1$ folds and test on the one left out.
Repeat $K$ times, so each fold is the test set once.
Average the $K$ scores.

Here's what that looks like with 10 rows and 5 folds. Each row is a round; T marks the test rows, and the rest are training rows.

Round	Rows 1–2	Rows 3–4	Rows 5–6	Rows 7–8	Rows 9–10
1	T	train	train	train	train
2	train	T	train	train	train
3	train	train	T	train	train
4	train	train	train	T	train
5	train	train	train	train	T

Every row is tested exactly once and used for training four times.

Watch the folds rotate:

In scikit-learn

cross_val_score does the splitting, training and scoring in one line:

python

from sklearn.model_selection import cross_val_score, KFold

cv = KFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(LinearRegression(), X, y, cv=cv, scoring="r2")
print("fold scores:", scores.round(3))
print(f"mean R2: {scores.mean():.3f}  (std {scores.std():.3f})")

Output

fold scores: [0.453 0.573 0.391 0.584 0.391]
mean R2: 0.478  (std 0.085)

Report the mean as the model's score, and the standard deviation as how much it wobbles from fold to fold. A model with a high mean and a low spread is reliable.

Choosing the Right Kind of Split

Method	How it splits	Use when
K-fold	$K$ equal folds	Regression, and most problems
Stratified K-fold	Each fold keeps the class proportions	Classification, especially with rare classes
Leave-one-out (LOOCV)	$K = n$ : every row is its own test fold	Very small datasets (slow on large ones)
Time-series split	Always trains on the past, tests on the future	Data ordered in time

For classifiers, scikit-learn uses stratified K-fold automatically when you pass cv=5:

python

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier

X_iris, y_iris = load_iris(return_X_y=True)
scores = cross_val_score(DecisionTreeClassifier(random_state=0), X_iris, y_iris, cv=5)
print("accuracy per fold:", scores.round(3), " mean:", round(scores.mean(), 3))

Output

accuracy per fold: [0.967 0.967 0.9   0.967 1.   ]  mean: 0.96

Watch Out

Never use ordinary K-fold on time-ordered data such as daily sales or stock prices. Shuffling lets the model train on next month and test on last month, which can't happen in real life and makes the score look far too good. Use TimeSeriesSplit instead.

Note

Cross-validation is for choosing between models and settings. Once you've chosen, train the final model on all the training data and, if you kept one aside, check it once on the untouched test set.

Try It Yourself

Change n_splits from 5 to 10. Does the mean change much? What about the spread?
Compare LinearRegression() with DecisionTreeRegressor(random_state=0) on the diabetes data using 5-fold cross-validation. Which one would you pick?

The Key Idea

Cross-validation tests on every row once and averages the scores, so your result doesn't depend on one lucky split.

Summary

One train/test split can give a lucky or unlucky score.
K-fold: split into $K$ folds, test on each once, average the scores.
Use stratified folds for classification and time-series splits for data in time order.
Report the mean and the spread of the fold scores.
Next: what it means when a model does well on training data but badly on new data, overfitting.

Your Path Through This Lesson

One Split Can Be Lucky

K-Fold Cross-Validation

In scikit-learn

Choosing the Right Kind of Split

Summary