Train and Test Split

Why You Can't Test on the Training Data

Imagine a teacher who gives students the exact exam questions, with answers, a week before the exam. Everyone scores 100%. Does that prove they understand the subject? Of course not. They might just have memorised the answers.

Models do the same thing. A flexible model can memorise its training data and score perfectly on it, then fail badly on new data. So we hold back some data the model never sees during training, and grade it only on that. That hidden part is the test set.

How the Split Works

1. Shuffle the rows, so both parts get a fair mix. If your file is sorted by date or by class, an unshuffled split would put all the old rows in training and all the new ones in testing.
2. Cut it into two parts. A common choice is 80% training, 20% testing. With a big dataset, 90/10 or even 99/1 is fine, because 1% of a million rows is still 10,000 test rows.
3. Train only on the training part.
4. Evaluate only on the test part.

In scikit-learn

train_test_split does the shuffle and the cut in one line.

import numpy as np
from sklearn.model_selection import train_test_split

X = np.arange(20).reshape(-1, 1)   # 20 rows, 1 feature
y = np.arange(20) * 2              # 20 answers

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

print(X_train.shape, X_test.shape)
print(y_train.shape, y_test.shape)

(16, 1) (4, 1)
(16,) (4,)

16 rows for training and 4 for testing: exactly 80/20. Three arguments matter:

Why random_state?

Without it, every run gives a different shuffle, so every run gives a slightly different score. That makes it impossible to tell whether a change you made actually helped. The number itself doesn't matter: 42, 0 or 7 all work. Just keep it fixed.

Why stratify?

Suppose 95 of your 100 emails are normal and 5 are spam. A random 20-row test set might, by bad luck, contain no spam at all. Then you can't measure spam detection. With stratify=y, both parts keep the 95/5 ratio.

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

Data Leakage: The Silent Bug

Data leakage means information from the test set sneaks into training. The model then looks better than it really is, and that only shows up when it fails in the real world.

The most common leak happens when you scale or fill in values before splitting:

# Wrong: the scaler sees the test rows, so test info leaks into training
X_scaled = scaler.fit_transform(X)
X_train, X_test = train_test_split(X_scaled, test_size=0.2)

# Right: split first, learn the scaling from training rows only
X_train, X_test = train_test_split(X, test_size=0.2, random_state=42)
X_train = scaler.fit_transform(X_train)   # fit on train
X_test = scaler.transform(X_test)         # only transform test

What About a Validation Set?

When you try many models or settings, you choose the one with the best score. If you choose using the test set, the test set slowly stops being "unseen". So bigger projects split three ways:

• Training set: fit the model.
• Validation set: compare models and settings.
• Test set: touched once, at the very end, for the final honest score.

Later, cross-validation gives you a smarter way to do the validation step without wasting data.

Summary

• Never judge a model on the data it learned from.
• Shuffle, then split, usually 80/20, with random_state so the split is repeatable.
• Use stratify=y for classification with uneven classes.
• Split first to avoid data leakage.
• Next: your first real algorithm, simple linear regression.