Start learning

Principal Component Analysis (PCA)

Lesson 47. About 7 minutes.

PCA reduces many features to a few new ones, called principal components. Each component is a weighted mix of the original features, chosen to keep as much of the data's spread (variance) as possible.

Your Path Through This Lesson

0 of 5 done

By the End of This Lesson You Can
  • Explain what a principal component is.
  • Use explained variance to decide how many components to keep.
  • Use PCA in a pipeline before a model.

The Idea

Imagine the marks of many students in Maths and Physics. The two subjects rise and fall together, so the points form a long, thin cloud along a diagonal line.

  • Along the diagonal, the points are spread out a lot: that's where the information is ("overall science ability").
  • Across the diagonal, they barely vary: little information there.

PCA finds that diagonal direction and calls it principal component 1 (PC1). PC2 is the next-best direction, at right angles to PC1. If PC1 holds most of the spread, you can keep just PC1 and replace two features with one, losing little.

The Steps

  1. Standardise each feature (mean 0, standard deviation 1).
  2. Find the direction of greatest variance: PC1.
  3. Find the direction of greatest remaining variance, perpendicular to the previous ones: PC2, PC3, …
  4. Keep the first components and project the data onto them.

Mathematically, the components are the eigenvectors of the features' covariance matrix, and the variance along each is its eigenvalue. scikit-learn computes them for you.

Two Correlated Features

python
import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(0)
maths = rng.normal(60, 12, size=200)
physics = 0.9 * maths + rng.normal(0, 5, size=200)
X = StandardScaler().fit_transform(np.column_stack([maths, physics]))

pca = PCA().fit(X)
print("explained variance ratio:", pca.explained_variance_ratio_.round(3))
print("PC1 direction (maths, physics):", pca.components_[0].round(3))
Output
explained variance ratio: [0.946 0.054]
PC1 direction (maths, physics): [0.707 0.707]

PC1 carries almost all of the variance, and its direction weights maths and physics equally: it really is "overall ability". PC2 adds very little, so one number per student is nearly as informative as two.

Note

The sign of a component is arbitrary: (0.707, 0.707) and (−0.707, −0.707) describe the same direction.

How Many Components?

The explained variance ratio tells you how much of the total spread each component keeps. A common rule: keep enough components to explain about 95% of the variance.

python
from sklearn.datasets import load_breast_cancer

Xc, yc = load_breast_cancer(return_X_y=True)
Xc_scaled = StandardScaler().fit_transform(Xc)

pca = PCA().fit(Xc_scaled)
cumulative = np.cumsum(pca.explained_variance_ratio_)
for k in [1, 2, 3, 5, 10]:
    print(f"{k:>2} components keep {cumulative[k - 1]:.1%} of the variance")
print("components needed for 95%:", int(np.argmax(cumulative >= 0.95)) + 1)
Output
 1 components keep 44.3% of the variance
 2 components keep 63.2% of the variance
 3 components keep 72.6% of the variance
 5 components keep 84.7% of the variance
10 components keep 95.2% of the variance
components needed for 95%: 10

The 30 tumour measurements are heavily correlated (radius, perimeter and area move together), so 10 components keep about 95% of the information.

You can also pass the target directly: PCA(n_components=0.95) keeps just enough components to reach 95%.

PCA Before a Model

python
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline

for n in [2, 5, 10, None]:
    steps = [StandardScaler()] + ([PCA(n_components=n)] if n else []) + [LogisticRegression(max_iter=5000)]
    score = cross_val_score(make_pipeline(*steps), Xc, yc, cv=5).mean()
    label = f"{n} components" if n else "all 30 features"
    print(f"{label:>15}: CV accuracy {score:.3f}")
Output
   2 components: CV accuracy 0.951
   5 components: CV accuracy 0.970
  10 components: CV accuracy 0.981
all 30 features: CV accuracy 0.981

Even 2 components keep most of the predictive power, and 10 are about as good as all 30. With fewer features, models train faster and suffer less from the curse of dimensionality.

Watch Out

Put StandardScaler and PCA inside the pipeline, so they're fitted only on training folds. Fitting PCA on all the data before splitting leaks information from the test set.

Uses and Limits

Good for Watch out for
Compressing many correlated features Components are mixes, so they're harder to explain than original features
Speeding up models It only finds straight-line (linear) structure
Plotting high-dimensional data in 2D It ignores the target: high-variance directions aren't always the predictive ones
Removing noise Always standardise first, or large-unit features dominate
Tip

For plotting complex data in 2D, t-SNE and UMAP often show clusters more clearly than PCA, but they're for visualisation only, not for feeding models.

Try It Yourself
  1. Make physics independent of maths (rng.normal(55, 12, 200)). What explained variance ratios do you get now?
  2. Plot the breast-cancer data on PC1 and PC2, coloured by yc. Are the two classes visibly separated?
The Key Idea

PCA rotates the data onto new axes ordered by variance; keep the first few components that explain most of it. Standardise first and put PCA inside a pipeline.

Summary

  • PCA builds new uncorrelated features (components) ordered by variance.
  • Use the explained variance ratio to choose how many to keep (e.g. 95%).
  • Standardise first; put PCA in a pipeline before the model.
  • It's linear and ignores the target; components are harder to interpret.
  • Next part: putting everything together, the machine learning workflow.