Start learning

SVM Kernels

Lesson 41. About 7 minutes.

Kernels let a support vector machine draw curved boundaries. They work by implicitly moving the data into a higher-dimensional space where a straight boundary can separate it, without ever computing that space directly.

Your Path Through This Lesson

0 of 5 done

By the End of This Lesson You Can
  • See how adding a dimension turns a curved problem into a straight one.
  • Know the common kernels: linear, polynomial, RBF.
  • Tune C and gamma for an RBF kernel.

When No Line Can Separate the Classes

Imagine points on a number line: class A sits in the middle (between −1 and 1) and class B on both sides. No single cut on the line separates them.

Now add a second feature: . In the new plane, class A sits low (small ) and class B sits high. A horizontal line at, say, separates them perfectly.

That's the whole trick: a curved boundary in the original space is a straight boundary in a bigger space.

The Kernel Trick

Adding features by hand gets out of control: with 30 features and degree-3 polynomials you'd have thousands of new columns. The SVM's maths only ever needs dot products between pairs of points. A kernel is a function that computes the dot product in the bigger space directly from the original features, cheaply. So the SVM gets the benefit of the high-dimensional space without building it. That's the kernel trick.

Common Kernels

Kernel Formula Boundary shape Settings
Linear Straight line C
Polynomial Polynomial curves degree, gamma, coef0, C
RBF (Gaussian) Any smooth shape gamma, C
Sigmoid Neural-network-like gamma, coef0, C

The RBF kernel is the default, and the most common choice. It measures similarity: nearby points get a kernel value close to 1, distant points close to 0.

Seeing the Difference

make_circles creates one class inside a ring of the other: impossible for a straight line.

python
from sklearn.datasets import make_circles
from sklearn.model_selection import cross_val_score
from sklearn.svm import SVC

X, y = make_circles(n_samples=300, factor=0.4, noise=0.08, random_state=0)

for kernel in ["linear", "poly", "rbf"]:
    model = SVC(kernel=kernel, degree=2) if kernel == "poly" else SVC(kernel=kernel)
    print(f"{kernel:>6}: CV accuracy {cross_val_score(model, X, y, cv=5).mean():.3f}")
Output
linear: CV accuracy 0.560
  poly: CV accuracy 1.000
   rbf: CV accuracy 1.000

The linear kernel is no better than guessing. A degree-2 polynomial or the RBF kernel separates the circles almost perfectly.

The 1D Example in Code

python
import numpy as np

x = np.array([-3, -2, -0.5, 0, 0.5, 2, 3]).reshape(-1, 1)
y = np.array([1, 1, 0, 0, 0, 1, 1])               # 0 = middle, 1 = outside

flat = SVC(kernel="linear").fit(x, y)
lifted = SVC(kernel="linear").fit(np.hstack([x, x ** 2]), y)   # add x^2 by hand
print(f"line on x only     : train accuracy {flat.score(x, y):.3f}")
print(f"line on (x, x^2)   : train accuracy {lifted.score(np.hstack([x, x ** 2]), y):.3f}")
Output
line on x only     : train accuracy 0.571
line on (x, x^2)   : train accuracy 1.000

Adding by hand makes a straight line work. A polynomial or RBF kernel does the same thing automatically.

Gamma: How Far One Point's Influence Reaches

In the RBF kernel, gamma sets how quickly similarity fades with distance.

gamma Each point influences… Boundary Risk
Small A wide area Smooth, gentle curves Underfitting
Large Only its close neighbours Wiggly, wraps around single points Overfitting

C and gamma work together, so tune them together:

python
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

Xc, yc = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(Xc, yc, test_size=0.3, random_state=0, stratify=yc)

pipe = make_pipeline(StandardScaler(), SVC(kernel="rbf"))
grid = {"svc__C": [0.1, 1, 10, 100], "svc__gamma": [0.001, 0.01, 0.1, 1]}
search = GridSearchCV(pipe, grid, cv=5).fit(X_train, y_train)
print("best settings:", search.best_params_)
print(f"test accuracy: {search.score(X_test, y_test):.3f}")
Output
best settings: {'svc__C': 100, 'svc__gamma': 0.001}
test accuracy: 0.953
Tip

Start with the RBF kernel and the default gamma="scale", then grid-search C and gamma on a log scale (0.001, 0.01, 0.1, …). Try the linear kernel too: with many features, it's often just as good and much faster.

Try It Yourself
  1. On the circles data, fit SVC(kernel="rbf", gamma=100). What's the training accuracy? The cross-validation accuracy?
  2. Compute the RBF kernel value by hand for two points at distance 1 with γ = 0.5. (Hint: .)
The Key Idea

A kernel lets an SVM draw a straight boundary in a hidden higher-dimensional space, which looks curved in the original one. RBF is the default; tune C and gamma together.

Summary

  • Curved boundaries in the original space can be straight in a higher-dimensional space.
  • The kernel trick computes that space's dot products without building it.
  • Kernels: linear, polynomial, RBF (default), sigmoid.
  • gamma controls how local each point's influence is; tune it together with C.
  • Next part: unsupervised learning, starting with K-means clustering.