Support Vector Machines (SVM)

Many Lines, One Best Line

Picture pass and fail students plotted by hours and attendance, with a clear gap between the groups. Infinitely many straight lines separate them. Which one is best?

An SVM picks the line with the largest margin: the widest street you can draw between the classes without any points inside it. The decision boundary runs down the middle of the street.

Why widest? A boundary that only just squeezes past the points is fragile: a slightly different new student lands on the wrong side. A boundary with lots of room on both sides is more likely to classify new points correctly.

Watch the street being drawn:

Support Vectors

Only the points on the edges of the street decide where it goes. These are the support vectors. Move any other point, as long as it stays outside the street, and the boundary doesn't change at all.

That's unusual: logistic regression uses every point; an SVM depends only on the few hardest ones.

The Maths, Briefly

The boundary is a line (or plane) $w\cdot x+b=0$. The street's edges are $w\cdot x+b=\pm 1$, and its width is $\frac{2}{\Vert w\Vert }$. So maximising the margin means minimising $\Vert w\Vert$, while keeping every point on its correct side:

(with labels $y_i=+1$ or $-1$).

Soft Margins and C

Real data is rarely perfectly separable, and one odd point shouldn't wreck the boundary. A soft-margin SVM allows some points inside the street, or even on the wrong side, but charges a penalty for each. The setting C controls how heavy that penalty is.

import numpy as np
from sklearn.datasets import make_blobs
from sklearn.svm import SVC

X, y = make_blobs(n_samples=60, centers=2, cluster_std=1.2, random_state=6)

for C in [0.01, 1, 100]:
    svm = SVC(kernel="linear", C=C).fit(X, y)
    width = 2 / np.linalg.norm(svm.coef_)
    print(f"C={C:>6}: support vectors {len(svm.support_vectors_):>2}   margin width {width:.2f}   train accuracy {svm.score(X, y):.3f}")

C=  0.01: support vectors 22   margin width 5.53   train accuracy 1.000
C=     1: support vectors  3   margin width 1.91   train accuracy 1.000
C=   100: support vectors  3   margin width 1.91   train accuracy 1.000

Small C gives a wide margin with many points inside it, and all of them are support vectors. From C = 1 upwards, the SVM refuses to let points into the street, so the margin narrows until just 3 points touch its edges. These two blobs are fully separable, so raising C further changes nothing.

A Real Dataset

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

Xc, yc = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(Xc, yc, test_size=0.3, random_state=0, stratify=yc)

pipe = make_pipeline(StandardScaler(), SVC(kernel="linear"))
search = GridSearchCV(pipe, {"svc__C": [0.001, 0.01, 0.1, 1, 10]}, cv=5).fit(X_train, y_train)
print("best C:", search.best_params_["svc__C"])
print(f"test accuracy: {search.score(X_test, y_test):.3f}")

best C: 0.1
test accuracy: 0.959

Summary

• An SVM chooses the boundary with the maximum margin.
• Support vectors are the points that define the margin.
• C trades margin width against training errors.
• Scale features first; use GridSearchCV to choose C.
• Next: using the same idea to predict numbers, support vector regression.