Support Vector Regression (SVR)

Turning the SVM Idea Around

In classification, an SVM wants a wide street with no points inside it. SVR flips this: it wants a tube that contains as many points as possible inside it.

• Points inside the tube cost nothing. An error of 2 marks is fine if the tube is ±5 marks wide.
• Points outside the tube are penalised by how far they stick out.
• Among all tubes that fit, SVR prefers the flattest function, the simplest explanation.

The half-width of the tube is called epsilon ($\epsilon$).

How It Differs From Linear Regression

The SVR loss is called the ε-insensitive loss:

With ε = 5: an error of 3 costs 0; an error of 8 costs 3.

The Settings

Our Students Again

import numpy as np
from sklearn.svm import SVR

hours = np.array([[1], [2], [3], [4], [5]])
marks = np.array([20, 40, 50, 40, 50])

for eps in [1, 5, 12]:
    svr = SVR(kernel="linear", C=100, epsilon=eps).fit(hours, marks)
    print(f"epsilon={eps:>2}: slope {svr.coef_[0, 0]:5.2f}   intercept {svr.intercept_[0]:6.2f}   support vectors {len(svr.support_)}")

epsilon= 1: slope  3.33   intercept  32.33   support vectors 5
epsilon= 5: slope  5.00   intercept  25.00   support vectors 4
epsilon=12: slope  3.00   intercept  29.00   support vectors 2

As epsilon grows, more students fit inside the tube, so fewer act as support vectors: 5, then 4, then only 2. The line moves each time, because only the points on or outside the tube shape it. With ε = 12 the tube is ±12 marks wide, and most students sit comfortably inside it.

A Real Dataset

SVR needs scaled features, and usually a scaled target too when the target's numbers are large. TransformedTargetRegressor scales the target and un-scales the predictions for you:

from sklearn.compose import TransformedTargetRegressor
from sklearn.datasets import load_diabetes
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)

model = TransformedTargetRegressor(
    regressor=make_pipeline(StandardScaler(), SVR(kernel="rbf")),
    transformer=StandardScaler(),
)
grid = {"regressor__svr__C": [0.1, 1, 10], "regressor__svr__epsilon": [0.05, 0.2, 0.5]}
search = GridSearchCV(model, grid, cv=5).fit(X_train, y_train)
print("best settings:", search.best_params_)
print(f"test R2: {search.score(X_test, y_test):.3f}")

best settings: {'regressor__svr__C': 1, 'regressor__svr__epsilon': 0.2}
test R2: 0.239

Because the target is scaled, epsilon here is measured in standard deviations of the target rather than in raw units.

Summary

• SVR fits an ε-tube; errors inside are ignored.
• epsilon sets the tube width; C sets the penalty for points outside.
• Scale features (and usually the target), then tune with GridSearchCV.
• Next: kernels, which let SVMs and SVR draw curved boundaries.