Start learning

Logistic Regression in scikit-learn

Lesson 24. About 6 minutes.

This lesson puts the last four together: a complete logistic regression project in scikit-learn, from loading data to tuning the model and reading its results.

Your Path Through This Lesson

0 of 5 done

By the End of This Lesson You Can
  • Build a clean logistic regression pipeline with scaling.
  • Tune the regularisation strength C and the penalty type.
  • Evaluate it with a confusion matrix, a classification report and log loss.

The Dataset

We'll use scikit-learn's breast cancer dataset: 569 tumours, 30 measurements each (radius, texture, smoothness and so on), labelled malignant or benign. We'll make malignant the positive class, because that's the one we must not miss.

python
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

data = load_breast_cancer()
X, y = data.data, 1 - data.target              # 1 = malignant, 0 = benign
print("rows, features:", X.shape)
print("malignant:", int(y.sum()), " benign:", int((y == 0).sum()))

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)
Output
rows, features: (569, 30)
malignant: 212  benign: 357

We use stratify=y so both parts keep the same share of malignant tumours.

Step 1: A Baseline Pipeline

Logistic regression works best with scaled features, and putting the scaler in a pipeline guarantees it's fitted on training data only.

python
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

pipe = Pipeline([
    ("scale", StandardScaler()),
    ("clf", LogisticRegression(max_iter=1000)),
])
pipe.fit(X_train, y_train)
print("test accuracy:", round(pipe.score(X_test, y_test), 3))
Output
test accuracy: 0.965

Step 2: The Important Settings

Setting What it does Typical values
C Inverse of regularisation strength: small C = strong penalty 0.01 to 100
l1_ratio Type of penalty: 0 = L2 (Ridge-like), 1 = L1 (Lasso-like) 0 (default), 1, or in between
solver The optimisation method lbfgs (default), liblinear, saga
max_iter Maximum training steps Raise it if you see a convergence warning
class_weight Give the rarer class more weight None or "balanced"
Watch Out

C works the opposite way to alpha in Ridge and Lasso. A larger C means less regularisation. Also, older tutorials write penalty="l1"; in current scikit-learn you choose the penalty type with l1_ratio instead, and L1 needs the saga or liblinear solver.

Step 3: Tuning With GridSearchCV

python
from sklearn.model_selection import GridSearchCV

grid = {
    "clf__C": [0.01, 0.1, 1, 10, 100],
    "clf__l1_ratio": [0, 1],
    "clf__solver": ["saga"],
}
search = GridSearchCV(Pipeline([("scale", StandardScaler()), ("clf", LogisticRegression(max_iter=5000))]),
                      grid, cv=5, scoring="recall")
search.fit(X_train, y_train)
print("best settings:", search.best_params_)
print("best CV recall:", round(search.best_score_, 3))
Output
best settings: {'clf__C': 1, 'clf__l1_ratio': 0, 'clf__solver': 'saga'}
best CV recall: 0.956

We tuned for recall, because a missed malignant tumour is the costliest mistake.

Step 4: Evaluating Properly

python
from sklearn.metrics import confusion_matrix, classification_report, log_loss

best = search.best_estimator_
pred = best.predict(X_test)
proba = best.predict_proba(X_test)[:, 1]

print(confusion_matrix(y_test, pred))
print(classification_report(y_test, pred, target_names=["benign", "malignant"], digits=3))
print("log loss:", round(log_loss(y_test, proba), 3))
Output
[[89  1]
 [ 4 49]]
              precision    recall  f1-score   support

      benign      0.957     0.989     0.973        90
   malignant      0.980     0.925     0.951        53

    accuracy                          0.965       143
   macro avg      0.968     0.957     0.962       143
weighted avg      0.966     0.965     0.965       143

log loss: 0.077

Read the confusion matrix as [[TN, FP], [FN, TP]]. The bottom-left number counts malignant tumours the model missed. The report gives precision, recall and F1 for each class.

Step 5: Which Features Drive the Prediction?

Because the features are scaled, the coefficients can be compared. Positive coefficients push towards malignant:

python
coefs = best.named_steps["clf"].coef_[0]
for i in np.argsort(-np.abs(coefs))[:5]:
    print(f"{data.feature_names[i]:>22}: {coefs[i]:+.2f}")
Output
         worst texture: +1.34
          radius error: +1.26
        worst symmetry: +1.03
   mean concave points: +0.95
     compactness error: -0.93
Tip

If the classes were very imbalanced, set class_weight="balanced". It makes mistakes on the rare class count more during training, which usually raises recall for that class.

Try It Yourself
  1. Change scoring="recall" to scoring="f1" and rerun the search. Do the best settings change?
  2. Use best.predict_proba with a threshold of 0.3 instead of 0.5. How many malignant tumours are missed now?
The Key Idea

A solid recipe: scale in a pipeline, tune C with cross-validation on the metric that matters, then check the confusion matrix on the test set.

Summary

  • Pipeline: StandardScaler → LogisticRegression.
  • Tune C (inverse strength) and l1_ratio (penalty type) with GridSearchCV on the metric that matters.
  • Evaluate with a confusion matrix, classification report and log loss.
  • Scaled coefficients show which features push towards each class.
  • Next: handling more than two classes with one-vs-rest.