Lesson 9 · Scikit-learn deep dive

Scikit-learn Tutorial #9: Hyperparameter Tuning

Video nine of the eighteen-part series: systematically searching for the best model settings. GridSearchCV, RandomizedSearchCV, bestparams, and…

⬇ Download notebook Open in Colab ↗

CourseScikit-learn deep dive
Lesson9 of 18
Video15 min
FormatJupyter notebook · 10 code cells

scikit-learn pandas SciPy

What you'll learn

Hyperparameters vs Learned Parameters
GridSearchCV Basics - paramgrid
bestparams, bestscore, bestestimator
GridSearchCV with a Pipeline - stepparam Naming
cvresults - Inspecting All Combinations
RandomizedSearchCV - paramdistributions, niter
Continuous Distributions - loguniform
scoring - Optimizing for a Specific Metric

Data

No separate download needed — the notebook creates or downloads everything it uses.

📓 Full notebook

Download .ipynb

Scikit-learn Deep-Dive, Video 9: Hyperparameter Tuning#

Video nine of the eighteen-part series: systematically searching for the best model settings.
GridSearchCV, RandomizedSearchCV, best_params_, and best_estimator_.
Let's get into it.

Part 1: Hyperparameters vs Learned Parameters#

from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
X, y = load_wine(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)
manual_scores = {}
for c_value in [0.1, 1, 10]:
    manual_scores[c_value] = SVC(C=c_value).fit(X_train, y_train).score(X_test, y_test)
print(manual_scores)

{0.1: 0.6888888888888889, 1: 0.7111111111111111, 10: 0.7333333333333333}

Part 2: GridSearchCV Basics - param_grid#

from sklearn.model_selection import GridSearchCV
param_grid = {'C': [0.1, 1, 10], 'kernel': ['linear', 'rbf']}
grid = GridSearchCV(SVC(), param_grid, cv=5)
grid.fit(X_train, y_train)
print(round(grid.score(X_test, y_test), 3))

0.956

Part 3: best_params_, best_score_, best_estimator_#

print(grid.best_params_)
print(round(grid.best_score_, 3))
print(type(grid.best_estimator_).__name__)

{'C': 1, 'kernel': 'linear'}
0.955
SVC

Part 4: GridSearchCV with a Pipeline - step__param Naming#

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
pipe = Pipeline([('scaler', StandardScaler()), ('svc', SVC())])
pipe_grid = {'svc__C': [0.1, 1, 10], 'svc__kernel': ['linear', 'rbf']}
pipe_search = GridSearchCV(pipe, pipe_grid, cv=5)
pipe_search.fit(X_train, y_train)
print(pipe_search.best_params_)

{'svc__C': 1, 'svc__kernel': 'linear'}

Part 5: cv_results_ - Inspecting All Combinations#

import pandas as pd
results_df = pd.DataFrame(grid.cv_results_)
summary = results_df[['param_C', 'param_kernel', 'mean_test_score', 'rank_test_score']]
print(summary.sort_values('rank_test_score').to_string(index=False))

 param_C param_kernel  mean_test_score  rank_test_score
     1.0       linear         0.954986                1
     0.1       linear         0.947578                2
    10.0       linear         0.947578                2
    10.0          rbf         0.737892                4
     0.1          rbf         0.662108                5
     1.0          rbf         0.639886                6

Part 6: RandomizedSearchCV - param_distributions, n_iter#

from sklearn.model_selection import RandomizedSearchCV
from sklearn.ensemble import RandomForestClassifier
param_dist = {'n_estimators': [50, 100, 150, 200], 'max_depth': [3, 5, 7, None]}
random_search = RandomizedSearchCV(
    RandomForestClassifier(random_state=42), param_dist, n_iter=6, cv=5, random_state=42
)
random_search.fit(X_train, y_train)
print(random_search.best_params_)

{'n_estimators': 100, 'max_depth': 3}

Part 7: Continuous Distributions - loguniform#

from scipy.stats import loguniform
continuous_dist = {'C': loguniform(1e-2, 1e2), 'kernel': ['linear', 'rbf']}
continuous_search = RandomizedSearchCV(
    SVC(), continuous_dist, n_iter=8, cv=5, random_state=42
)
continuous_search.fit(X_train, y_train)
print(round(continuous_search.best_params_['C'], 4))

2.481

Part 8: scoring - Optimizing for a Specific Metric#

f1_grid = GridSearchCV(SVC(), param_grid, cv=5, scoring='f1_macro')
f1_grid.fit(X_train, y_train)
print(f1_grid.best_params_)
print(round(f1_grid.best_score_, 3))

{'C': 1, 'kernel': 'linear'}
0.957

Part 9: refit - Using the Tuned Model Directly#

predictions = grid.predict(X_test)
print(predictions[:10])
print(y_test[:10])
print((predictions == y_test).mean().round(3))

[0 1 0 0 1 0 0 1 1 1]
[0 1 0 0 1 0 0 1 1 2]
0.956

Part 10: A Real Pattern - a Reusable tune_model Function#

def tune_model(estimator, param_grid, X, y, scoring='accuracy', cv=5):
    search = GridSearchCV(estimator, param_grid, cv=cv, scoring=scoring)
    search.fit(X, y)
    return search.best_estimator_, search.best_params_, round(search.best_score_, 3)
best_model, best_params, best_score = tune_model(SVC(), param_grid, X_train, y_train)
print(best_params, best_score)

{'C': 1, 'kernel': 'linear'} 0.955

Wrap-Up: What You Learned#

Hyperparameters are chosen before fitting; manually trying combinations is tedious and inconsistent.
GridSearchCV tries every combination in a param_grid, cross-validating each one automatically.
best_params_, best_score_, and best_estimator_ report the winning combination and give a ready-to-use fitted model.
Tuning inside a Pipeline uses step__param naming, keeping preprocessing leakage-safe during the search.
cv_results_ holds every combination tried, loadable into a DataFrame for full comparison.
RandomizedSearchCV samples n_iter random combinations from param_distributions, scaling better than a full grid.
param_distributions accepts scipy distributions like loguniform for parameters spanning orders of magnitude.
scoring controls what the search actually optimizes for, not just the estimator's default score method.
refit=True (the default) automatically refits best_estimator_, so the search object itself can predict directly.
That wraps up hyperparameter tuning. Next up: Classification Metrics, precision, recall, F1, and ROC-AUC.

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.

☕ Buy me a coffee Other ways to help