Lesson 8 · Scikit-learn deep dive
Scikit-learn Tutorial #8: Splitting & Cross-Validation
Honestly measuring how well a model generalizes. traintestsplit, KFold, StratifiedKFold, crossvalscore, and crossvalidate. Let's get into it. traintestsplit…
- CourseScikit-learn deep dive
- Lesson8 of 18
- Video15 min
- FormatJupyter notebook · 10 code cells
What you'll learn
Data
No separate download needed — the notebook creates or downloads everything it uses.
📓 Full notebook
Download .ipynbScikit-learn Deep-Dive, Video 8: Model Selection - Splitting and Cross-Validation#
- Video eight of the eighteen-part series: honestly measuring how well a model generalizes.
- train_test_split, KFold, StratifiedKFold, cross_val_score, and cross_validate.
- Let's get into it.
Part 1: train_test_split Basics#
from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
X, y = load_wine(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
print(X_train.shape, X_test.shape)
Part 2: stratify - Preserving Class Balance#
import numpy as np
X_tr_strat, X_te_strat, y_tr_strat, y_te_strat = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)
print(np.bincount(y) / len(y))
print(np.bincount(y_te_strat) / len(y_te_strat))
Part 3: KFold - Basic K-Fold Splitting#
from sklearn.model_selection import KFold
kf = KFold(n_splits=5, shuffle=True, random_state=42)
for i, (train_idx, val_idx) in enumerate(kf.split(X)):
print(i, len(train_idx), len(val_idx))
Part 4: StratifiedKFold - Preserving Class Balance Across Folds#
from sklearn.model_selection import StratifiedKFold
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
for i, (train_idx, val_idx) in enumerate(skf.split(X, y)):
fold_props = np.bincount(y[val_idx]) / len(val_idx)
print(i, fold_props.round(2))
Part 5: cross_val_score with a cv Object#
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
scores = cross_val_score(model, X, y, cv=skf)
print(scores.round(3))
print(round(scores.mean(), 3), round(scores.std(), 3))
Part 6: cross_validate - Multiple Metrics and Timing#
from sklearn.model_selection import cross_validate
results = cross_validate(
model, X, y, cv=skf,
scoring=['accuracy', 'f1_macro'],
return_train_score=True
)
print(sorted(results.keys()))
print(results['test_accuracy'].round(3))
print(results['test_f1_macro'].round(3))
Part 7: shuffle and random_state - Why They Matter#
kf_unshuffled = KFold(n_splits=5, shuffle=False)
kf_shuffled_a = KFold(n_splits=5, shuffle=True, random_state=1)
kf_shuffled_b = KFold(n_splits=5, shuffle=True, random_state=1)
first_fold_a = next(kf_shuffled_a.split(X))[1]
first_fold_b = next(kf_shuffled_b.split(X))[1]
print(np.array_equal(first_fold_a, first_fold_b))
Part 8: ShuffleSplit - Repeated Random Subsampling#
from sklearn.model_selection import ShuffleSplit
ss = ShuffleSplit(n_splits=10, test_size=0.2, random_state=42)
ss_scores = cross_val_score(model, X, y, cv=ss)
print(ss_scores.round(3))
print(round(ss_scores.mean(), 3))
Part 9: KFold vs StratifiedKFold on Imbalanced Data#
y_imbalanced = (y == 2).astype(int)
print(np.bincount(y_imbalanced))
kf_plain = KFold(n_splits=5, shuffle=True, random_state=42)
skf_strat = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
plain_props = [np.bincount(y_imbalanced[val])[1] for _, val in kf_plain.split(X, y_imbalanced)]
strat_props = [np.bincount(y_imbalanced[val])[1] for _, val in skf_strat.split(X, y_imbalanced)]
print('plain KFold rare-class counts:', plain_props)
print('StratifiedKFold rare-class counts:', strat_props)
Part 10: A Real Pattern - a Reusable evaluate_model_cv Function#
def evaluate_model_cv(estimator, X, y, n_splits=5, metrics=('accuracy', 'f1_macro')):
cv = StratifiedKFold(n_splits=n_splits, shuffle=True, random_state=42)
results = cross_validate(estimator, X, y, cv=cv, scoring=list(metrics))
summary = {}
for metric in metrics:
scores = results[f'test_{metric}']
summary[metric] = (round(scores.mean(), 3), round(scores.std(), 3))
return summary
print(evaluate_model_cv(model, X, y))
Wrap-Up: What You Learned#
- train_test_split shuffles and splits data in one call; test_size and random_state control the split.
- stratify=y keeps each class's proportion consistent across train and test subsets.
- KFold divides data into k folds, using each once for validation while training on the rest.
- StratifiedKFold extends KFold by keeping class balance consistent within every fold, the recommended default for classification.
- Passing a configured splitter object as cv gives full control over the exact cross-validation strategy.
- cross_validate returns a dictionary supporting multiple scoring metrics and fit/score timing at once.
- shuffle=True plus random_state avoids splitting on ordered data while staying reproducible.
- ShuffleSplit generates independent random train/test splits, useful for more repeats than KFold alone allows.
- Plain KFold can under-represent a rare class in a fold; StratifiedKFold guards against that by construction.
- That wraps up model selection. Next up: Hyperparameter Tuning with GridSearchCV and RandomizedSearchCV.
🧠 Test yourself
Pick an answer to check it. Your best score is saved in this browser.
What does
stratify=ydo intrain_test_split(X, y, test_size=0.2, random_state=42, stratify=y)?The lesson compared np.bincount(y) / len(y) with the test set's proportions and they matched closely. That keeps small classes from vanishing from a split.
What does this code do?
kf = KFold(n_splits=5, shuffle=True, random_state=42) for i, (train_idx, val_idx) in enumerate(kf.split(X)): print(i, len(train_idx), len(val_idx))
Each loop gives index arrays: about 80 percent for training and 20 percent for validation. Every row is validated exactly once.
Why does the lesson recommend StratifiedKFold for classification?
On imbalanced data plain KFold can put too few rare-class rows in a fold. StratifiedKFold avoids that by construction.
Two splitters are created with
KFold(n_splits=5, shuffle=True, random_state=1). What doesnp.array_equal(first_fold_a, first_fold_b)print?shuffle=True avoids ordered data problems, and random_state makes the shuffle repeatable. Same seed, same folds.
What extra does cross_validate give you over cross_val_score, as used here?
results = cross_validate(model, X, y, cv=skf, scoring=['accuracy', 'f1_macro'], return_train_score=True)
The result keys include test_accuracy, test_f1_macro, train scores, fit_time and score_time. cross_val_score returns just one array.
💻 Practice exercises
Try each one in the notebook or in your own editor before opening the solution.
Split the wine data 80/20 with stratify, then print the class proportions of the full data and of the test set.
💡 Hint
np.bincount(y) / len(y) gives class proportions.
✅ Show solution
import matplotlib matplotlib.use('Agg') import numpy as np from sklearn.datasets import load_wine from sklearn.model_selection import train_test_split X, y = load_wine(return_X_y=True) X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42, stratify=y) print('shapes:', X_train.shape, X_test.shape) print('full data:', (np.bincount(y) / len(y)).round(3)) print('test set: ', (np.bincount(y_test) / len(y_test)).round(3))
Evaluate a scaled logistic regression with 5-fold StratifiedKFold using cross_val_score. Print each fold score, the mean and the standard deviation.
💡 Hint
Wrap StandardScaler and LogisticRegression with make_pipeline.
✅ Show solution
import matplotlib matplotlib.use('Agg') from sklearn.datasets import load_wine from sklearn.model_selection import StratifiedKFold, cross_val_score from sklearn.pipeline import make_pipeline from sklearn.preprocessing import StandardScaler from sklearn.linear_model import LogisticRegression X, y = load_wine(return_X_y=True) model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)) skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42) scores = cross_val_score(model, X, y, cv=skf) print('fold scores:', scores.round(3)) print('mean:', round(scores.mean(), 3), 'std:', round(scores.std(), 3))
Turn the wine problem into an imbalanced one (class 2 versus the rest). Count how many rare-class rows land in each validation fold for plain KFold versus StratifiedKFold, then print both lists.
💡 Hint
Use (y == 2).astype(int) and np.bincount on y_imbalanced[val_idx].
✅ Show solution
import matplotlib matplotlib.use('Agg') import numpy as np from sklearn.datasets import load_wine from sklearn.model_selection import KFold, StratifiedKFold X, y = load_wine(return_X_y=True) y_imb = (y == 2).astype(int) print('class counts:', np.bincount(y_imb)) kf = KFold(n_splits=5, shuffle=True, random_state=42) skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42) plain = [int(np.bincount(y_imb[val], minlength=2)[1]) for _, val in kf.split(X)] strat = [int(np.bincount(y_imb[val], minlength=2)[1]) for _, val in skf.split(X, y_imb)] print('plain KFold rare-class counts: ', plain) print('StratifiedKFold rare-class counts: ', strat) print('stratified spread is no larger:', max(strat) - min(strat) <= max(plain) - min(plain))
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.



