Mathew K Analytics

Lesson 53 · Pandas Full Course

Train-Test Split in Pandas and scikit-learn (Avoid Data Leakage) | Pandas Course #53

In this lesson you'll be able to: Explain why a model must be tested on rows it never saw Split a DataFrame with pandas sample and with traintestsplit Keep…

📓 Full notebook

Download .ipynb

Pandas 53: Splitting Data into Train-Test Sets#

In this lesson you'll be able to:

  • Explain why a model must be tested on rows it never saw
  • Split a DataFrame with pandas sample and with train_test_split
  • Keep class balance with stratify, and avoid data leakage
  • Split time series by date, and add a validation step
  • Use the split to catch an overfitting model Data: the Titanic passenger list, plus monthly airline passengers

Warm-up: features, from lesson 52#

Lesson 52 engineered new features. We rebuild two of them.

import pandas as pd
df = pd.read_csv("data/pandas/titanic.csv")
df["family_size"] = df["SibSp"] + df["Parch"] + 1
df["is_female"] = (df["Sex"] == "female").astype(int)
feats = ["Pclass", "is_female", "Age", "Fare", "family_size"]
df[feats + ["Survived"]].head()
Pclass is_female Age Fare family_size Survived
0 3 0 22.0 7.2500 2 0
1 1 1 38.0 71.2833 2 1
2 3 1 26.0 7.9250 1 1
3 1 1 35.0 53.1000 2 1
4 3 0 35.0 8.0500 1 0
df[feats].isna().sum()
Pclass           0
is_female        0
Age            177
Fare             0
family_size      0
dtype: int64

Why split? A model that marks its own homework#

Problem: a score on the training rows says nothing about new rows.

from sklearn.tree import DecisionTreeClassifier
X_all = df[feats].fillna({"Age": df["Age"].median()})
y_all = df["Survived"]
tree = DecisionTreeClassifier(random_state=42)
tree.fit(X_all, y_all)
print(round(tree.score(X_all, y_all), 3))
0.98

Quick check: is 98 percent accuracy believable for the Titanic?

print(tree.get_depth())
print(tree.get_n_leaves())
22
202

No. The tree memorised these rows. We need rows it has never seen.

  • Train set: the rows the model learns from
  • Test set: rows locked away until the final exam

Split with pandas: sample and drop#

Problem: deal the rows into two piles at random, with no overlap.

train = df.sample(frac=0.8, random_state=42)
test = df.drop(train.index)
print(train.shape, test.shape)
print(len(train.index.intersection(test.index)))
(713, 14) (178, 14)
0

Quick check: what happens if we leave out random_state?

a = df.sample(frac=0.8)
b = df.sample(frac=0.8)
print(a.index.equals(b.index))
c = df.sample(frac=0.8, random_state=42)
print(c.index.equals(train.index))
False
True

Common mistake: asking sample for more than 100 percent

df.sample(frac=80, random_state=42)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[8], line 1
----> 1 df.sample(frac=80, random_state=42)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\generic.py:5980, in NDFrame.sample(self, n, frac, replace, weights, random_state, axis, ignore_index)
   5977 # Process random_state argument
   5978 rs = common.random_state(random_state)
-> 5980 size = sample.process_sampling_size(n, frac, replace)
   5981 if size is None:
   5982     assert frac is not None

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\sample.py:106, in process_sampling_size(n, frac, replace)
    104 assert frac is not None  # for mypy
    105 if frac > 1 and not replace:
--> 106     raise ValueError(
    107         "Replace has to be set to `True` when "
    108         "upsampling the population `frac` > 1."
    109     )
    110 if frac < 0:
    111     raise ValueError(
    112         "A negative number of rows requested. Please provide `frac` >= 0."
    113     )

ValueError: Replace has to be set to `True` when upsampling the population `frac` > 1.
big = df.sample(frac=0.8, random_state=42)
print(len(big), round(len(big) / len(df), 2))
713 0.8

Your turn#

Make a 70/30 split with sample and drop. Compare survival rates.

tr70 = df.sample(frac=0.7, random_state=42)
te30 = df.drop(tr70.index)
print(len(tr70), len(te30))
print(round(tr70["Survived"].mean(), 3), round(te30["Survived"].mean(), 3))
624 267
0.391 0.367
import os
os.makedirs("output/pandas", exist_ok=True)
train.to_csv("output/pandas/l53_train.csv", index=False)
test.to_csv("output/pandas/l53_test.csv", index=False)
back = pd.read_csv("output/pandas/l53_test.csv")
print(back.shape)
(178, 14)

Split features and labels with train_test_split#

Problem: split X and y together, keeping rows lined up.

from sklearn.model_selection import train_test_split
X = df[feats]
y = df["Survived"]
X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.2, random_state=42)
print(X_tr.shape, X_te.shape, y_tr.shape, y_te.shape)
(712, 5) (179, 5) (712,) (179,)
print(X_tr.index[:5].tolist())
print(y_tr.index[:5].tolist())
[331, 733, 382, 704, 813]
[331, 733, 382, 704, 813]

Common mistake: unpacking four pieces into two names

train_part, test_part = train_test_split(X, y, random_state=42)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[14], line 1
----> 1 train_part, test_part = train_test_split(X, y, random_state=42)

ValueError: too many values to unpack (expected 2)
parts = train_test_split(X, y, random_state=42)
print(len(parts))
4

Quick check: is the survival rate the same in both piles?

for seed in range(4):
    _, _, a_tr, a_te = train_test_split(X, y, test_size=0.2,
                                        random_state=seed)
    print(seed, round(a_tr.mean(), 3), round(a_te.mean(), 3))
0 0.383 0.385
1 0.378 0.408
2 0.369 0.441
3 0.382 0.391
X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)
print(round(y.mean(), 3))
print(round(y_tr.mean(), 3), round(y_te.mean(), 3))
0.384
0.383 0.385
share = pd.DataFrame({"all": X["Pclass"].value_counts(normalize=True),
                      "test": X_te["Pclass"].value_counts(normalize=True)})
share.round(3)
all test
Pclass
3 0.551 0.559
1 0.242 0.251
2 0.207 0.190

Avoid leakage: split first, then fill and scale#

Problem: test rows must not influence anything the model learns.

print(df["Age"].median())
print(X_tr["Age"].median())
28.0
28.5

Rule: learn every number (medians, means, scales) from train only.

age_med = X_tr["Age"].median()
X_tr = X_tr.fillna({"Age": age_med})
X_te = X_te.fillna({"Age": age_med})
print(X_tr.isna().sum().sum(), X_te.isna().sum().sum())
0 0
from sklearn.preprocessing import StandardScaler
sc = StandardScaler().set_output(transform="pandas")
X_tr_s = sc.fit_transform(X_tr)
X_te_s = sc.transform(X_te)
print(X_tr_s.mean().abs().max().round(3))
print(X_te_s["Fare"].mean().round(3))
0.0
0.04
shared = test["Ticket"].isin(train["Ticket"])
print(shared.sum(), len(test))
52
 178

Your turn#

Fill Embarked gaps in train and test with train's most common port.

port = train["Embarked"].mode()[0]
train_e = train["Embarked"].fillna(port)
test_e = test["Embarked"].fillna(port)
print(port, train_e.isna().sum(), test_e.isna().sum())
S 0 0

Time series: split by date, never shuffle#

Problem: a forecast must be tested on the future, not the past.

flights = pd.read_csv("data/pandas/flights.csv")
flights["date"] = pd.to_datetime(flights["year"].astype(str) + "-"
                                 + flights["month"], format="%Y-%B")
flights[["date", "passengers"]].head(3)
date passengers
0 1949-01-01 112
1 1949-02-01 118
2 1949-03-01 132
f_tr, f_te = train_test_split(flights, test_size=0.2,
                              random_state=42)
early = f_te["date"] < f_tr["date"].max()
print(early.sum(), len(f_te))
29 29

Quick check: why is that a problem for forecasting?

cut = flights["date"] < "1959-01-01"
f_tr, f_te = flights[cut], flights[~cut]
print(f_tr["date"].max().date(), f_te["date"].min().date())
print(f_tr["passengers"].mean().round(1),
      f_te["passengers"].mean().round(1))
1958-12-01 1959-01-01
245.9 452.2
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(8, 3.5))
ax.plot(f_tr["date"], f_tr["passengers"], label="Train")
ax.plot(f_te["date"], f_te["passengers"], label="Test")
ax.set_title("Airline passengers: train before 1959, test after")
ax.set_xlabel("Month")
ax.set_ylabel("Passengers (thousands)")
ax.legend()
No description has been provided for this image

Your turn#

Use iloc to make the last 24 rows of flights the test set.

tr_i, te_i = flights.iloc[:-24], flights.iloc[-24:]
print(len(tr_i), len(te_i), te_i["date"].min().date())
120 24 1959-01-01

Validation sets and cross-validation#

Problem: tuning on the test set quietly turns it into training data.

X_rest, X_test, y_rest, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)
X_train, X_val, y_train, y_val = train_test_split(
    X_rest, y_rest, test_size=0.25, random_state=42, stratify=y_rest)
print(len(X_train), len(X_val), len(X_test))
534 178 179

Quick check: why 0.25 the second time, not 0.2?

print(round(0.25 * 0.8, 2))
0.2
from sklearn.model_selection import StratifiedKFold, cross_val_score
folds = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
small = DecisionTreeClassifier(max_depth=3, random_state=42)
scores = cross_val_score(small, X_tr, y_tr, cv=folds)
print(scores.round(3))
print(round(scores.mean(), 3))
[0.818 0.79  0.824 0.803 0.824]
0.812

Your turn#

Cross-validate a tree with max_depth=6. Is its average higher or lower?

deep = DecisionTreeClassifier(max_depth=6, random_state=42)
deep_scores = cross_val_score(deep, X_tr, y_tr, cv=folds)
print(deep_scores.round(3), round(deep_scores.mean(), 3))
[0.797 0.79  0.852 0.803 0.845] 0.817

Mini-project: catch an overfitting tree#

Plan: compare train and validation scores for depths 1 to 10, pick one, test once.

rows = []
for d in range(1, 11):
    m = DecisionTreeClassifier(max_depth=d, random_state=42)
    cv = cross_val_score(m, X_tr, y_tr, cv=folds).mean()
    fit = m.fit(X_tr, y_tr).score(X_tr, y_tr)
    rows.append({"depth": d, "train": fit, "cv": cv})
res = pd.DataFrame(rows).set_index("depth").round(3)
res
train cv
depth
1 0.789 0.789
2 0.805 0.791
3 0.829 0.812
4 0.834 0.817
5 0.865 0.815
6 0.882 0.817
7 0.890 0.805
8 0.909 0.802
9 0.926 0.806
10 0.938 0.798
ax = res.plot(marker="o", figsize=(8, 3.5))
ax.set_title("Decision tree: training vs cross-validated accuracy")
ax.set_xlabel("Maximum tree depth")
ax.set_ylabel("Accuracy")
No description has been provided for this image
best = res["cv"].idxmax()
gap = res.loc[10, "train"] - res.loc[10, "cv"]
print(best, round(gap, 3))
4 0.14
baseline = 1 - y_tr.mean()
print(round(baseline, 3))
0.617
final = DecisionTreeClassifier(max_depth=best, random_state=42)
final.fit(X_tr, y_tr)
test_acc = final.score(X_te, y_te)
print(f"Depth {best}: test accuracy {test_acc:.3f}")
Depth 4: test accuracy 0.777

Recap#

  • Never score a model on the rows it learned from
  • sample plus drop, or train_test_split, with a fixed random_state
  • stratify keeps class shares equal in every pile
  • Split first, then learn fills and scales from train only
  • Time series: cut by date; tune with validation or CV, test once

Challenges#

  1. Repeat the mini-project with a 70/30 split. Does the best depth change?
  2. Cut flights at 1958 instead. How do train and test means change?
  3. Use StratifiedKFold with 10 folds. How much do the scores vary?

What's next#

Lesson 54: Preparing Data for scikit-learn Models Bundle encoding, filling and scaling into one leak-proof pipeline.

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.