Lesson 53 · Pandas Full Course
Train-Test Split in Pandas and scikit-learn (Avoid Data Leakage) | Pandas Course #53
In this lesson you'll be able to: Explain why a model must be tested on rows it never saw Split a DataFrame with pandas sample and with traintestsplit Keep…
- CoursePandas Full Course
- Lesson53 of 65
- FormatJupyter notebook · 36 code cells
- Data2 datasets
What you'll learn
- Warm-up: features, from lesson 52
- Why split? A model that marks its own homework
- Split with pandas: sample and drop
- Split features and labels with traintestsplit
- Avoid leakage: split first, then fill and scale
- Time series: split by date, never shuffle
- Validation sets and cross-validation
- Mini-project: catch an overfitting tree
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
- titanic.csv preview58.9 KB
- flights.csv preview2.3 KB
📓 Full notebook
Download .ipynbPandas 53: Splitting Data into Train-Test Sets#
In this lesson you'll be able to:
- Explain why a model must be tested on rows it never saw
- Split a DataFrame with pandas sample and with train_test_split
- Keep class balance with stratify, and avoid data leakage
- Split time series by date, and add a validation step
- Use the split to catch an overfitting model Data: the Titanic passenger list, plus monthly airline passengers
Warm-up: features, from lesson 52#
Lesson 52 engineered new features. We rebuild two of them.
import pandas as pd
df = pd.read_csv("data/pandas/titanic.csv")
df["family_size"] = df["SibSp"] + df["Parch"] + 1
df["is_female"] = (df["Sex"] == "female").astype(int)
feats = ["Pclass", "is_female", "Age", "Fare", "family_size"]
df[feats + ["Survived"]].head()
df[feats].isna().sum()
Why split? A model that marks its own homework#
Problem: a score on the training rows says nothing about new rows.
from sklearn.tree import DecisionTreeClassifier
X_all = df[feats].fillna({"Age": df["Age"].median()})
y_all = df["Survived"]
tree = DecisionTreeClassifier(random_state=42)
tree.fit(X_all, y_all)
print(round(tree.score(X_all, y_all), 3))
Quick check: is 98 percent accuracy believable for the Titanic?
print(tree.get_depth())
print(tree.get_n_leaves())
No. The tree memorised these rows. We need rows it has never seen.
- Train set: the rows the model learns from
- Test set: rows locked away until the final exam
Split with pandas: sample and drop#
Problem: deal the rows into two piles at random, with no overlap.
train = df.sample(frac=0.8, random_state=42)
test = df.drop(train.index)
print(train.shape, test.shape)
print(len(train.index.intersection(test.index)))
Quick check: what happens if we leave out random_state?
a = df.sample(frac=0.8)
b = df.sample(frac=0.8)
print(a.index.equals(b.index))
c = df.sample(frac=0.8, random_state=42)
print(c.index.equals(train.index))
Common mistake: asking sample for more than 100 percent
df.sample(frac=80, random_state=42)
big = df.sample(frac=0.8, random_state=42)
print(len(big), round(len(big) / len(df), 2))
Your turn#
Make a 70/30 split with sample and drop. Compare survival rates.
tr70 = df.sample(frac=0.7, random_state=42)
te30 = df.drop(tr70.index)
print(len(tr70), len(te30))
print(round(tr70["Survived"].mean(), 3), round(te30["Survived"].mean(), 3))
import os
os.makedirs("output/pandas", exist_ok=True)
train.to_csv("output/pandas/l53_train.csv", index=False)
test.to_csv("output/pandas/l53_test.csv", index=False)
back = pd.read_csv("output/pandas/l53_test.csv")
print(back.shape)
Split features and labels with train_test_split#
Problem: split X and y together, keeping rows lined up.
from sklearn.model_selection import train_test_split
X = df[feats]
y = df["Survived"]
X_tr, X_te, y_tr, y_te = train_test_split(
X, y, test_size=0.2, random_state=42)
print(X_tr.shape, X_te.shape, y_tr.shape, y_te.shape)
print(X_tr.index[:5].tolist())
print(y_tr.index[:5].tolist())
Common mistake: unpacking four pieces into two names
train_part, test_part = train_test_split(X, y, random_state=42)
parts = train_test_split(X, y, random_state=42)
print(len(parts))
Quick check: is the survival rate the same in both piles?
for seed in range(4):
_, _, a_tr, a_te = train_test_split(X, y, test_size=0.2,
random_state=seed)
print(seed, round(a_tr.mean(), 3), round(a_te.mean(), 3))
X_tr, X_te, y_tr, y_te = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y)
print(round(y.mean(), 3))
print(round(y_tr.mean(), 3), round(y_te.mean(), 3))
share = pd.DataFrame({"all": X["Pclass"].value_counts(normalize=True),
"test": X_te["Pclass"].value_counts(normalize=True)})
share.round(3)
Avoid leakage: split first, then fill and scale#
Problem: test rows must not influence anything the model learns.
print(df["Age"].median())
print(X_tr["Age"].median())
Rule: learn every number (medians, means, scales) from train only.
age_med = X_tr["Age"].median()
X_tr = X_tr.fillna({"Age": age_med})
X_te = X_te.fillna({"Age": age_med})
print(X_tr.isna().sum().sum(), X_te.isna().sum().sum())
from sklearn.preprocessing import StandardScaler
sc = StandardScaler().set_output(transform="pandas")
X_tr_s = sc.fit_transform(X_tr)
X_te_s = sc.transform(X_te)
print(X_tr_s.mean().abs().max().round(3))
print(X_te_s["Fare"].mean().round(3))
shared = test["Ticket"].isin(train["Ticket"])
print(shared.sum(), len(test))
Your turn#
Fill Embarked gaps in train and test with train's most common port.
port = train["Embarked"].mode()[0]
train_e = train["Embarked"].fillna(port)
test_e = test["Embarked"].fillna(port)
print(port, train_e.isna().sum(), test_e.isna().sum())
Time series: split by date, never shuffle#
Problem: a forecast must be tested on the future, not the past.
flights = pd.read_csv("data/pandas/flights.csv")
flights["date"] = pd.to_datetime(flights["year"].astype(str) + "-"
+ flights["month"], format="%Y-%B")
flights[["date", "passengers"]].head(3)
f_tr, f_te = train_test_split(flights, test_size=0.2,
random_state=42)
early = f_te["date"] < f_tr["date"].max()
print(early.sum(), len(f_te))
Quick check: why is that a problem for forecasting?
cut = flights["date"] < "1959-01-01"
f_tr, f_te = flights[cut], flights[~cut]
print(f_tr["date"].max().date(), f_te["date"].min().date())
print(f_tr["passengers"].mean().round(1),
f_te["passengers"].mean().round(1))
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(8, 3.5))
ax.plot(f_tr["date"], f_tr["passengers"], label="Train")
ax.plot(f_te["date"], f_te["passengers"], label="Test")
ax.set_title("Airline passengers: train before 1959, test after")
ax.set_xlabel("Month")
ax.set_ylabel("Passengers (thousands)")
ax.legend()
Your turn#
Use iloc to make the last 24 rows of flights the test set.
tr_i, te_i = flights.iloc[:-24], flights.iloc[-24:]
print(len(tr_i), len(te_i), te_i["date"].min().date())
Validation sets and cross-validation#
Problem: tuning on the test set quietly turns it into training data.
X_rest, X_test, y_rest, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y)
X_train, X_val, y_train, y_val = train_test_split(
X_rest, y_rest, test_size=0.25, random_state=42, stratify=y_rest)
print(len(X_train), len(X_val), len(X_test))
Quick check: why 0.25 the second time, not 0.2?
print(round(0.25 * 0.8, 2))
from sklearn.model_selection import StratifiedKFold, cross_val_score
folds = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
small = DecisionTreeClassifier(max_depth=3, random_state=42)
scores = cross_val_score(small, X_tr, y_tr, cv=folds)
print(scores.round(3))
print(round(scores.mean(), 3))
Your turn#
Cross-validate a tree with max_depth=6. Is its average higher or lower?
deep = DecisionTreeClassifier(max_depth=6, random_state=42)
deep_scores = cross_val_score(deep, X_tr, y_tr, cv=folds)
print(deep_scores.round(3), round(deep_scores.mean(), 3))
Mini-project: catch an overfitting tree#
Plan: compare train and validation scores for depths 1 to 10, pick one, test once.
rows = []
for d in range(1, 11):
m = DecisionTreeClassifier(max_depth=d, random_state=42)
cv = cross_val_score(m, X_tr, y_tr, cv=folds).mean()
fit = m.fit(X_tr, y_tr).score(X_tr, y_tr)
rows.append({"depth": d, "train": fit, "cv": cv})
res = pd.DataFrame(rows).set_index("depth").round(3)
res
ax = res.plot(marker="o", figsize=(8, 3.5))
ax.set_title("Decision tree: training vs cross-validated accuracy")
ax.set_xlabel("Maximum tree depth")
ax.set_ylabel("Accuracy")
best = res["cv"].idxmax()
gap = res.loc[10, "train"] - res.loc[10, "cv"]
print(best, round(gap, 3))
baseline = 1 - y_tr.mean()
print(round(baseline, 3))
final = DecisionTreeClassifier(max_depth=best, random_state=42)
final.fit(X_tr, y_tr)
test_acc = final.score(X_te, y_te)
print(f"Depth {best}: test accuracy {test_acc:.3f}")
Recap#
- Never score a model on the rows it learned from
- sample plus drop, or train_test_split, with a fixed random_state
- stratify keeps class shares equal in every pile
- Split first, then learn fills and scales from train only
- Time series: cut by date; tune with validation or CV, test once
Challenges#
- Repeat the mini-project with a 70/30 split. Does the best depth change?
- Cut flights at 1958 instead. How do train and test means change?
- Use StratifiedKFold with 10 folds. How much do the scores vary?
What's next#
Lesson 54: Preparing Data for scikit-learn Models Bundle encoding, filling and scaling into one leak-proof pipeline.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.



