Lesson 54 · Pandas Full Course
Prepare Data for scikit-learn with Pandas: Pipelines Done Right | Pandas Course #54
In this lesson you'll be able to: Turn a DataFrame into the X and y that scikit-learn expects Fill missing values using training data only, with no leakage…
- CoursePandas Full Course
- Lesson54 of 65
- FormatJupyter notebook · 39 code cells
- Data1 dataset
What you'll learn
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
- titanic.csv preview58.9 KB
📓 Full notebook
Download .ipynbPandas 54: Preparing Data for scikit-learn Models#
In this lesson you'll be able to:
- Turn a DataFrame into the X and y that scikit-learn expects
- Fill missing values using training data only, with no leakage
- One-hot encode text columns and keep train and test aligned
- Scale numbers and wrap every step in a Pipeline
- Read a fitted model's coefficients back into pandas Dataset: the Titanic passenger list (data/pandas/titanic.csv)
Warm-up: split before anything else#
Lesson 53 split data into train and test sets. Same idea, same data.
import pandas as pd
df = pd.read_csv("data/pandas/titanic.csv")
print(df.shape)
df["Survived"].value_counts(normalize=True).round(3)
from sklearn.model_selection import train_test_split
train, test = train_test_split(df, test_size=0.2, random_state=42,
stratify=df["Survived"])
print(train.shape, test.shape)
Shape the table into X and y#
Problem: a model needs feature columns and one target column.
features = ["Pclass", "Sex", "Age", "SibSp", "Parch", "Fare",
"Embarked"]
X = df[features]
y = df["Survived"]
print(X.shape, y.shape)
Quick check: why leave out Name, Ticket and PassengerId?
df[["PassengerId", "Name", "Ticket"]].nunique()
X.dtypes
X.isna().sum()
Common mistake: fitting a model on the raw table
from sklearn.linear_model import LogisticRegression
LogisticRegression().fit(X, y)
X_num = X.select_dtypes(include="number")
print(X_num.columns.tolist())
print(X_num.isna().any().to_dict())
Fill gaps with training statistics only#
Problem: test rows must not influence anything the model learns.
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y)
print(X_train.shape, X_test.shape)
print(X_train["Age"].isna().sum(), X_test["Age"].isna().sum())
age_fill = X_train["Age"].median()
print(age_fill, df["Age"].median())
Quick check: the test rows have their own median. Why not use it?
print(X_test["Age"].median())
X_train = X_train.assign(Age=X_train["Age"].fillna(age_fill))
X_test = X_test.assign(Age=X_test["Age"].fillna(age_fill))
print(X_train["Age"].isna().sum(), X_test["Age"].isna().sum())
Your turn#
Fill the missing Embarked ports with the most common training port.
port_fill = X_train["Embarked"].mode()[0]
X_train = X_train.assign(Embarked=X_train["Embarked"]
.fillna(port_fill))
X_test = X_test.assign(Embarked=X_test["Embarked"]
.fillna(port_fill))
print(port_fill, X_train.isna().sum().sum())
Encode text columns as numbers#
Problem: Sex and Embarked are words, and models need numbers.
cats = ["Sex", "Embarked"]
X_train_enc = pd.get_dummies(X_train, columns=cats,
drop_first=True, dtype=int)
X_train_enc.head()
X_test_enc = pd.get_dummies(X_test, columns=cats,
drop_first=True, dtype=int)
print(list(X_train_enc.columns) == list(X_test_enc.columns))
Your turn#
Pclass is a number, but is it really a quantity? One-hot encode it too.
alt = pd.get_dummies(X_train, columns=cats + ["Pclass"],
drop_first=True, dtype=int)
print(X_train_enc.shape[1], alt.shape[1])
alt.filter(like="Pclass").head(3)
model = LogisticRegression(max_iter=1000)
model.fit(X_train_enc, y_train)
print(round(model.score(X_test_enc, y_test), 3))
Common mistake: encoding a few new rows on their own
new = X_test.head(3)
new_enc = pd.get_dummies(new, columns=cats, drop_first=True,
dtype=int)
model.predict(new_enc)
print(new_enc.columns.tolist())
full = pd.get_dummies(new, columns=cats, dtype=int)
fixed = full.reindex(columns=X_train_enc.columns, fill_value=0)
model.predict(fixed)
Scale numbers onto a common footing#
Problem: Fare runs into hundreds while Parch stays below 7.
num = ["Age", "Fare", "SibSp", "Parch", "Pclass"]
X_train_enc[num].describe().loc[["mean", "std", "max"]].round(1)
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler().set_output(transform="pandas")
train_sc = scaler.fit_transform(X_train_enc[num])
test_sc = scaler.transform(X_test_enc[num])
train_sc.head().round(2)
train_all = X_train_enc.copy()
train_all[num] = train_sc
test_all = X_test_enc.copy()
test_all[num] = test_sc
scaled_model = LogisticRegression().fit(train_all, y_train)
print(round(scaled_model.score(test_all, y_test), 3))
Quick check: the training means are now 0. Will the test means be 0 too?
means = pd.DataFrame({"train": train_sc.mean(),
"test": test_sc.mean()})
means.round(2)
Common mistake: transforming with a scaler that was never fitted
fresh = StandardScaler()
fresh.transform(X_test_enc[num])
fresh.fit(X_train_enc[num])
print(fresh.mean_.round(2))
Wrap every step in a Pipeline#
Problem: many manual steps are easy to forget or run out of order.
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder
num_steps = make_pipeline(SimpleImputer(strategy="median"),
StandardScaler())
cat_steps = make_pipeline(SimpleImputer(strategy="most_frequent"),
OneHotEncoder(handle_unknown="ignore"))
prep = ColumnTransformer([("num", num_steps, num),
("cat", cat_steps, cats)])
pipe = make_pipeline(prep, LogisticRegression(max_iter=1000))
raw_train, raw_test = train_test_split(
X, test_size=0.2, random_state=42, stratify=y)
pipe.fit(raw_train, y_train)
print(round(pipe.score(raw_test, y_test), 3))
pipe.predict(X_test.head(3))
prepared = pipe[0].transform(raw_test.head(3))
cols = pipe[0].get_feature_names_out()
pd.DataFrame(prepared, columns=cols, index=raw_test.index[:3]).round(2).T
Common mistake: a column name with a typo inside the pipeline
bad = ColumnTransformer([("num", num_steps, ["age", "Fare"])])
bad.fit(raw_train)
good = ColumnTransformer([("num", num_steps, ["Age", "Fare"])])
print(good.fit_transform(raw_train).shape)
Your turn#
Score the pipeline with 5-fold cross-validation on the training rows.
from sklearn.model_selection import cross_val_score
cv = cross_val_score(pipe, raw_train, y_train, cv=5)
print(cv.round(3), round(cv.mean(), 3))
Mini-project: what drives the model's predictions?#
Read the fitted coefficients back into a pandas Series and rank them.
names = pipe[0].get_feature_names_out()
coefs = pipe[-1].coef_[0]
weights = pd.Series(coefs, index=names).round(2)
weights.sort_values()
top = weights.abs().sort_values(ascending=False).head(3)
print(top)
sample = X_test.head(5)
probs = pipe.predict_proba(sample)[:, 1].round(2)
sample.assign(chance=probs, actual=y_test.head(5))
import matplotlib.pyplot as plt
ax = weights.sort_values().plot(kind="barh", figsize=(7, 4))
ax.set_title("What pushes predicted Titanic survival up or down")
ax.set_xlabel("Coefficient (scaled features)")
ax.set_ylabel("Feature")
plt.show()
right = pipe.predict(raw_test) == y_test
check = pd.DataFrame({"Sex": raw_test["Sex"], "right": right})
check.groupby("Sex")["right"].agg(["size", "mean"]).round(3)
base = y_test.value_counts(normalize=True).max()
acc = pipe.score(raw_test, y_test)
print(f"Baseline {base:.0%}, model {acc:.0%}")
Recap#
- Split X and y first; learn every statistic from training rows
- Fill gaps with training medians and modes, never test values
- Encode text with get_dummies; reindex new rows to train columns
- Fit scalers on train, only transform test
- A Pipeline with ColumnTransformer does it all in one object
Challenges#
- Add family_size = SibSp + Parch + 1 and re-score the pipeline.
- Swap StandardScaler for MinMaxScaler. Does accuracy change?
- Drop Fare from the features and compare cross-validation scores.
What's next#
Lesson 55: Clustering and Outlier Detection Group similar rows with no target column, and flag the odd ones out.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.



