Lesson 52 · Pandas Full Course
Feature Engineering with Pandas: Build Features That Boost Accuracy | Pandas Course #52
In this lesson you'll be able to: Build new features by combining columns with arithmetic Pull features out of text, and bin numbers into bands Create…
- CoursePandas Full Course
- Lesson52 of 65
- FormatJupyter notebook · 43 code cells
- Data2 datasets
What you'll learn
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
- titanic.csv preview58.9 KB
- taxis.csv preview849.0 KB
📓 Full notebook
Download .ipynbPandas 52: Feature Engineering with Pandas#
In this lesson you'll be able to:
- Build new features by combining columns with arithmetic
- Pull features out of text, and bin numbers into bands
- Create group-aware features with groupby and transform
- Extract hour, weekday and duration features from dates
- Show that good features make a model more accurate Data: the Titanic passenger list, plus New York taxi trips for dates
Warm-up: scaling, from lesson 51#
Lesson 51 scaled numbers. Here it is on Titanic fares.
import pandas as pd
import numpy as np
df = pd.read_csv("data/pandas/titanic.csv")
print(df.shape)
df[["Name", "SibSp", "Parch", "Ticket", "Fare", "Cabin"]].head(3)
fare_log = np.log1p(df["Fare"])
fare_z = (fare_log - fare_log.mean()) / fare_log.std()
print(round(df["Fare"].skew(), 2), round(fare_z.skew(), 2))
Combine columns with arithmetic#
Problem: the useful fact is spread across several columns.
df["family_size"] = df["SibSp"] + df["Parch"] + 1
df[["SibSp", "Parch", "family_size"]].head()
df["is_alone"] = (df["family_size"] == 1).astype(int)
df.groupby("is_alone")["Survived"].agg(["size", "mean"]).round(3)
by_fam = df.groupby("family_size")["Survived"]
by_fam.agg(["size", "mean"]).round(2)
Quick check: is the Fare column per person or per ticket?
sage = df[df["Name"].str.startswith("Sage,")]
sage[["Name", "Ticket", "Fare"]]
df["ticket_size"] = df.groupby("Ticket")["Ticket"].transform("size")
df["fare_pp"] = df["Fare"] / df["ticket_size"]
df.loc[sage.index, ["Fare", "ticket_size", "fare_pp"]].round(2)
Your turn#
Compare median Fare and median fare_pp for each class.
cols = ["Fare", "fare_pp"]
df.groupby("Pclass")[cols].median().round(2)
Pull features out of text#
Problem: names and cabin codes hold facts a model can't read as text.
Common mistake: a regex with no capture group
df["Name"].str.extract(r",\s*[^.]+\.")
df["title"] = df["Name"].str.extract(r",\s*([^.]+)\.")
df["title"].value_counts().head(8)
common = ["Mr", "Mrs", "Miss", "Master"]
df["title"] = df["title"].where(df["title"].isin(common), "Rare")
df.groupby("title")["Survived"].agg(["size", "mean"]).round(3)
df["deck"] = df["Cabin"].str[0]
df["has_cabin"] = df["Cabin"].notna().astype(int)
df.groupby("has_cabin")["Survived"].agg(["size", "mean"]).round(3)
Quick check: which deck letter is most common among recorded cabins?
df["deck"].value_counts().head(4)
by_deck = df.groupby("deck")["Survived"].agg(["size", "mean"])
by_deck[by_deck["size"] >= 10].round(2)
df["surname"] = df["Name"].str.split(",").str[0]
print(df["surname"].nunique())
df["surname"].value_counts().head(3)
Bin numbers into bands#
Problem: the effect of age is not a straight line.
Common mistake: one label too many for pd.cut
edges = [0, 12, 18, 60, 80]
names = ["Child", "Teen", "Adult", "Senior", "Elder"]
pd.cut(df["Age"], bins=edges, labels=names)
names = ["Child", "Teen", "Adult", "Senior"]
df["age_band"] = pd.cut(df["Age"], bins=edges, labels=names)
df["age_band"].value_counts(dropna=False)
by_band = df.groupby("age_band", observed=True)["Survived"]
by_band.agg(["size", "mean"]).round(3)
df["fare_q"] = pd.qcut(df["fare_pp"], q=4, labels=["Q1", "Q2",
"Q3", "Q4"])
df.groupby("fare_q", observed=True)["Survived"].mean().round(3)
Your turn#
Bin family_size into Alone (1), Small (2-4) and Large (5+). Compare survival.
fam_names = ["Alone", "Small", "Large"]
df["family_band"] = pd.cut(df["family_size"], bins=[0, 1, 4, 11],
labels=fam_names)
df.groupby("family_band", observed=True)["Survived"].mean().round(3)
Group-aware features with transform#
Problem: a value means more when compared with similar rows.
df["age_missing"] = df["Age"].isna().astype(int)
title_med = df.groupby("title")["Age"].transform("median")
df["age_filled"] = df["Age"].fillna(title_med)
df.groupby("title")["age_filled"].median()
Quick check: why fill by title rather than one overall median?
print(df["Age"].median())
print(df.loc[df["title"] == "Master", "Age"].max())
class_fare = df.groupby("Pclass")["fare_pp"].transform("median")
df["fare_vs_class"] = (df["fare_pp"] / class_fare).round(2)
df[["Pclass", "fare_pp", "fare_vs_class"]].head()
grp = df.groupby("Pclass")["fare_pp"]
df["fare_rank"] = grp.rank(pct=True).round(2)
df.groupby("Pclass")["fare_rank"].agg(["min", "max"])
Your turn#
Add 'age_vs_title': each age_filled minus its title's mean age_filled.
title_mean = df.groupby("title")["age_filled"].transform("mean")
df["age_vs_title"] = df["age_filled"] - title_mean
df[["title", "age_filled", "age_vs_title"]].head().round(1)
Features from dates and times#
Problem: a timestamp hides the hour, weekday and duration.
taxis = pd.read_csv("data/pandas/taxis.csv")
taxis[["pickup", "dropoff", "fare", "tip", "payment"]].head(3)
Common mistake: using .dt on a column that is still text
taxis["pickup"].dt.hour
for col in ["pickup", "dropoff"]:
taxis[col] = pd.to_datetime(taxis[col])
taxis["hour"] = taxis["pickup"].dt.hour
taxis["weekday"] = taxis["pickup"].dt.day_name()
taxis["weekend"] = taxis["pickup"].dt.dayofweek >= 5
taxis[["pickup", "hour", "weekday", "weekend"]].head(3)
trip = taxis["dropoff"] - taxis["pickup"]
taxis["minutes"] = trip.dt.total_seconds() / 60
taxis["tip_pct"] = taxis["tip"] / taxis["fare"] * 100
taxis[["minutes", "tip_pct"]].describe().round(1)
per_hour = taxis["hour"].value_counts().sort_index()
ax = per_hour.plot(kind="bar", figsize=(9, 3.5), rot=0)
ax.set_title("Taxi pickups by hour of day")
ax.set_xlabel("Hour of pickup")
ax.set_ylabel("Number of trips")
taxis["weekday"].value_counts()
taxis["rush"] = taxis["hour"].between(16, 19)
taxis.groupby("rush")["minutes"].median().round(1)
Quick check: do cash riders really tip less?
taxis.groupby("payment")["tip_pct"].agg(["size", "median"])
card = taxis[taxis["payment"] == "credit card"]
card.groupby("weekend")["tip_pct"].median().round(1)
Make the features model-ready#
Problem: models need numbers only, and no peeking at the answer.
feats = ["Pclass", "Sex", "title", "family_size", "is_alone",
"fare_pp", "has_cabin", "age_filled", "age_missing"]
Xf = pd.get_dummies(df[feats], drop_first=True, dtype=int)
print(Xf.shape)
Xf.dtypes.value_counts()
df["class_sex"] = df["Pclass"].astype(str) + "_" + df["Sex"]
df.groupby("class_sex")["Survived"].mean().round(2)
Your turn#
Make an is_child flag (age_filled under 13). Survival by is_child and Sex?
df["is_child"] = (df["age_filled"] < 13).astype(int)
kids = df.groupby(["is_child", "Sex"])["Survived"].mean()
kids.unstack().round(2)
Leakage: never build a feature from the answer, like Survived.
Quick check: which single feature moves most with Survived?
corr = Xf.corrwith(df["Survived"])
corr.sort_values(key=abs, ascending=False).round(3).head(6)
Mini-project: do engineered features beat the raw columns?#
Plan: same model, raw features vs engineered features, scored 5 times.
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
model = LogisticRegression(max_iter=1000)
raw = pd.get_dummies(df[["Pclass", "Sex", "Age", "Fare"]],
drop_first=True, dtype=int)
raw["Age"] = raw["Age"].fillna(raw["Age"].median())
y = df["Survived"]
raw_acc = cross_val_score(model, raw, y, cv=5).mean()
eng_acc = cross_val_score(model, Xf, y, cv=5).mean()
print(round(raw_acc, 3), round(eng_acc, 3))
no_title = Xf.drop(columns=Xf.filter(like="title_").columns)
no_title_acc = cross_val_score(model, no_title, y, cv=5).mean()
print(round(no_title_acc, 3))
import matplotlib.pyplot as plt
accs = pd.Series({"Raw columns": raw_acc, "Engineered": eng_acc})
ax = accs.plot(kind="bar", rot=0, figsize=(6, 3.5), ylim=(0.7, 0.85))
ax.set_title("Titanic survival model accuracy")
ax.set_xlabel("Feature set")
ax.set_ylabel("Mean accuracy, 5 folds")
extra = (eng_acc - raw_acc) * len(df)
print(f"About {extra:.0f} more of {len(df)} passengers called right")
Recap#
- Arithmetic combines columns: family size, fare per person
- str.extract needs a capture group; pd.cut needs one label per gap
- transform gives each row its group's figure
- Convert to datetime, then use .dt for hours, weekdays, durations
- Never build features from the answer
Challenges#
- Add a 'surname_size' feature: passengers sharing each surname.
- Use pd.qcut to split age_filled into 5 equal bands; compare survival.
- For taxis, find the median trip minutes for each pickup hour.
What's next#
Lesson 53: Splitting Data into Train-Test Sets Test models fairly on data they have never seen.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.



