Mathew K AnalyticsLesson 23 · Pandas Full Course
Pandas map vs replace vs apply: Which One to Use and When | Pandas Course #23
In this lesson you'll be able to: Look up new values for a column with map Swap selected values and keep the rest with replace Run your own rules per value,…
- CoursePandas Full Course
- Lesson23 of 65
- FormatJupyter notebook · 37 code cells
- Data1 dataset
What you'll learn
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
📓 Full notebook
Download .ipynbPandas 23: Mapping, Replacing, and Applying Functions#
In this lesson you'll be able to:
- Look up new values for a column with map
- Swap selected values and keep the rest with replace
- Run your own rules per value, per column or per row with apply
- Put categories in a sensible order with a mapping
- Pick the right tool for each clean-up job Dataset: restaurant bills and tips (data/pandas/tips.csv)
Warm-up: apply from lesson 22#
Lesson 22 ran functions across columns and rows. One quick recall.
import pandas as pd
df = pd.read_csv("data/pandas/tips.csv")
print(df.shape)
df.head()
def spread(col):
return col.max() - col.min()
df[["total_bill", "tip"]].apply(spread)
Look up new values with map#
Problem: turn short codes like Thur into readable labels.
day_names = {
"Thur": "Thursday", "Fri": "Friday",
"Sat": "Saturday", "Sun": "Sunday",
}
df["day_name"] = df["day"].map(day_names)
df[["day", "day_name"]].drop_duplicates()
weekend = {
"Thur": False, "Fri": False,
"Sat": True, "Sun": True,
}
df["weekend"] = df["day"].map(weekend)
df["weekend"].value_counts()
Quick check: what does map do with a code the dictionary doesn't know?
short = {"Sat": "Saturday", "Sun": "Sunday"}
partial = df["day"].map(short)
print(partial.isna().sum())
print(df.loc[partial.isna(), "day"].unique().tolist())
filled = partial.fillna(df["day"])
filled.value_counts()
Common mistake: calling map on a whole DataFrame with a dictionary
df[["day", "time"]].map(day_names)
df["day"].map(day_names).value_counts()
Map with a function or a Series#
Problem: some lookups need a rule, or numbers from another table.
def party(n):
if n == 1:
return "Solo"
if n == 2:
return "Couple"
return "Group"
df["party"] = df["size"].map(party)
df["party"].value_counts()
avg_tip = df.groupby("day")["tip"].mean()
print(avg_tip.round(2))
df["day_avg"] = df["day"].map(avg_tip)
df["vs_day"] = df["tip"] - df["day_avg"]
df[["day", "tip", "day_avg", "vs_day"]].head(3).round(2)
Your turn#
Map each bill's time (Lunch or Dinner) to that meal's average bill.
avg_bill = df.groupby("time")["total_bill"].mean()
df["meal_avg"] = df["time"].map(avg_bill)
df[["time", "meal_avg"]].drop_duplicates().round(2)
Swap values with replace#
Problem: tidy a few labels and leave everything else untouched.
Quick check: map and replace get the same one-entry dictionary. What differs?
fix = {"Thur": "Thu"}
print(df["day"].map(fix).unique().tolist())
print(df["day"].replace(fix).unique().tolist())
labels = {
"smoker": {"Yes": "Smoker", "No": "Non-smoker"},
"sex": {"Female": "F", "Male": "M"},
}
tidy = df.replace(labels)
tidy[["sex", "smoker", "day"]].head(3)
df["day"].replace(["Sat", "Sun"], "Weekend").value_counts()
Common mistake: two lists of different lengths
df["day"].replace(["Thur", "Fri"], ["Thursday"])
old, new = ["Thur", "Fri"], ["Thursday", "Friday"]
df["day"].replace(old, new).value_counts()
Quick check: to turn Yes and No into 1 and 0, should you use replace or map?
flags = {"Yes": 1, "No": 0}
print(df["smoker"].replace(flags).dtype)
print(df["smoker"].map(flags).dtype)
df["smoker_flag"] = df["smoker"].map(flags)
print(round(df["smoker_flag"].mean(), 2))
Run custom logic with apply#
Problem: a rule needs several steps, or reads several columns.
df["tip_pct"] = (df["tip"] / df["total_bill"] * 100).round(1)
df["tip_pct"].describe().round(1)
def tip_band(pct):
if pct < 12:
return "Low"
if pct < 20:
return "Standard"
return "Generous"
df["tip_band"] = df["tip_pct"].apply(tip_band)
df["tip_band"].value_counts()
def at_least(pct, limit):
return pct >= limit
big = df["tip_pct"].apply(at_least, limit=25)
print(big.sum())
def profile(col):
return pd.Series({
"low": col.min(),
"typical": col.median(),
"high": col.max(),
})
df[["total_bill", "tip", "tip_pct"]].apply(profile).round(2)
def service_charge(row):
if row["size"] >= 6:
return round(row["total_bill"] * 0.18, 2)
return 0.0
df["charge"] = df.apply(service_charge, axis=1)
df.loc[df["charge"] > 0, ["size", "total_bill", "tip", "charge"]]
Common mistake: forgetting axis=1 for a row rule
df.apply(service_charge)
print(round(df.apply(service_charge, axis=1).sum(), 2))
Your turn#
Use apply with axis=1 to work out the bill per person. Show the top 3.
def per_person(row):
return row["total_bill"] / row["size"]
df["per_head"] = df.apply(per_person, axis=1).round(2)
df.nlargest(3, "per_head")[["total_bill", "size", "per_head"]]
fast = (df["total_bill"] / df["size"]).round(2)
print(fast.equals(df["per_head"]))
Put categories in order with a mapping#
Problem: groupby sorts days alphabetically, not by the calendar.
by_day = df.groupby("day")["tip_pct"].mean().round(2)
by_day
Quick check: why does Thursday come last, and how could map fix it?
order = {"Thur": 4, "Fri": 5, "Sat": 6, "Sun": 7}
def calendar(idx):
return idx.map(order)
week = by_day.sort_index(key=calendar)
week.rename(index=day_names)
| Tool | Use it when |
|---|---|
| map | every value gets a new value (dict, Series, function) |
| replace | only some values change, the rest stay |
| apply | the rule needs steps, or a whole row |
Your turn#
Sort the average bill by time, Lunch first, with readable labels.
meals = df.groupby("time")["total_bill"].mean().round(2)
meal_order = {"Lunch": 1, "Dinner": 2}
def by_clock(idx):
return idx.map(meal_order)
meals = meals.sort_index(key=by_clock)
meals.rename(index={"Lunch": "Midday", "Dinner": "Evening"})
Mini-project: do bigger parties tip a smaller share?#
Plan: band the party sizes, summarise tips, check the bands, conclude.
bands = {
1: "1-2", 2: "1-2",
3: "3-4", 4: "3-4",
5: "5-6", 6: "5-6",
}
df["band"] = df["size"].map(bands)
print(df["band"].isna().sum())
summary = df.groupby("band").agg(
bills=("tip_pct", "size"),
mean_pct=("tip_pct", "mean"),
median_pct=("tip_pct", "median"),
)
summary.round(2)
mix = df.groupby("band")["tip_band"].value_counts(normalize=True)
mix.unstack().round(2)
by_meal = df.groupby(["time", "band"])["tip_pct"].median()
by_meal.unstack().round(1)
import matplotlib.pyplot as plt
ax = summary["median_pct"].plot(kind="bar", rot=0, figsize=(7, 4))
ax.set_title("Median tip by party size")
ax.set_xlabel("Diners at the table")
ax.set_ylabel("Tip as % of bill")
plt.show()
small = summary.loc["1-2", "median_pct"]
large = summary.loc["5-6", "median_pct"]
print(f"1-2 diners: {small:.1f}%, 5-6 diners: {large:.1f}%")
Recap#
- map gives every value a new value: dict, Series or function
- Values missing from a map dictionary become NaN
- replace swaps only what you name; lists must match in length
- apply runs your function per value, per column, or per row with axis=1
- Mapping to positions puts categories in a real-world order
Challenges#
- Map sex to F and M, then find the average tip % for each.
- Use replace to merge Thur and Fri into Weekday, and count the bills.
- Write a row rule with apply that flags dinners with a tip under 10%.
What's next#
Lesson 24: Lambda Functions and Vectorized Operations Write rules in one line, and make them run far faster.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.
Related lessons
26 minData Mining with Python: pandas & scikit-learn Crash Course
14 minUnderstanding Map, Replace, and Apply Functions for Effective Data Transformation in Pandas
57 minPandas Zero to Hero: The Complete Beginner-to-Advanced Course
18 min