Mathew K AnalyticsLesson 22 · Pandas Full Course
Pandas apply() Explained: Run Your Own Function on Every Row or Column | Pandas Course #22
In this lesson you'll be able to: Run your own function on every value with Series.apply Apply functions down columns and across rows with axis Change every…
- CoursePandas Full Course
- Lesson22 of 65
- FormatJupyter notebook · 37 code cells
- Data3 datasets
What you'll learn
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
- salary_survey.csv preview4.9 MB
- titanic.csv preview58.9 KB
- penguins.csv preview13.2 KB
📓 Full notebook
Download .ipynbPandas 22: Applying Functions across Columns and Rows#
In this lesson you'll be able to:
- Run your own function on every value with
Series.apply - Apply functions down columns and across rows with
axis - Change every cell with
DataFrame.map - Choose between
agg,transformand plain vectorised code Data: the Titanic passenger list and the Palmer penguins
Warm-up: one cleaning step from last time#
In lesson 21 we cleaned the salary survey. Here's the country trick.
import pandas as pd
import numpy as np
raw = pd.read_csv("data/pandas/salary_survey.csv")
tidy = raw["country"].str.strip().str.lower()
print(raw["country"].nunique(), "->", tidy.nunique(), "spellings")
Run your own function on every value#
Series.apply calls your function once per value in a column.
df = pd.read_csv("data/pandas/titanic.csv")
name_len = df["Name"].apply(len)
df.assign(name_len=name_len)[["Name", "name_len"]].head(3)
def age_group(age):
if pd.isna(age):
return "unknown"
if age < 13:
return "child"
if age < 20:
return "teen"
return "adult" if age < 60 else "senior"
df["age_group"] = df["Age"].apply(age_group)
df["age_group"].value_counts()
df.groupby("age_group")["Survived"].mean().round(2)
def is_young(age, limit=13):
return age < limit
print(df["Age"].apply(is_young).sum(), "under 13")
print(df["Age"].apply(is_young, limit=18).sum(), "under 18")
Quick check: what would happen to missing ages without the
isnacheck?
def no_check(age):
return "child" if age < 13 else "adult"
missing = df.loc[df["Age"].isna(), "Age"]
missing.apply(no_check).value_counts()
Common mistake: a function that assumes every value is text.
df["Cabin"].apply(lambda c: c[0])
def deck(cabin):
return cabin[0] if isinstance(cabin, str) else "none"
df["deck"] = df["Cabin"].apply(deck)
df["deck"].value_counts().head(5)
vectorised = df["Cabin"].str[0].fillna("none")
print(vectorised.equals(df["deck"]))
Apply a function down every column#
DataFrame.apply passes each whole column to your function.
penguins = pd.read_csv("data/pandas/penguins.csv")
nums = penguins.select_dtypes("number")
print(penguins.shape)
nums.head(3)
nums.apply(lambda col: col.max() - col.min())
def summary(col):
return pd.Series({"min": col.min(), "mean": col.mean(),
"max": col.max()})
nums.apply(summary).round(1)
nums.agg(["min", "mean", "max"]).round(1)
Quick check: which measurement varies most relative to its average?
cv = nums.apply(lambda col: col.std() / col.mean() * 100)
cv.round(1).sort_values(ascending=False)
Your turn: count the values in each column that are above that column's mean.
nums.apply(lambda col: (col > col.mean()).sum())
Apply a function across each row#
With axis=1, your function gets one row and can combine columns.
def family_size(row):
return row["SibSp"] + row["Parch"] + 1
df["family"] = df.apply(family_size, axis=1)
df[["SibSp", "Parch", "family"]].head()
Common mistake: forgetting
axis=1.
df.apply(family_size)
fast = df["SibSp"] + df["Parch"] + 1
print(len(fast), "rows; identical:", fast.equals(df["family"]))
def ticket_type(row):
if row["Fare"] == 0:
return "free"
if row["Pclass"] == 1 and row["Fare"] > 100:
return "luxury"
return "standard"
df["ticket_type"] = df.apply(ticket_type, axis=1)
df.groupby("ticket_type")["Survived"].agg(["mean", "count"]).round(2)
def fare_info(row):
per_head = row["Fare"] / row["family"]
return pd.Series({"per_head": round(per_head, 2),
"pricey": per_head > 50})
df.apply(fare_info, axis=1).head(4)
Your turn: build a short description for each passenger, like "Braund, male, 22".
def describe(row):
surname = row["Name"].split(",")[0]
age = "?" if pd.isna(row["Age"]) else int(row["Age"])
return f"{surname}, {row['Sex']}, {age}"
df.apply(describe, axis=1).head(6)
parts = df["Name"].str.split(", ", n=1, expand=True)
parts.columns = ["surname", "rest"]
parts["title"] = parts["rest"].apply(lambda s: s.split(".")[0])
parts[["surname", "title"]].head(3)
Change every cell with DataFrame.map#
map on a DataFrame runs a function on each cell, one by one.
Common mistake: copying
applymapfrom older tutorials.
cols = ["bill_length_mm", "bill_depth_mm"]
penguins[cols].applymap(lambda x: f"{x:.0f} mm")
labels = penguins[cols].map(lambda x: f"{x:.0f} mm",
na_action="ignore")
labels.head(4)
text = df[["Name", "Ticket", "Cabin"]]
widths = text.map(lambda v: len(v) if isinstance(v, str) else 0)
widths.max()
Quick check: do we need
mapto turn millimetres into centimetres?
in_cm = penguins[cols] / 10
in_cm.head(4)
agg summarises, transform keeps the shape#
Both take functions; they differ in how many rows come back.
mass = penguins.groupby("species")["body_mass_g"]
mass.agg(["mean", "std", "count"]).round(1)
Quick check:
agggave 3 rows. How many willtransformgive?
z = mass.transform(lambda s: (s - s.mean()) / s.std())
print(len(z), "rows, same as", len(penguins), "penguins")
penguins.assign(z=z.round(2))[["species", "body_mass_g", "z"]].head()
penguins["mass_z"] = z
cols = ["species", "sex", "body_mass_g", "mass_z"]
penguins.nlargest(3, "mass_z")[cols].round(2)
pair = penguins.groupby(["species", "sex"])["body_mass_g"]
penguins["vs_pair"] = penguins["body_mass_g"] - pair.transform("mean")
penguins[["species", "sex", "body_mass_g", "vs_pair"]].round(0).head(4)
Your turn: give each penguin its flipper length minus its species median.
flip = penguins.groupby("species")["flipper_length_mm"]
penguins["flip_diff"] = (penguins["flipper_length_mm"]
- flip.transform("median"))
penguins.groupby("species")["flip_diff"].agg(["min", "max"])
heavy = penguins.groupby("species")["body_mass_g"].transform("max")
penguins["share_of_max"] = (penguins["body_mass_g"] / heavy).round(2)
penguins[["species", "body_mass_g", "share_of_max"]].head(3)
Mini-project: does family size change survival odds?#
Use our family column and a custom function to find out.
def family_kind(n):
if n == 1:
return "1 alone"
return "2-4 small" if n <= 4 else "5+ large"
df["family_kind"] = df["family"].apply(family_kind)
df.groupby("family_kind")["Survived"].agg(["mean", "count"]).round(2)
by_class = df.pivot_table(index="family_kind", columns="Pclass",
values="Survived", aggfunc="mean")
by_class.round(2)
by_sex = df.pivot_table(index="family_kind", columns="Sex",
values="Survived", aggfunc="mean")
by_sex.round(2)
pd.crosstab(df["family_kind"], df["Pclass"])
import matplotlib.pyplot as plt
rate = df.groupby("family")["Survived"].mean()
ax = rate.plot.bar(figsize=(8, 4), rot=0)
ax.set_title("Titanic survival rate by family size")
ax.set_xlabel("Family size (including the passenger)")
ax.set_ylabel("Share who survived")
plt.show()
Finding: small families of 2 to 4 survived most often. Travelling alone or in a family of 5 or more was far riskier.
Recap#
Series.apply(func)runs your function on every value- Handle missing values inside the function, or they slip through
DataFrame.apply: columns by default, rows withaxis=1DataFrame.mapchanges every cell (it replacedapplymap)aggshrinks groups,transformkeeps one value per row- Prefer vectorised code when it exists: it's simpler and faster
Challenges#
- Write
fare_band(fare)with 4 bands and compare survival by band. - For penguins, use
apply(axis=1)to compute bill length / depth. Then do it vectorised and check they match. - Use
transformto find each passenger's fare minus their class median.
What's next#
Lesson 23: Mapping, Replacing, and Applying Functions.
Swap values with dictionaries using map and replace.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.
Related lessons
26 minData Mining with Python: pandas & scikit-learn Crash Course
24 minInstalling Pandas and Setting Up Your Python Environment for Data Science Success
28 minIntroduction to Pandas DataFrame: Essential Operations and Attributes Explained
17 min