Mathew K AnalyticsLesson 24 · Pandas Full Course
Lambda and Vectorized Operations in Pandas: Faster, Cleaner Code | Pandas Course #24
In this lesson you'll be able to: Write one-line rules with lambda in apply, map, agg and assign Do maths and comparisons on whole columns at once Build…
- CoursePandas Full Course
- Lesson24 of 65
- FormatJupyter notebook · 42 code cells
- Data1 dataset
What you'll learn
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
- diamonds.csv preview2.6 MB
📓 Full notebook
Download .ipynbPandas 24: Lambda Functions and Vectorized Operations#
In this lesson you'll be able to:
- Write one-line rules with lambda in apply, map, agg and assign
- Do maths and comparisons on whole columns at once
- Build conditional columns with np.where and np.select
- Measure why vectorised code beats apply for speed
- Handle zeros and missing values safely in column maths Dataset: 53,940 diamonds with prices (data/pandas/diamonds.csv)
Warm-up: map from lesson 23#
Lesson 23 mapped labels to new values. Same move, new data.
import pandas as pd
import numpy as np
df = pd.read_csv("data/pandas/diamonds.csv")
print(df.shape)
df.head()
grades = {"Fair": 1, "Good": 2, "Very Good": 3,
"Premium": 4, "Ideal": 5}
df["cut_score"] = df["cut"].map(grades)
df["cut_score"].value_counts().sort_index()
Write one-line rules with lambda#
Problem: a full def is too much ceremony for a tiny rule.
def to_mg(c): return c * 200is a named functionlambda c: c * 200is the same rule with no name- One carat weighs 200 milligrams
def to_mg(c):
return c * 200
print(df["carat"].apply(to_mg).head(3).tolist())
print(df["carat"].apply(lambda c: c * 200).head(3).tolist())
Quick check: what does (lambda c: c * 200)(1.5) give?
print((lambda c: c * 200)(1.5))
size = df["carat"].apply(lambda c: "1ct+" if c >= 1 else "small")
size.value_counts()
family = df["clarity"].map(lambda g: g[:2])
family.value_counts()
cuts = df["cut"].unique().tolist()
print(cuts)
print(sorted(cuts, key=lambda c: grades[c]))
Common mistake: writing return inside a lambda
df["carat"].apply(lambda c: return c * 200)
print(df["carat"].apply(lambda c: c * 200).max())
Use lambda with rows, groups and assign#
Problem: quick rules that need several columns or a whole group.
dims = df[['x', 'y', 'z']].head()
dims.apply(lambda r: r['x'] * r['y'] * r['z'], axis=1).round(1)
top10 = df.groupby("cut")["price"].agg(lambda s: s.quantile(0.9))
top10.sort_values()
priced = df.assign(
per_carat=lambda t: t["price"] / t["carat"],
)
priced[["carat", "price", "per_carat"]].head(3).round(2)
rel = df.groupby("cut")["price"].transform(lambda s: s / s.median())
print(rel.head(3).round(2).tolist())
pricey = df.loc[lambda t: t["price"] > 15000]
print(len(pricey), pricey["carat"].median())
Your turn#
With a lambda in agg, find the price range (max minus min) for each color.
spread = df.groupby("color")["price"].agg(lambda s: s.max() - s.min())
spread
Do maths on whole columns at once#
Problem: apply runs Python once per row, which is slow.
df["per_carat"] = df["price"] / df["carat"]
df["per_carat"].describe().round(0)
vat = df["price"] * 1.2
print(vat.head(3).tolist())
big = df["carat"] >= 1
print(big.sum())
print(round(big.mean(), 3))
combo = df["cut"] + " / " + df["color"]
combo.value_counts().head(3)
Your turn#
Depth should equal z / average of x and y, times 100. Check it, vectorised.
calc = df["z"] / ((df["x"] + df["y"]) / 2) * 100
diff = (calc - df["depth"]).abs()
print(round(diff.median(), 2), (diff > 1).sum())
Common mistake: using
andbetween two columns
df[(df["carat"] >= 1) and (df["price"] < 5000)]
bargain = (df["carat"] >= 1) & (df["price"] < 5000)
print(bargain.sum())
Quick check: of the one carat stones, what share cost under 5000?
print(round(bargain.sum() / big.sum(), 3))
Build conditional columns with np.where and np.select#
Problem: an if-else rule for every row, at vectorised speed.
df["size"] = np.where(df["carat"] >= 1, "1ct+", "small")
df["size"].value_counts()
rules = [
df["price"] < 1000,
df["price"] < 5000,
]
names = ["budget", "mid"]
df["tier"] = np.select(rules, names, default="luxury")
df["tier"].value_counts()
cuts_tier = pd.cut(df["price"], bins=[0, 1000, 5000, np.inf],
right=False, labels=names + ["luxury"])
print((cuts_tier.astype(str) == df["tier"]).all())
Common mistake: leaving out the brackets around each condition
np.where(df["carat"] >= 1 & df["price"] < 5000, "deal", "")
cond = (df["carat"] >= 1) & (df["price"] < 5000)
deal = pd.Series(np.where(cond, "deal", "other"))
deal.value_counts()
Your turn#
Use np.select to label depth: under 60 shallow, over 64 deep, else ideal.
depth_rules = [df["depth"] < 60, df["depth"] > 64]
depth_names = ["shallow", "deep"]
df["depth_band"] = np.select(depth_rules, depth_names, "ideal")
df["depth_band"].value_counts()
Measure the speed, and keep the maths safe#
Problem: prove vectorised is faster, and avoid nonsense from bad values.
import timeit
def slow_way():
return df.apply(lambda r: r["price"] / r["carat"], axis=1)
def fast_way():
return df["price"] / df["carat"]
print(slow_way().equals(fast_way()))
slow = timeit.timeit(slow_way, number=1)
fast = min(timeit.repeat(fast_way, number=1, repeat=3))
print("Vectorised at least 20x faster:", slow > 20 * fast)
print(df["clarity"].str[:2].equals(family))
flat = (df[["x", "y", "z"]] == 0).sum()
flat
Quick check: what volume does a stone with z = 0 get?
vol = df["x"] * df["y"] * df["z"]
print((vol == 0).sum())
sizes = df[["x", "y", "z"]].replace(0, np.nan)
df["volume"] = sizes["x"] * sizes["y"] * sizes["z"]
print(df["volume"].isna().sum(), round(df["volume"].min(), 1))
print(round(vol.mean(), 2), round(df["volume"].mean(), 2))
Mini-project: is the Ideal cut really cheaper?#
Plan: compare prices, check size, compare like with like, conclude.
by_cut = df.groupby("cut").agg(
median_price=("price", "median"),
median_carat=("carat", "median"),
)
by_cut.sort_values("median_price")
fair = df["cut"] == "Fair"
ideal = df["cut"] == "Ideal"
print(round(df.loc[fair, "size"].eq("1ct+").mean(), 2))
print(round(df.loc[ideal, "size"].eq("1ct+").mean(), 2))
grid = df.groupby(["size", "cut"])["per_carat"].median()
grid = grid.unstack().round(0)
grid[["Fair", "Good", "Very Good", "Premium", "Ideal"]]
counts = df.groupby(["size", "cut"]).size().unstack()
counts[["Fair", "Good", "Very Good", "Premium", "Ideal"]]
import matplotlib.pyplot as plt
big_ones = grid.loc["1ct+", ["Fair", "Good", "Very Good",
"Premium", "Ideal"]]
ax = big_ones.plot(kind="bar", rot=0, figsize=(7, 4))
ax.set_title("Price per carat, stones of 1 carat or more")
ax.set_xlabel("Cut grade")
ax.set_ylabel("Median price per carat ($)")
plt.show()
gap = big_ones["Ideal"] / big_ones["Fair"] - 1
print(f"Ideal costs {gap:.0%} more per carat than Fair at 1ct+")
Recap#
- lambda writes a one-line function: input, colon, result
- Lambdas fit anywhere a function is expected: apply, map, agg, assign
- Whole-column maths and comparisons are vectorised and fast
- Combine conditions with & and |, with brackets around each one
- np.where and np.select replace if-else lambdas at full speed
Challenges#
- Add a column with the price in thousands, rounded to 1 decimal.
- Use np.where to flag stones whose table is above 60.
- Use a lambda in agg to find each clarity's median carat weight.
What's next#
Lesson 25: Chaining Operations and Method Pipelines Join many steps into one clear, readable pipeline.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.
Related lessons
20 minLesson 24
Master Lambda Functions & Vectorized Operations in Pandas for Faster Data Analysis
26 minLesson 4
Data Mining with Python: pandas & scikit-learn Crash Course
28 minLesson 6
Introduction to Pandas DataFrame: Essential Operations and Attributes Explained
23 minLesson 11