Lesson 43 · Pandas Full Course
Correlation and Heatmaps in Pandas: Find What Really Drives Your Data | Pandas Course #43
In this lesson you'll be able to: Measure how two columns move together with corr() Build a correlation matrix for a whole table Draw it as a clear, honest…
- CoursePandas Full Course
- Lesson43 of 65
- FormatJupyter notebook · 39 code cells
- Data2 datasets
What you'll learn
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
- mpg.csv preview20.7 KB
- titanic.csv preview58.9 KB
📓 Full notebook
Download .ipynbPandas 43: Data Correlation and Heatmaps#
In this lesson you'll be able to:
- Measure how two columns move together with corr()
- Build a correlation matrix for a whole table
- Draw it as a clear, honest heatmap
- Rank the strongest pairs and spot redundant columns
- Choose Pearson or Spearman, and avoid the classic traps Datasets: cars (data/pandas/mpg.csv), Titanic (data/pandas/titanic.csv)
Warm-up: one Seaborn scatter#
Lesson 42 combined pandas, Matplotlib and Seaborn.
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
cars = pd.read_csv("data/pandas/mpg.csv")
print(cars.shape)
cars[["mpg", "cylinders", "horsepower", "weight", "origin"]].head()
ax = sns.scatterplot(data=cars, x="weight", y="mpg", alpha=0.6)
ax.set(title="Heavier cars go fewer miles per gallon",
xlabel="Weight (lb)", ylabel="Miles per gallon")
plt.show()
Measure two columns with one number#
Problem: turn "they move together" into a number we can compare.
- +1: rise together in a perfect straight line
- -1: one rises exactly as the other falls
- 0: no straight-line link at all
up = cars["cylinders"].corr(cars["displacement"])
down = cars["displacement"].corr(cars["mpg"])
titanic = pd.read_csv("data/pandas/titanic.csv")
none = titanic["PassengerId"].corr(titanic["Survived"])
print(round(up, 2), round(down, 2), round(none, 2))
r = cars["weight"].corr(cars["mpg"])
print(round(r, 2))
Quick check: acceleration and mpg: a positive or negative link?
print(round(cars["acceleration"].corr(cars["mpg"]), 2))
Your turn#
Find the correlation between horsepower and acceleration.
hp_acc = cars["horsepower"].corr(cars["acceleration"])
print(round(hp_acc, 2))
Build a correlation matrix#
Problem: compare every pair of columns at once.
Common mistake: calling corr() on a table with text columns
cars.corr()
corr = cars.corr(numeric_only=True)
corr.round(2)
corr["mpg"].drop("mpg").sort_values().round(2)
Your turn#
Which column is most strongly linked to horsepower?
hp = corr["horsepower"].drop("horsepower")
print(hp.abs().idxmax(), round(hp.abs().max(), 2))
Turn the matrix into a heatmap#
Problem: a grid of 49 numbers is hard to scan.
fig, ax = plt.subplots(figsize=(8, 6))
sns.heatmap(corr, annot=True, fmt=".2f", cmap="coolwarm",
vmin=-1, vmax=1, ax=ax)
ax.set_title("Correlation between car features")
plt.show()
Quick check: why fix vmin=-1 and vmax=1?
fig, ax = plt.subplots(figsize=(8, 6))
sns.heatmap(corr, annot=True, fmt=".2f", cmap="coolwarm", ax=ax)
ax.set_title("No fixed scale: colours stretch to the data")
plt.show()
import numpy as np
mask = np.triu(np.ones_like(corr, dtype=bool))
fig, ax = plt.subplots(figsize=(8, 6))
sns.heatmap(corr, mask=mask, annot=True, fmt=".2f",
cmap="coolwarm", vmin=-1, vmax=1, ax=ax)
ax.set_title("Each pair shown once")
plt.show()
target = corr[["mpg"]].drop("mpg").sort_values("mpg")
fig, ax = plt.subplots(figsize=(3, 5))
sns.heatmap(target, annot=True, fmt=".2f", cmap="coolwarm",
vmin=-1, vmax=1, ax=ax)
ax.set_title("Linked to mpg")
plt.show()
cols = ["mpg", "weight", "horsepower", "acceleration"]
grid = sns.pairplot(cars[cols], corner=True,
plot_kws={"alpha": 0.5, "s": 15})
grid.figure.suptitle("Every pair as a scatter plot", y=1.02)
plt.show()
Rank the strongest pairs#
Problem: list the strongest links, without duplicates or self-matches.
lower = np.tril(np.ones_like(corr, dtype=bool), k=-1)
pairs = corr.where(lower).stack().dropna()
print(len(pairs))
pairs.round(2).head()
Common mistake: sorting a Series with by=
pairs.sort_values(by="mpg")
ranked = pairs.sort_values(key=abs, ascending=False)
ranked.round(2).head(6)
strong = ranked[ranked.abs() >= 0.9]
strong.round(2)
Your turn#
List the pairs with a weak link: between -0.4 and 0.4.
weak = ranked[ranked.between(-0.4, 0.4)]
weak.round(2)
Pearson, Spearman and the classic traps#
Problem: a curve, a gap or a hidden cause can fool one number.
ax = sns.scatterplot(data=cars, x="horsepower", y="mpg", alpha=0.6)
ax.set(title="A curve, not a straight line",
xlabel="Horsepower", ylabel="Miles per gallon")
plt.show()
pair = cars[["horsepower", "mpg"]]
print(round(pair.corr(method="pearson").iloc[0, 1], 2))
print(round(pair.corr(method="spearman").iloc[0, 1], 2))
Quick check: weight and mpg also curve. Will Spearman be stronger than -0.83?
ranks = cars[["weight", "mpg"]].corr(method="spearman")
print(round(ranks.loc["weight", "mpg"], 2))
print(cars["horsepower"].isna().sum())
print(pair.dropna().shape[0])
by_year = cars.groupby("model_year")[["mpg", "weight"]].mean()
by_year.round(0).iloc[[0, 4, 8, 12]]
print(round(cars["model_year"].corr(cars["weight"]), 2))
Correlation with a target: Titanic survival#
Problem: which columns are linked to the outcome we care about?
titanic = pd.read_csv("data/pandas/titanic.csv")
num = titanic.drop(columns="PassengerId")
num.corrwith(num["Survived"], numeric_only=True).round(2)
Common mistake: correlating a text column
titanic[["Sex", "Survived"]].corr()
titanic["is_female"] = (titanic["Sex"] == "female").astype(int)
cols = ["Survived", "is_female", "Pclass", "Fare", "Age"]
titanic[cols].corr()["Survived"].round(2)
fig, ax = plt.subplots(figsize=(6, 5))
sns.heatmap(titanic[cols].corr(), annot=True, fmt=".2f",
cmap="coolwarm", vmin=-1, vmax=1, ax=ax)
ax.set_title("Titanic: what is linked to survival?")
plt.show()
Your turn#
Fares are very skewed. Compare Pearson and Spearman for Fare and Survived.
fs = titanic[["Fare", "Survived"]]
print(round(fs.corr().iloc[0, 1], 2))
print(round(fs.corr(method="spearman").iloc[0, 1], 2))
Mini-project: what really drives fuel economy?#
Plan: rank features, remove repeats, test within each origin.
feats = ["cylinders", "displacement", "horsepower", "weight"]
spear = cars[feats + ["mpg"]].corr(method="spearman")["mpg"]
spear.drop("mpg").sort_values().round(2)
among = cars[feats].corr().round(2)
among
per_origin = cars.groupby("origin")[["weight", "mpg"]].corr()
per_origin = per_origin.xs("weight", level=1)["mpg"]
per_origin.round(2)
cars.groupby("origin")["weight"].agg(["min", "max", "std"]).round()
ax = sns.scatterplot(data=cars, x="weight", y="mpg", hue="origin",
alpha=0.6)
ax.set(title="Japanese and European cars cover a narrow weight range",
xlabel="Weight (lb)", ylabel="Miles per gallon")
plt.show()
ax = per_origin.sort_values().plot(kind="barh", color="teal")
ax.set_title("Weight vs mpg correlation by origin")
ax.set_xlabel("Pearson correlation")
ax.set_ylabel("Origin")
plt.show()
best = spear.drop("mpg").abs().idxmax()
print(f"Strongest: {best} ({spear[best]:.2f})")
print(f"USA cars: {per_origin['usa']:.2f}")
print(f"Japanese cars: {per_origin['japan']:.2f}")
import os
os.makedirs("output/pandas", exist_ok=True)
table = ranked.round(2).rename("r").rename_axis(["col_a", "col_b"])
table.to_csv("output/pandas/l43_ranked_pairs.csv")
back = pd.read_csv("output/pandas/l43_ranked_pairs.csv")
print(back.shape)
back.head(3)
Recap#
- corr() runs from -1 to +1; 0 means no straight-line link
- Use numeric_only=True, or turn text into numbers first
- Heatmaps need a fixed scale: vmin=-1, vmax=1
- Mask, stack and sort by abs to rank pairs
- Spearman for curves; correlation is not causation
Challenges#
- Masked heatmap of the Titanic columns, Spearman this time.
- Which car feature is most linked to acceleration?
- Compare weight and mpg correlation before and after 1977.
What's next#
Lesson 44: Building Interactive Visuals with Plotly and Pandas Charts you can hover, zoom and animate, straight from a DataFrame.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.



