Mathew K Analytics

Lesson 43 · Pandas Full Course

Correlation and Heatmaps in Pandas: Find What Really Drives Your Data | Pandas Course #43

In this lesson you'll be able to: Measure how two columns move together with corr() Build a correlation matrix for a whole table Draw it as a clear, honest…

What you'll learn

Datasets used in this lesson

Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.

📓 Full notebook

Download .ipynb

Pandas 43: Data Correlation and Heatmaps#

In this lesson you'll be able to:

  • Measure how two columns move together with corr()
  • Build a correlation matrix for a whole table
  • Draw it as a clear, honest heatmap
  • Rank the strongest pairs and spot redundant columns
  • Choose Pearson or Spearman, and avoid the classic traps Datasets: cars (data/pandas/mpg.csv), Titanic (data/pandas/titanic.csv)

Warm-up: one Seaborn scatter#

Lesson 42 combined pandas, Matplotlib and Seaborn.

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
cars = pd.read_csv("data/pandas/mpg.csv")
print(cars.shape)
cars[["mpg", "cylinders", "horsepower", "weight", "origin"]].head()
(398, 9)
mpg cylinders horsepower weight origin
0 18.0 8 130.0 3504 usa
1 15.0 8 165.0 3693 usa
2 18.0 8 150.0 3436 usa
3 16.0 8 150.0 3433 usa
4 17.0 8 140.0 3449 usa
ax = sns.scatterplot(data=cars, x="weight", y="mpg", alpha=0.6)
ax.set(title="Heavier cars go fewer miles per gallon",
       xlabel="Weight (lb)", ylabel="Miles per gallon")
plt.show()
No description has been provided for this image

Measure two columns with one number#

Problem: turn "they move together" into a number we can compare.

  • +1: rise together in a perfect straight line
  • -1: one rises exactly as the other falls
  • 0: no straight-line link at all
up = cars["cylinders"].corr(cars["displacement"])
down = cars["displacement"].corr(cars["mpg"])
titanic = pd.read_csv("data/pandas/titanic.csv")
none = titanic["PassengerId"].corr(titanic["Survived"])
print(round(up, 2), round(down, 2), round(none, 2))
0.95 -0.8 -0.01
r = cars["weight"].corr(cars["mpg"])
print(round(r, 2))
-0.83

Quick check: acceleration and mpg: a positive or negative link?

print(round(cars["acceleration"].corr(cars["mpg"]), 2))
0.42

Your turn#

Find the correlation between horsepower and acceleration.

hp_acc = cars["horsepower"].corr(cars["acceleration"])
print(round(hp_acc, 2))
-0.69

Build a correlation matrix#

Problem: compare every pair of columns at once.

Common mistake: calling corr() on a table with text columns

cars.corr()
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[8], line 1
----> 1 cars.corr()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\frame.py:13135, in DataFrame.corr(self, method, min_periods, numeric_only)
  13133 cols = data.columns
  13134 idx = cols.copy()
> 13135 mat = data.to_numpy(dtype=float, na_value=np.nan, copy=False)
  13137 if method == "pearson":
  13138     correl = libalgos.nancorr(mat, minp=min_periods)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\frame.py:2081, in DataFrame.to_numpy(self, dtype, copy, na_value)
   2079 if dtype is not None:
   2080     dtype = np.dtype(dtype)
-> 2081 result = self._mgr.as_array(dtype=dtype, copy=copy, na_value=na_value)
   2082 if result.dtype is not dtype:
   2083     result = np.asarray(result, dtype=dtype)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\managers.py:1885, in BlockManager.as_array(self, dtype, copy, na_value)
   1883         arr.flags.writeable = False
   1884 else:
-> 1885     arr = self._interleave(dtype=dtype, na_value=na_value)
   1886     # The underlying data was copied within _interleave, so no need
   1887     # to further copy if copy=True or setting na_value
   1889 if na_value is lib.no_default:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\managers.py:1938, in BlockManager._interleave(self, dtype, na_value)
   1932 rl = blk.mgr_locs
   1933 if blk.is_extension:
   1934     # Avoid implicit conversion of extension blocks to object
   1935 
   1936     # error: Item "ndarray" of "Union[ndarray, ExtensionArray]" has no
   1937     # attribute "to_numpy"
-> 1938     arr = blk.values.to_numpy(  # type: ignore[union-attr]
   1939         dtype=dtype,
   1940         na_value=na_value,
   1941     )
   1942 else:
   1943     arr = blk.get_values(dtype)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\arrow\array.py:1803, in ArrowExtensionArray.to_numpy(self, dtype, copy, na_value)
   1801 result = data._pa_array.to_numpy()
   1802 if dtype is not None:
-> 1803     result = result.astype(dtype, copy=False)
   1804 if copy:
   1805     result = result.copy()

ValueError: could not convert string to float: 'usa'
corr = cars.corr(numeric_only=True)
corr.round(2)
mpg cylinders displacement horsepower weight acceleration model_year
mpg 1.00 -0.78 -0.80 -0.78 -0.83 0.42 0.58
cylinders -0.78 1.00 0.95 0.84 0.90 -0.51 -0.35
displacement -0.80 0.95 1.00 0.90 0.93 -0.54 -0.37
horsepower -0.78 0.84 0.90 1.00 0.86 -0.69 -0.42
weight -0.83 0.90 0.93 0.86 1.00 -0.42 -0.31
acceleration 0.42 -0.51 -0.54 -0.69 -0.42 1.00 0.29
model_year 0.58 -0.35 -0.37 -0.42 -0.31 0.29 1.00
corr["mpg"].drop("mpg").sort_values().round(2)
weight         -0.83
displacement   -0.80
horsepower     -0.78
cylinders      -0.78
acceleration    0.42
model_year      0.58
Name: mpg, dtype: float64

Your turn#

Which column is most strongly linked to horsepower?

hp = corr["horsepower"].drop("horsepower")
print(hp.abs().idxmax(), round(hp.abs().max(), 2))
displacement 0.9

Turn the matrix into a heatmap#

Problem: a grid of 49 numbers is hard to scan.

fig, ax = plt.subplots(figsize=(8, 6))
sns.heatmap(corr, annot=True, fmt=".2f", cmap="coolwarm",
            vmin=-1, vmax=1, ax=ax)
ax.set_title("Correlation between car features")
plt.show()
No description has been provided for this image

Quick check: why fix vmin=-1 and vmax=1?

fig, ax = plt.subplots(figsize=(8, 6))
sns.heatmap(corr, annot=True, fmt=".2f", cmap="coolwarm", ax=ax)
ax.set_title("No fixed scale: colours stretch to the data")
plt.show()
No description has been provided for this image
import numpy as np
mask = np.triu(np.ones_like(corr, dtype=bool))
fig, ax = plt.subplots(figsize=(8, 6))
sns.heatmap(corr, mask=mask, annot=True, fmt=".2f",
            cmap="coolwarm", vmin=-1, vmax=1, ax=ax)
ax.set_title("Each pair shown once")
plt.show()
No description has been provided for this image
target = corr[["mpg"]].drop("mpg").sort_values("mpg")
fig, ax = plt.subplots(figsize=(3, 5))
sns.heatmap(target, annot=True, fmt=".2f", cmap="coolwarm",
            vmin=-1, vmax=1, ax=ax)
ax.set_title("Linked to mpg")
plt.show()
No description has been provided for this image
cols = ["mpg", "weight", "horsepower", "acceleration"]
grid = sns.pairplot(cars[cols], corner=True,
                    plot_kws={"alpha": 0.5, "s": 15})
grid.figure.suptitle("Every pair as a scatter plot", y=1.02)
plt.show()
No description has been provided for this image

Rank the strongest pairs#

Problem: list the strongest links, without duplicates or self-matches.

lower = np.tril(np.ones_like(corr, dtype=bool), k=-1)
pairs = corr.where(lower).stack().dropna()
print(len(pairs))
pairs.round(2).head()
21
cylinders     mpg         -0.78
displacement  mpg         -0.80
              cylinders    0.95
horsepower    mpg         -0.78
              cylinders    0.84
dtype: float64

Common mistake: sorting a Series with by=

pairs.sort_values(by="mpg")
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[18], line 1
----> 1 pairs.sort_values(by="mpg")

TypeError: Series.sort_values() got an unexpected keyword argument 'by'
ranked = pairs.sort_values(key=abs, ascending=False)
ranked.round(2).head(6)
displacement  cylinders       0.95
weight        displacement    0.93
horsepower    displacement    0.90
weight        cylinders       0.90
              horsepower      0.86
horsepower    cylinders       0.84
dtype: float64
strong = ranked[ranked.abs() >= 0.9]
strong.round(2)
displacement  cylinders       0.95
weight        displacement    0.93
dtype: float64

Your turn#

List the pairs with a weak link: between -0.4 and 0.4.

weak = ranked[ranked.between(-0.4, 0.4)]
weak.round(2)
model_year  displacement   -0.37
            cylinders      -0.35
            weight         -0.31
            acceleration    0.29
dtype: float64

Pearson, Spearman and the classic traps#

Problem: a curve, a gap or a hidden cause can fool one number.

ax = sns.scatterplot(data=cars, x="horsepower", y="mpg", alpha=0.6)
ax.set(title="A curve, not a straight line",
       xlabel="Horsepower", ylabel="Miles per gallon")
plt.show()
No description has been provided for this image
pair = cars[["horsepower", "mpg"]]
print(round(pair.corr(method="pearson").iloc[0, 1], 2))
print(round(pair.corr(method="spearman").iloc[0, 1], 2))
-0.78
-0.85

Quick check: weight and mpg also curve. Will Spearman be stronger than -0.83?

ranks = cars[["weight", "mpg"]].corr(method="spearman")
print(round(ranks.loc["weight", "mpg"], 2))
-0.87
print(cars["horsepower"].isna().sum())
print(pair.dropna().shape[0])
6
392
by_year = cars.groupby("model_year")[["mpg", "weight"]].mean()
by_year.round(0).iloc[[0, 4, 8, 12]]
mpg weight
model_year
70 18.0 3373.0
74 23.0 2878.0
78 24.0 2862.0
82 32.0 2454.0
print(round(cars["model_year"].corr(cars["weight"]), 2))
-0.31

Correlation with a target: Titanic survival#

Problem: which columns are linked to the outcome we care about?

titanic = pd.read_csv("data/pandas/titanic.csv")
num = titanic.drop(columns="PassengerId")
num.corrwith(num["Survived"], numeric_only=True).round(2)
Survived    1.00
Pclass     -0.34
Age        -0.08
SibSp      -0.04
Parch       0.08
Fare        0.26
dtype: float64

Common mistake: correlating a text column

titanic[["Sex", "Survived"]].corr()
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[29], line 1
----> 1 titanic[["Sex", "Survived"]].corr()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\frame.py:13135, in DataFrame.corr(self, method, min_periods, numeric_only)
  13133 cols = data.columns
  13134 idx = cols.copy()
> 13135 mat = data.to_numpy(dtype=float, na_value=np.nan, copy=False)
  13137 if method == "pearson":
  13138     correl = libalgos.nancorr(mat, minp=min_periods)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\frame.py:2081, in DataFrame.to_numpy(self, dtype, copy, na_value)
   2079 if dtype is not None:
   2080     dtype = np.dtype(dtype)
-> 2081 result = self._mgr.as_array(dtype=dtype, copy=copy, na_value=na_value)
   2082 if result.dtype is not dtype:
   2083     result = np.asarray(result, dtype=dtype)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\managers.py:1885, in BlockManager.as_array(self, dtype, copy, na_value)
   1883         arr.flags.writeable = False
   1884 else:
-> 1885     arr = self._interleave(dtype=dtype, na_value=na_value)
   1886     # The underlying data was copied within _interleave, so no need
   1887     # to further copy if copy=True or setting na_value
   1889 if na_value is lib.no_default:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\managers.py:1938, in BlockManager._interleave(self, dtype, na_value)
   1932 rl = blk.mgr_locs
   1933 if blk.is_extension:
   1934     # Avoid implicit conversion of extension blocks to object
   1935 
   1936     # error: Item "ndarray" of "Union[ndarray, ExtensionArray]" has no
   1937     # attribute "to_numpy"
-> 1938     arr = blk.values.to_numpy(  # type: ignore[union-attr]
   1939         dtype=dtype,
   1940         na_value=na_value,
   1941     )
   1942 else:
   1943     arr = blk.get_values(dtype)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\arrow\array.py:1803, in ArrowExtensionArray.to_numpy(self, dtype, copy, na_value)
   1801 result = data._pa_array.to_numpy()
   1802 if dtype is not None:
-> 1803     result = result.astype(dtype, copy=False)
   1804 if copy:
   1805     result = result.copy()

ValueError: could not convert string to float: 'male'
titanic["is_female"] = (titanic["Sex"] == "female").astype(int)
cols = ["Survived", "is_female", "Pclass", "Fare", "Age"]
titanic[cols].corr()["Survived"].round(2)
Survived     1.00
is_female    0.54
Pclass      -0.34
Fare         0.26
Age         -0.08
Name: Survived, dtype: float64
fig, ax = plt.subplots(figsize=(6, 5))
sns.heatmap(titanic[cols].corr(), annot=True, fmt=".2f",
            cmap="coolwarm", vmin=-1, vmax=1, ax=ax)
ax.set_title("Titanic: what is linked to survival?")
plt.show()
No description has been provided for this image

Your turn#

Fares are very skewed. Compare Pearson and Spearman for Fare and Survived.

fs = titanic[["Fare", "Survived"]]
print(round(fs.corr().iloc[0, 1], 2))
print(round(fs.corr(method="spearman").iloc[0, 1], 2))
0.26
0.32

Mini-project: what really drives fuel economy?#

Plan: rank features, remove repeats, test within each origin.

feats = ["cylinders", "displacement", "horsepower", "weight"]
spear = cars[feats + ["mpg"]].corr(method="spearman")["mpg"]
spear.drop("mpg").sort_values().round(2)
weight         -0.87
displacement   -0.86
horsepower     -0.85
cylinders      -0.82
Name: mpg, dtype: float64
among = cars[feats].corr().round(2)
among
cylinders displacement horsepower weight
cylinders 1.00 0.95 0.84 0.90
displacement 0.95 1.00 0.90 0.93
horsepower 0.84 0.90 1.00 0.86
weight 0.90 0.93 0.86 1.00
per_origin = cars.groupby("origin")[["weight", "mpg"]].corr()
per_origin = per_origin.xs("weight", level=1)["mpg"]
per_origin.round(2)
origin
europe   -0.53
japan    -0.56
usa      -0.85
Name: mpg, dtype: float64
cars.groupby("origin")["weight"].agg(["min", "max", "std"]).round()
min max std
origin
europe 1825 3820 490.0
japan 1613 2930 320.0
usa 1800 5140 795.0
ax = sns.scatterplot(data=cars, x="weight", y="mpg", hue="origin",
                     alpha=0.6)
ax.set(title="Japanese and European cars cover a narrow weight range",
       xlabel="Weight (lb)", ylabel="Miles per gallon")
plt.show()
No description has been provided for this image
ax = per_origin.sort_values().plot(kind="barh", color="teal")
ax.set_title("Weight vs mpg correlation by origin")
ax.set_xlabel("Pearson correlation")
ax.set_ylabel("Origin")
plt.show()
No description has been provided for this image
best = spear.drop("mpg").abs().idxmax()
print(f"Strongest: {best} ({spear[best]:.2f})")
print(f"USA cars: {per_origin['usa']:.2f}")
print(f"Japanese cars: {per_origin['japan']:.2f}")
Strongest: weight (-0.87)
USA cars: -0.85
Japanese cars: -0.56
import os
os.makedirs("output/pandas", exist_ok=True)
table = ranked.round(2).rename("r").rename_axis(["col_a", "col_b"])
table.to_csv("output/pandas/l43_ranked_pairs.csv")
back = pd.read_csv("output/pandas/l43_ranked_pairs.csv")
print(back.shape)
back.head(3)
(21, 3)
col_a col_b r
0 displacement cylinders 0.95
1 weight displacement 0.93
2 horsepower displacement 0.90

Recap#

  • corr() runs from -1 to +1; 0 means no straight-line link
  • Use numeric_only=True, or turn text into numbers first
  • Heatmaps need a fixed scale: vmin=-1, vmax=1
  • Mask, stack and sort by abs to rank pairs
  • Spearman for curves; correlation is not causation

Challenges#

  1. Masked heatmap of the Titanic columns, Spearman this time.
  2. Which car feature is most linked to acceleration?
  3. Compare weight and mpg correlation before and after 1977.

What's next#

Lesson 44: Building Interactive Visuals with Plotly and Pandas Charts you can hover, zoom and animate, straight from a DataFrame.

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.