Mathew K AnalyticsLesson 49 · Mastering Pandas
Understanding Data Correlation and Heatmap Visualization Using Python and Pandas
In this lesson, you will learn how to explore relationships between features using correlation coefficients and visualize these connections with heatmaps.…
- CourseMastering Pandas
- Lesson49 of 44
- Video15 min
- FormatJupyter notebook · 13 code cells
What you'll learn
Data
No separate download needed — the notebook creates or downloads everything it uses.
📓 Full notebook
Download .ipynbData Correlation and Heatmaps in Pandas#
In this lesson, you will learn how to explore relationships between features using correlation coefficients and visualize these connections with heatmaps.
import warnings; warnings.filterwarnings('ignore') # Ignore warnings for clean output
import numpy as np
np.random.seed(42)
Dataset Introduction: The Titanic Dataset#
We will use the Titanic dataset, a classic for learning data analysis. This dataset contains information about passengers like age, fare, and survival status.
# Data setup (Titanic Dataset)
import pandas as pd
url = 'https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv'
df = pd.read_csv(url)
print(df.shape)
print(df.head(3))
# Quick data info
print(df.info())
# Select only numeric columns for correlation
numeric_df = df.select_dtypes(include='number')
print(numeric_df.columns.tolist())
What is Correlation?#
Correlation shows how two variables move together. A correlation of 1.0 means both always move in the same direction. A value of -1.0 means they move in exactly opposite directions. A correlation near 0 means there is little or no relationship.
# Basic correlation matrix
corr_matrix = numeric_df.corr()
print(corr_matrix)
# Find which features are most related to survival
print(corr_matrix['Survived'].sort_values(ascending=False))
# Visualize the correlation matrix with a heatmap
import seaborn as sns
import matplotlib.pyplot as plt
plt.figure(figsize=(8, 6))
sns.heatmap(corr_matrix, annot=True, cmap='coolwarm', fmt='.2f')
plt.title('Correlation Heatmap (Titanic Dataset)')
plt.show()
Optional: Focus on Top Correlations Only#
Sometimes, you want to see only the strongest or most interesting relationships. Let us filter for large absolute correlations (ignoring the diagonal).
# List feature pairs with high absolute correlation (excluding the diagonal)
corr_pairs = corr_matrix.unstack()
sorted_pairs = corr_pairs[(corr_pairs.abs() > 0.5) & (corr_pairs.index.get_level_values(0) != corr_pairs.index.get_level_values(1))]
print(sorted_pairs.sort_values(ascending=False))
# Visualize pairwise relationships with Seaborn pairplot
sns.pairplot(numeric_df[['Survived', 'Fare', 'Age', 'Pclass']], diag_kind='kde')
plt.suptitle('Pairwise Plots of Key Titanic Features', y=1.02)
plt.show()
How Correlation Informs Feature Engineering#
Features with strong correlation might overlap in what they tell us, so including both may not always help. But, finding relationships helps us create new insights or spot problems such as data leakage.
# Mini-challenge: Remove one column with high correlation to another
# Let us drop 'SibSp' if it is very similar to 'Parch', for example
reduced_df = numeric_df.drop('SibSp', axis=1)
print(reduced_df.head(2))
# Heatmap after column removal
corr_new = reduced_df.corr()
plt.figure(figsize=(7, 5))
sns.heatmap(corr_new, annot=True, cmap='coolwarm', fmt='.2f')
plt.title('Correlation Heatmap After Feature Removal')
plt.show()
# Handling missing data before correlation (if needed)
missing_cols = numeric_df.columns[numeric_df.isnull().any()].tolist()
print('Columns with missing values:', missing_cols)
filled_df = numeric_df.fillna(numeric_df.mean())
print('Any NaN still? ', filled_df.isnull().any().any())
Troubleshooting: Why Am I Getting All Zeros or NaNs?#
If you ever see a correlation matrix with only zeros or NaN values, check your data! Maybe you tried to correlate text columns, have missing values, or used an empty DataFrame.
# Intentional error: Try correlating object columns
try:
obj_df = df.select_dtypes(include='object')
print(obj_df.corr())
except Exception as e:
print('Error:', e)
Recap: Key Takeaways#
- Correlation measures how strongly two numeric features move together.
- Use heatmaps and pairplots to spot relationships, outliers, and redundant features.
- Always check for missing values before you calculate correlations.
Practice & Challenge#
Can you use the same approach to explore the Iris or Tips datasets?
Share your favorite correlation findings or post your questions in the video comments!
If you found this lesson helpful, like and subscribe for more pandas tutorials!
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.
Related lessons
18 minComprehensive Analysis and Visualization of Global Covid-19 Data Using Python
23 minBuilding a Car Price Prediction Model Using Python and Data Science Techniques
19 minBuilding a News Data Scraping and Analysis Project Using Python
23 min