Mathew K AnalyticsLesson 30 · Mastering Pandas
Understanding Multi-Level Grouping and Custom Aggregations in Pandas for Data Analysis
Welcome! Today we will master advanced grouping and aggregation in pandas. Learn why grouping by more than one column is so powerful. See how custom…
- CourseMastering Pandas
- Lesson30 of 44
- Video21 min
- FormatJupyter notebook · 16 code cells
What you'll learn
Data
No separate download needed — the notebook creates or downloads everything it uses.
📓 Full notebook
Download .ipynbMulti-level Grouping and Custom Aggregations in Pandas#
Welcome! Today we will master advanced grouping and aggregation in pandas.
- Learn why grouping by more than one column is so powerful.
- See how custom functions reveal new insights from your data.
We'll practice with the Titanic dataset, exploring survival rates by gender and passenger class. Ready? Let's dive in!
# Suppress warnings for a clean experience
import warnings
import numpy as np
np.random.seed(42)
warnings.filterwarnings("ignore")
# Data setup (Titanic Dataset)
import pandas as pd
url = 'https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv'
df = pd.read_csv(url)
print(df.shape)
print(df.head(3))
Why Group By Multiple Columns?#
Grouping by more than one column lets us break down patterns within subgroups.
For example:
- How likely were women in First Class to survive?
- Did survival differ by 'Sex' across all passenger classes?
Pandas lets us answer these questions in just a few lines!
# Preview value counts for Sex and Pclass
print(df['Sex'].value_counts())
print(df['Pclass'].value_counts())
# Group by Sex and Pclass, then count survivors
grouped = df.groupby(['Sex', 'Pclass'])['Survived']
print(grouped.sum())
# Group by Sex and Pclass, then calculate survival rate
survival_rates = grouped.mean()
print(survival_rates)
What is a MultiIndex?#
Grouping by multiple columns creates a MultiIndex for the rows.
A MultiIndex lets pandas identify groups based on several columns.
This structure allows us to drill deep into subgroup details.
We can reset it back to columns using .reset_index().
# Reset index to turn MultiIndex into columns
survival_rates = survival_rates.reset_index()
print(survival_rates.head())
# Custom aggregation: count, mean, and median simultaneously
agg_results = df.groupby(['Sex', 'Pclass'])['Fare'].agg(['count', 'mean', 'median'])
print(agg_results)
# Using a custom aggregation function: percent missing age
def percent_missing(series):
return series.isnull().mean() * 100
missing_age = df.groupby(['Sex', 'Pclass'])['Age'].agg(percent_missing)
print(missing_age)
# Aggregating different functions per column using a dictionary
agg_dict = {
'Fare': ['mean', 'std'],
'Age': ['median', percent_missing]
}
multiagg = df.groupby(['Sex', 'Pclass']).agg(agg_dict)
print(multiagg.head())
# Sorting aggregation results by mean Fare
sorted_multiagg = multiagg.sort_values(('Fare', 'mean'), ascending=False)
print(sorted_multiagg)
# Applying a lambda: survival rates for minors versus adults by class
df['IsMinor'] = df['Age'] < 18
minor_rates = df.groupby(['Pclass', 'IsMinor'])['Survived'].mean()
print(minor_rates)
# Unstack to make the aggregation easier to read as a table
table_rates = minor_rates.unstack()
print(table_rates)
# Mini-project: Find median fares and average ages per embark port and class
results = df.groupby(['Embarked', 'Pclass']).agg({'Fare': 'median', 'Age': 'mean'})
print(results)
# Practice: Use input() to explore custom groupings
col1 = input("First group-by column (Sex/Pclass/Embarked): ")
col2 = input("Second group-by column (Sex/Pclass/Embarked): ")
stat = input("Choose stat (mean/count/median): ")
col = input("Target numeric column (Fare/Age/SibSp): ")
custom = df.groupby([col1, col2])[col].agg(stat)
print(custom)
Best Practices and Performance Tips#
- Group only by columns you need: Less memory, faster code.
- Try
.agg()with built-in and custom functions together. - For big data, watch for NaN and missing values.
- Use
.sort_values()to help spot most important groups. - Reset unwanted multi-level indexes for tidy tables.
Troubleshooting:
- Errors about missing columns usually mean a typo or missing reset_index().
- For KeyError: check all groupby columns existed in your DataFrame.
- Use
df.info()to check your columns' types if grouping fails.
# Challenge: Calculate total family members on board and average survival by family size and class
df['FamilySize'] = df['SibSp'] + df['Parch'] + 1
fam_group = df.groupby(['FamilySize', 'Pclass'])['Survived'].mean()
print(fam_group.head(10))
# Extra: Visualize survival rates by class and gender
import matplotlib.pyplot as plt
rates = df.groupby(['Sex', 'Pclass'])['Survived'].mean().unstack()
rates.plot(kind='bar')
plt.title('Survival Rate by Gender and Class')
plt.ylabel('Survival Rate')
plt.show()
Recap: Key Concepts for Multi-level Grouping and Custom Aggregation#
- Use
groupbywith lists for multiple columns. - Custom aggregation functions help answer unique questions.
- Chain with
.agg()for clear, powerful group summaries. - Use
.reset_index()and.unstack()for easier reading and plotting.
The next time you need detailed breakdowns or hidden patterns, you will be ready!
Thank you for joining! If this lesson was helpful, hit Like and Subscribe on YouTube.#
Share your own groupby ideas in the comments!
Happy coding with pandas!
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.
Related lessons
24 minLesson 5
Comprehensive Guide to Pandas Series and DataFrames for Data Analysis in Python
21 minLesson 9
Introduction to Pandas for Market Research: Master Data Analysis with Python
26 minLesson 4
Introduction to Python, Pandas, and scikit-learn for Data Analysis and Machine Learning
14 minLesson 6