Mathew K AnalyticsLesson 45 · Mastering Pandas
Optimizing Large Datasets in Pandas: Efficient Techniques for Data Processing
Working with big data in pandas is powerful, but optimizing for speed and memory is key. In this lesson, we will: Load real-world data. Explore efficient…
- CourseMastering Pandas
- Lesson45 of 44
- Video18 min
- FormatJupyter notebook · 19 code cells
What you'll learn
Data
No separate download needed — the notebook creates or downloads everything it uses.
📓 Full notebook
Download .ipynbHands-On: Optimizing Large Datasets in Pandas#
Working with big data in pandas is powerful, but optimizing for speed and memory is key.
In this lesson, we will:
- Load real-world data.
- Explore efficient data types.
- Use chunking to read large files.
- Apply vectorized operations.
- Save memory with categories.
- Handle missing data cleverly.
- Filter efficiently.
- Aggregate and group smartly.
- Merge and join big datasets.
- Reshape and pivot quickly.
- Use time-series efficiently.
- Visualize with pandas built-ins.
- Practice with a mini Titanic challenge.
- Review best practices and speed tips.
- Tackle troubleshooting tricks.
- Try a hands-on challenge.
Lets begin optimizing your pandas skills!
# Suppress warnings before anything else
import warnings; warnings.filterwarnings("ignore")
import numpy as np
np.random.seed(42)
# Data setup (Titanic Dataset)
import pandas as pd
url = 'https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv'
df = pd.read_csv(url)
print(df.shape)
print(df.head(3))
Why Optimize?#
Big datasets can make pandas slow or crash with memory errors.
- Optimizing lets you work with more data without specialized hardware.
- Good habits prevent frustrating errors down the line.
# Exploring memory usage
print(df.info(memory_usage="deep"))
# Convert object columns to categoricals
for col in ['Sex', 'Embarked', 'Cabin', 'Ticket']:
df[col] = df[col].astype('category')
print(df.dtypes[['Sex', 'Embarked', 'Cabin', 'Ticket']])
# Memory usage after conversion
print(df.info(memory_usage="deep"))
# Downcast numerical types to save space
df['Age'] = pd.to_numeric(df['Age'], downcast='float')
df['Fare'] = pd.to_numeric(df['Fare'], downcast='float')
df['Pclass'] = pd.to_numeric(df['Pclass'], downcast='integer')
print(df[['Age', 'Fare', 'Pclass']].dtypes)
# Handling missing values efficiently
print(df.isnull().sum())
# Fill missing ages with the median age
df['Age'].fillna(df['Age'].median(), inplace=True)
# Fast filtering with query
adults = df.query('Age >= 18')
print(adults[['Name', 'Age']].head())
# Using vectorized operations instead of loops
df['FamilySize'] = df['SibSp'] + df['Parch'] + 1
# Use chunking to load large CSVs (simulate with a small chunk)
chunk_iter = pd.read_csv(url, chunksize=100)
first_chunk = next(chunk_iter)
print(first_chunk.head(2))
Grouping and Aggregation for Big Datasets#
Pandas groupby gives quick summaries, even for millions of rows. We use it to answer questions like 'Average fare by class?' or 'Survival rate by family size.'
# Efficient groupby and aggregation
class_fare = df.groupby('Pclass')['Fare'].mean()
print(class_fare)
# Optimized merging: Join Titanic with itself on FamilySize
df_join = pd.merge(df, df, on='FamilySize', suffixes=('_a', '_b'))
print(df_join[['Name_a', 'FamilySize', 'Name_b']].head(2))
# Pivoting for wide-to-long reshaping
pvt = df.pivot_table(index='FamilySize', values='Survived', aggfunc='mean')
print(pvt.head())
# Speed up string operations using .str methods
df['NameStart'] = df['Name'].str.split(',').str[0]
# Time series: Set up with random sample data
import numpy as np
dates = pd.date_range('2020-01-01', periods=df.shape[0])
df['FakeDate'] = dates
df.set_index('FakeDate', inplace=True)
df['Survived'].resample('M').mean().plot(title='Monthly average survival rate')
Quick Visualization - Value Counts Barplot#
Pandas can quickly plot bar charts for categorical breakdowns. We do not always need a separate library for basic graphs.
# Plot survival by class
df['Survived'].groupby(df['Pclass']).mean().plot(kind='bar', title='Survival Rate by Class')
# Mini-challenge: How many people older than 50 survived?
survived_over_50 = df[(df['Age'] > 50) & (df['Survived'] == 1)]
print(len(survived_over_50))
Best Practice Tips#
- Always check dtypes to save memory.
- Use categories for repeated text.
- Downcast numbers when safe.
- Fill missing data up front.
- Use vectorized and built-in methods.
- Query and filter before merging or plotting.
- Visualize only what you need.
These habits make pandas work smoother and faster.
# Common error: Trying to load too much data at once
try:
giant_df = pd.read_csv(url, nrows=12_000_000)
except Exception as e:
print('Error:', str(e))
# Quick Challenge: Filter all children (under 16) in 1 line
children = df[df['Age'] < 16]
print(children[['Name', 'Age']].head())
Lesson Recap#
Today you learned how to:
- Analyze memory use in pandas.
- Use the right dtypes for speed and space.
- Load large files safely with chunking.
- Leverage vectorized operations.
- Aggregate and merge efficiently.
- Pivot, reshape, and summarize data.
- Handle time series smoothly.
- Catch common errors with big data.
With these skills, you can now approach much larger datasets on your own laptop!
Thanks for following along!
If you found this lesson helpful, please like, subscribe, and leave your pandas optimization tips or questions below.
Happy analyzing!
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.
Related lessons
21 minComprehensive PyArrow Training for Efficient Data Processing in Python
13 minEssential Data Preprocessing Techniques Using Titanic and Iris Datasets in Python
17 minOptimizing Memory Usage with Pandas Categorical Data Types for Efficient Analysis
15 min