Lesson 8 · Mastering Pandas

Exploring and Cleaning Real-World Datasets with Python Pandas: A Practical Guide

Welcome! In this lesson, you will learn how to use Python pandas to explore and analyze real-world datasets. By the end, you will be comfortable: loading…

⬇ Download notebook Open in Colab ↗

CourseMastering Pandas
Lesson8 of 44
Video19 min
FormatJupyter notebook · 15 code cells

NumPy pandas seaborn Matplotlib

Data

No separate download needed — the notebook creates or downloads everything it uses.

📓 Full notebook

Download .ipynb

Exploring Real Datasets with Pandas#

Welcome! In this lesson, you will learn how to use Python pandas to explore and analyze real-world datasets.

By the end, you will be comfortable: loading data, cleaning data, filtering, grouping, joining, reshaping, visualizing, and building mini-projects.

Let us get started!

import warnings; warnings.filterwarnings("ignore")
import numpy as np
np.random.seed(42)
# Suppress all warnings so your notebook stays clean

Step 1: Loading the Titanic Dataset#

We will use the Titanic dataset, a classic for data exploration. It captures information about passengers, such as age, gender, fare, and survival.

Let us load and preview this dataset.

import pandas as pd
url = 'https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv'
df = pd.read_csv(url)
print(df.shape)
print(df.head(3))

(891, 12)
   PassengerId  Survived  Pclass  \
0            1         0       3   
1            2         1       1   
2            3         1       3   

                                                Name     Sex   Age  SibSp  \
0                            Braund, Mr. Owen Harris    male  22.0      1   
1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0      1   
2                             Heikkinen, Miss. Laina  female  26.0      0   

   Parch            Ticket     Fare Cabin Embarked  
0      0         A/5 21171   7.2500   NaN        S  
1      0          PC 17599  71.2833   C85        C  
2      0  STON/O2. 3101282   7.9250   NaN        S

Step 2: Basic Exploration#

Let us quickly check columns, data types, and missing values.

This helps us understand the shape and health of our data.

print(df.columns)
print(df.dtypes)
print(df.isnull().sum())

Index(['PassengerId', 'Survived', 'Pclass', 'Name', 'Sex', 'Age', 'SibSp',
       'Parch', 'Ticket', 'Fare', 'Cabin', 'Embarked'],
      dtype='object')
PassengerId      int64
Survived         int64
Pclass           int64
Name            object
Sex             object
Age            float64
SibSp            int64
Parch            int64
Ticket          object
Fare           float64
Cabin           object
Embarked        object
dtype: object
PassengerId      0
Survived         0
Pclass           0
Name             0
Sex              0
Age            177
SibSp            0
Parch            0
Ticket           0
Fare             0
Cabin          687
Embarked         2
dtype: int64

Step 3: Quick Statistics#

Pandas can describe numerical and text columns for you. Check means, counts, and more with a single function.

Let us see a summary.

print(df.describe(include='all'))

        PassengerId    Survived      Pclass                     Name   Sex  \
count    891.000000  891.000000  891.000000                      891   891   
unique          NaN         NaN         NaN                      891     2   
top             NaN         NaN         NaN  Braund, Mr. Owen Harris  male   
freq            NaN         NaN         NaN                        1   577   
mean     446.000000    0.383838    2.308642                      NaN   NaN   
std      257.353842    0.486592    0.836071                      NaN   NaN   
min        1.000000    0.000000    1.000000                      NaN   NaN   
25%      223.500000    0.000000    2.000000                      NaN   NaN   
50%      446.000000    0.000000    3.000000                      NaN   NaN   
75%      668.500000    1.000000    3.000000                      NaN   NaN   
max      891.000000    1.000000    3.000000                      NaN   NaN   

               Age       SibSp       Parch  Ticket        Fare    Cabin  \
count   714.000000  891.000000  891.000000     891  891.000000      204   
unique         NaN         NaN         NaN     681         NaN      147   
top            NaN         NaN         NaN  347082         NaN  B96 B98   
freq           NaN         NaN         NaN       7         NaN        4   
mean     29.699118    0.523008    0.381594     NaN   32.204208      NaN   
std      14.526497    1.102743    0.806057     NaN   49.693429      NaN   
min       0.420000    0.000000    0.000000     NaN    0.000000      NaN   
25%      20.125000    0.000000    0.000000     NaN    7.910400      NaN   
50%      28.000000    0.000000    0.000000     NaN   14.454200      NaN   
75%      38.000000    1.000000    0.000000     NaN   31.000000      NaN   
max      80.000000    8.000000    6.000000     NaN  512.329200      NaN   

       Embarked  
count       889  
unique        3  
top           S  
freq        644  
mean        NaN  
std         NaN  
min         NaN  
25%         NaN  
50%         NaN  
75%         NaN  
max         NaN

Step 4: Cleaning Data#

Messy data is very common. Let us start simple: fill missing age values.

We will fill them with the average age.

mean_age = df['Age'].mean()
df['Age'].fillna(mean_age, inplace=True)
print(df['Age'].isnull().sum())

Step 5: Selecting and Filtering#

Let us look for passengers who paid high fares.

Filtering helps you focus on interesting data.

rich = df[df['Fare'] > 100]
print(rich[['Name', 'Fare']].head())

                                               Name      Fare
27                   Fortune, Mr. Charles Alexander  263.0000
31   Spencer, Mrs. William Augustus (Marie Eugenie)  146.5208
88                       Fortune, Miss. Mabel Helen  263.0000
118                        Baxter, Mr. Quigg Edmond  247.5208
195                            Lurette, Miss. Elise  146.5208

Step 6: Grouping and Aggregation#

Pandas makes it easy to group data and calculate statistics by group.

Let us find the survival rate by passenger class.

grouped = df.groupby('Pclass')['Survived'].mean()
print(grouped)

Pclass
1    0.629630
2    0.472826
3    0.242363
Name: Survived, dtype: float64

Step 7: Joining DataFrames#

You often work with several tables of data. Let us join two DataFrames together on a shared column.

We will make a little demo passengers table and join on the Name.

demo_passengers = pd.DataFrame({
    'Name': [df.iloc[0]['Name'], df.iloc[1]['Name']],
    'Extra_Info': ['Friend', 'Colleague']
})
merged = pd.merge(df, demo_passengers, on='Name', how='left')
print(merged[['Name', 'Extra_Info']].head(3))

                                                Name Extra_Info
0                            Braund, Mr. Owen Harris     Friend
1  Cumings, Mrs. John Bradley (Florence Briggs Th...  Colleague
2                             Heikkinen, Miss. Laina        NaN

Step 8: Pivot Tables and Reshaping#

Pivot tables reshape your data, making summaries by groups.

Let us see the average age by sex and class.

pivot = df.pivot_table(index='Sex', columns='Pclass', values='Age', aggfunc='mean')
print(pivot)

Pclass          1          2          3
Sex                                    
female  34.141405  28.748661  24.068493
male    39.287717  30.653908  27.372153

Step 9: Time-Series Handling#

Suppose our Titanic data included a Date column. For now, let us use a real time-series example: the Flights dataset.

We will load it and plot monthly airline passengers over time.

import seaborn as sns
flights_df = sns.load_dataset('flights')
print(flights_df.shape)
print(flights_df.head(3))

(144, 3)
   year month  passengers
0  1949   Jan         112
1  1949   Feb         118
2  1949   Mar         132

import matplotlib.pyplot as plt
monthly = flights_df.pivot_table(index='year', columns='month', values='passengers')
flights_df['date'] = pd.to_datetime(flights_df['year'].astype(str) + '-' + flights_df['month'].astype(str) + '-01')
plt.figure(figsize=(8,4))
plt.plot(flights_df['date'], flights_df['passengers'])
plt.xlabel('Date')
plt.ylabel('Passengers')
plt.title('Monthly Airline Passengers Over Time')
plt.tight_layout()
plt.show()

No description has been provided for this image

Step 10: Visualization with Pandas#

Pandas can also plot data directly.

Let us visualize the number of survivors by sex from the Titanic dataset.

df.groupby('Sex')['Survived'].sum().plot(kind='bar', color=['skyblue', 'lightcoral'])
plt.ylabel('Survivors')
plt.title('Survivors by Sex')
plt.show()

Step 11: Mini-Project Part 1 Exploratory Data Analysis (Titanic)#

Now let us do a hands-on mini project.

We will answer: What are some factors linked with survival? Which age group survived most?

Let us explore age, class, and survival together.

df['AgeGroup'] = pd.cut(df['Age'], bins=[0,12,18,40,80], labels=['Child','Teen','Adult','Senior'])
grouped_survival = df.groupby(['AgeGroup','Pclass'])['Survived'].mean().unstack()
print(grouped_survival)

# Visualize survival rates by age group and class
grouped_survival.plot(kind='bar', figsize=(8,5))
plt.ylabel('Survival Rate')
plt.title('Survival Rate by Age Group and Class')
plt.show()

Pclass           1         2         3
AgeGroup                              
Child     0.750000  1.000000  0.416667
Teen      0.916667  0.500000  0.282609
Adult     0.669355  0.421488  0.232493
Senior    0.513158  0.382353  0.075000

Step 12: Mini-Project Part 2 Feature Engineering#

Let us try creating a new feature: FamilySize. Then, we can check how family size relates to survival.

This is common when preparing data for machine learning.

df['FamilySize'] = df['SibSp'] + df['Parch'] + 1
print(df[['SibSp', 'Parch', 'FamilySize']].head())
family_survival = df.groupby('FamilySize')['Survived'].mean()
print(family_survival)

   SibSp  Parch  FamilySize
0      1      0           2
1      1      0           2
2      0      0           1
3      1      0           2
4      0      0           1
FamilySize
1     0.303538
2     0.552795
3     0.578431
4     0.724138
5     0.200000
6     0.136364
7     0.333333
8     0.000000
11    0.000000
Name: Survived, dtype: float64

Step 13: Best Practices and Performance Tips#

Always check for missing or strange data right away.
Use .copy() if you want a real copy of a DataFrame, not a reference.
For large datasets, try .info(), .head(), and .sample() to save time.
Chain commands in separate steps for clarity.
Use vectorized operations over loops; they are much faster in pandas.

Step 14: Troubleshooting Common Errors#

Common problems:

KeyError: Check for typos in column names.
ValueError when reshaping: Watch your index and column labels.
SettingWithCopyWarning: Use .loc or .copy() to avoid silent bugs.

If stuck, try printing shapes and dtypes, or use df.sample(5) for a quick sanity check.

try:
    print(df['NotARealColumn'].head())
except KeyError as e:
    print(f"Error: {e}")

Error: 'NotARealColumn'

Step 15: Challenge Exercises#

Can you find the top 3 oldest survivors?
Make a new column for fare per person (Fare divided by FamilySize).
Explore another public dataset and produce a simple pivot table.

Pause the video and give these a try!

Recap#

Congratulations on completing this lesson!

You have loaded, explored, cleaned, filtered, grouped, reshaped, and visualized real-world data. You are ready to start your own projects.

Keep practicing and experimenting.

Want more tutorials?#

Subscribe to the channel and leave a comment with your favorite dataset or question you want answered next!

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.

☕ Buy me a coffee Other ways to help