Lesson 69 · Mastering Pandas

Comprehensive Pandas Tutorial in Python: Data Analysis from Beginner to Advanced

Welcome to this hands-on Pandas tutorial! We will start from the basics and work our way through real-world data preparation, analysis, and visualization.…

⬇ Download notebook Open in Colab ↗

CourseMastering Pandas
Lesson69 of 44
Video36 min
FormatJupyter notebook · 33 code cells

seaborn pandas NumPy Matplotlib

Data

No separate download needed — the notebook creates or downloads everything it uses.

📓 Full notebook

Download .ipynb

Mastering Pandas: From Basics to Advanced#

Welcome to this hands-on Pandas tutorial! We will start from the basics and work our way through real-world data preparation, analysis, and visualization.

We will use the Titanic and Tips datasets. Both datasets come with Seaborn, so there is nothing extra to download.

You will learn how to explore, clean, transform, combine, and visualize data using Pandas.

Let us begin our journey!

Section 1: Data Setup and Exploration#

Let us load our first dataset (Titanic).

import seaborn as sns
import pandas as pd
import warnings
import numpy as np
np.random.seed(42)
warnings.filterwarnings("ignore")

# Load the Titanic dataset
df = sns.load_dataset("titanic")

# Preview the dataset
print("Shape:", df.shape)
print(df.head(3))

Shape: (891, 15)
   survived  pclass     sex   age  sibsp  parch     fare embarked  class  \
0         0       3    male  22.0      1      0   7.2500        S  Third   
1         1       1  female  38.0      1      0  71.2833        C  First   
2         1       3  female  26.0      0      0   7.9250        S  Third   

     who  adult_male deck  embark_town alive  alone  
0    man        True  NaN  Southampton    no  False  
1  woman       False    C    Cherbourg   yes  False  
2  woman       False  NaN  Southampton   yes   True

Exploring the DataFrame Structure#

Let us check the column names and data types in our dataset.

print("Columns:", df.columns.tolist())
print("\nData types:")
print(df.dtypes)

Columns: ['survived', 'pclass', 'sex', 'age', 'sibsp', 'parch', 'fare', 'embarked', 'class', 'who', 'adult_male', 'deck', 'embark_town', 'alive', 'alone']

Data types:
survived          int64
pclass            int64
sex              object
age             float64
sibsp             int64
parch             int64
fare            float64
embarked         object
class          category
who              object
adult_male         bool
deck           category
embark_town      object
alive            object
alone              bool
dtype: object

# How many unique values are in the "embarked" column?
print(df["embarked"].unique())

['S' 'C' 'Q' nan]

# Check basic statistics for numeric columns
print(df.describe())

         survived      pclass         age       sibsp       parch        fare
count  891.000000  891.000000  714.000000  891.000000  891.000000  891.000000
mean     0.383838    2.308642   29.699118    0.523008    0.381594   32.204208
std      0.486592    0.836071   14.526497    1.102743    0.806057   49.693429
min      0.000000    1.000000    0.420000    0.000000    0.000000    0.000000
25%      0.000000    2.000000   20.125000    0.000000    0.000000    7.910400
50%      0.000000    3.000000   28.000000    0.000000    0.000000   14.454200
75%      1.000000    3.000000   38.000000    1.000000    0.000000   31.000000
max      1.000000    3.000000   80.000000    8.000000    6.000000  512.329200

Section 2: Data Cleaning and Preprocessing#

It is important to clean your data before analysis.

We will drop missing values for some basic tasks.

# Drop rows with any missing values
df_clean = df.dropna()

# Preview the cleaned data
print("Original shape:", df.shape)
print("Cleaned shape:", df_clean.shape)

Original shape: (891, 15)
Cleaned shape: (182, 15)

# Fill missing 'age' values with the median
df_filled = df.copy()
median_age = df_filled['age'].median()
df_filled['age'] = df_filled['age'].fillna(median_age)
print(df_filled['age'].isnull().sum())

Checking for Duplicates#

Duplicate rows can cause problems in analysis.

Let us see if there are any duplicates.

print("Number of duplicate rows:", df.duplicated().sum())

Number of duplicate rows: 107

Section 3: Filtering and Conditional Selection#

Let us work with the Tips dataset and practice selecting some rows.

import seaborn as sns
import pandas as pd
import warnings
warnings.filterwarnings("ignore")

# Load the Tips dataset
tips = sns.load_dataset("tips")

# Preview the dataset
print("Shape:", tips.shape)
print(tips.head(3))

Shape: (244, 7)
   total_bill   tip     sex smoker  day    time  size
0       16.99  1.01  Female     No  Sun  Dinner     2
1       10.34  1.66    Male     No  Sun  Dinner     3
2       21.01  3.50    Male     No  Sun  Dinner     3

# Select only the records where the tip is greater than $5
big_tips = tips[tips["tip"] > 5]
print(big_tips)

     total_bill    tip     sex smoker   day    time  size
23        39.42   7.58    Male     No   Sat  Dinner     4
44        30.40   5.60    Male     No   Sun  Dinner     4
47        32.40   6.00    Male     No   Sun  Dinner     4
52        34.81   5.20  Female     No   Sun  Dinner     4
59        48.27   6.73    Male     No   Sat  Dinner     4
85        34.83   5.17  Female     No  Thur   Lunch     4
88        24.71   5.85    Male     No  Thur   Lunch     2
116       29.93   5.07    Male     No   Sun  Dinner     4
141       34.30   6.70    Male     No  Thur   Lunch     6
155       29.85   5.14  Female     No   Sun  Dinner     5
170       50.81  10.00    Male    Yes   Sat  Dinner     3
172        7.25   5.15    Male    Yes   Sun  Dinner     2
181       23.33   5.65    Male    Yes   Sun  Dinner     2
183       23.17   6.50    Male    Yes   Sun  Dinner     4
211       25.89   5.16    Male    Yes   Sat  Dinner     4
212       48.33   9.00    Male     No   Sat  Dinner     4
214       28.17   6.50  Female    Yes   Sat  Dinner     3
239       29.03   5.92    Male     No   Sat  Dinner     3

# Find all female customers who had dinner
dinner_female = tips[(tips['sex'] == 'Female') & (tips['time'] == 'Dinner')]
print(dinner_female.head())

    total_bill   tip     sex smoker  day    time  size
0        16.99  1.01  Female     No  Sun  Dinner     2
4        24.59  3.61  Female     No  Sun  Dinner     4
11       35.26  5.00  Female     No  Sun  Dinner     4
14       14.83  3.02  Female     No  Sun  Dinner     2
16       10.33  1.67  Female     No  Sun  Dinner     3

# Using isin to select multiple days
weekend_tips = tips[tips['day'].isin(['Sat', 'Sun'])]
print(weekend_tips.head())

   total_bill   tip     sex smoker  day    time  size
0       16.99  1.01  Female     No  Sun  Dinner     2
1       10.34  1.66    Male     No  Sun  Dinner     3
2       21.01  3.50    Male     No  Sun  Dinner     3
3       23.68  3.31    Male     No  Sun  Dinner     2
4       24.59  3.61  Female     No  Sun  Dinner     4

Section 4: GroupBy and Aggregation#

Pandas GroupBy lets us split data into groups and do calculations.

Let us see the average tip by day.

# Average tip by day
avg_tip_by_day = tips.groupby('day')["tip"].mean()
print(avg_tip_by_day)

day
Thur    2.771452
Fri     2.734737
Sat     2.993103
Sun     3.255132
Name: tip, dtype: float64

# Multiple aggregations: average and max tip per sex
grouped = tips.groupby('sex')['tip'].agg(['mean', 'max'])
print(grouped)

            mean   max
sex                   
Male    3.089618  10.0
Female  2.833448   6.5

# Count the number of tips for each size of group
count_by_size = tips.groupby('size').size()
print(count_by_size)

size
1      4
2    156
3     38
4     37
5      5
6      4
dtype: int64

Section 5: Joining and Merging DataFrames#

Imagine you have more than one table and you need to combine them.

Let us create two small DataFrames and merge them.

# Create two simple DataFrames
left = pd.DataFrame({"id": [1, 2, 3], "name": ["Alice", "Bob", "Cathy"]})
right = pd.DataFrame({"id": [2, 3, 4], "age": [24, 27, 22]})

# Merge on 'id'
merged = pd.merge(left, right, on="id", how="inner")
print(merged)

   id   name  age
0   2    Bob   24
1   3  Cathy   27

# Merge Titanic data with new ages DataFrame
ages = pd.DataFrame({"age": [22, 38, 26], "new_col": ["x", "y", "z"]})
merged_titanic = pd.merge(df.head(3), ages, on="age", how="left")
print(merged_titanic)

   survived  pclass     sex   age  sibsp  parch     fare embarked  class  \
0         0       3    male  22.0      1      0   7.2500        S  Third   
1         1       1  female  38.0      1      0  71.2833        C  First   
2         1       3  female  26.0      0      0   7.9250        S  Third   

     who  adult_male deck  embark_town alive  alone new_col  
0    man        True  NaN  Southampton    no  False       x  
1  woman       False    C    Cherbourg   yes  False       y  
2  woman       False  NaN  Southampton   yes   True       z

Section 6: Pivot Tables and Reshaping#

Pivot tables help you reorganize and summarize data quickly.

Let us create a pivot table with the Tips dataset.

# Pivot table: average tip by sex and day
pivot = pd.pivot_table(tips, values='tip', index='sex', columns='day', aggfunc='mean')
print(pivot)

day         Thur       Fri       Sat       Sun
sex                                           
Male    2.980333  2.693000  3.083898  3.220345
Female  2.575625  2.781111  2.801786  3.367222

# Melt to go long-form: unwind columns
melted = pd.melt(tips, id_vars=['day'], value_vars=['total_bill', 'tip'])
print(melted.head())

   day    variable  value
0  Sun  total_bill  16.99
1  Sun  total_bill  10.34
2  Sun  total_bill  21.01
3  Sun  total_bill  23.68
4  Sun  total_bill  24.59

Section 7: Time-Series Handling#

Pandas makes working with dates and times much easier.

Let us create a date column and plot a trend.

import numpy as np
import matplotlib.pyplot as plt
tips['visit_date'] = pd.date_range('2021-01-01', periods=len(tips), freq='D')

# Show what our new column looks like
print(tips[['visit_date', 'total_bill']].head())

  visit_date  total_bill
0 2021-01-01       16.99
1 2021-01-02       10.34
2 2021-01-03       21.01
3 2021-01-04       23.68
4 2021-01-05       24.59

# Convert visit_date to datetime if needed
tips['visit_date'] = pd.to_datetime(tips['visit_date'])

plt.figure(figsize=(8,3))
plt.plot(tips['visit_date'], tips['total_bill'], marker='o', linestyle='-')
plt.title('Total Bill Over Time')
plt.xlabel('Visit Date')
plt.ylabel('Total Bill ($)')
plt.tight_layout()
plt.show()

No description has been provided for this image

Section 8: Visualization with Pandas#

Let us use Pandas built-in plotting to visualize simple trends.

# Histogram of total bill amounts
tips['total_bill'].plot.hist(bins=20, alpha=0.7)
plt.title('Histogram of Total Bill')
plt.xlabel('Total Bill ($)')
plt.show()

# Box plot of tip by smoker status
tips.boxplot(column='tip', by='smoker')
plt.title('Tip by Smoker')
plt.suptitle('')
plt.xlabel('Smoker')
plt.ylabel('Tip Amount')
plt.show()

Section 9: Mini-Project Part 1 (Exploratory Data Analysis with Titanic)#

Now you will use your skills to look for patterns in the Titanic data.

# Percent of passengers who survived
survival_rate = df['survived'].mean() * 100
print(f"Survival rate: {survival_rate:.1f}%")

Survival rate: 38.4%

# Plot survival rate by sex
df.groupby('sex')['survived'].mean().plot(kind='bar')
plt.title('Survival Rate by Sex')
plt.ylabel('Survival Rate')
plt.ylim(0,1)
plt.show()

# Crosstab for class and survival
cross = pd.crosstab(df['pclass'], df['survived'], normalize='index')
cross.plot(kind='bar', stacked=True)
plt.title('Survival by Ticket Class')
plt.ylabel('Proportion')
plt.xlabel('Ticket Class')
plt.show()

Section 10: Mini-Project Part 2 (Feature Engineering & Deeper Analysis)#

Let us create a new feature to see if families survived together.

# Create a family_size feature
df['family_size'] = df['sibsp'] + df['parch'] + 1

# Check if large families had different survival rates
df['is_large_family'] = df['family_size'] >= 5
print(df.groupby('is_large_family')['survived'].mean())

is_large_family
False    0.400483
True     0.161290
Name: survived, dtype: float64

# Scatter plot of fare vs. age, colored by survival
df.plot.scatter(x='age', y='fare', c='survived', colormap='viridis', alpha=0.7)
plt.title('Fare vs. Age Colored by Survival')
plt.xlabel('Age')
plt.ylabel('Fare')
plt.show()

Section 11: Best Practices and Performance Tips#

Learn how to work efficiently with large data.

# Use .copy() to avoid changing the original DataFrame
tips_copy = tips.copy()
tips_copy['total_bill'] = tips_copy['total_bill'] * 1.1

# Downcast data types for memory savings
tips_small = tips.copy()
tips_small['size'] = pd.to_numeric(tips_small['size'], downcast='unsigned')
print(tips_small['size'].dtype)

uint8

Section 12: Troubleshooting Common Pandas Errors#

Let us see some common errors and how to fix them.

# Try to use a column name that does not exist
try:
    print(tips['total_tip'])
except KeyError as e:
    print("Column not found:", e)

Column not found: 'total_tip'

# Check data types before math
try:
    result = tips['day'] + 5
except TypeError as e:
    print("Cannot add a number to text:", e)

Cannot add a number to text: unsupported operand type(s) for +: 'Categorical' and 'int'

Section 13: Challenge Exercises#

Test your new skills with a fun short challenge!

# EXERCISE: What is the average tip for each time of day?
print(tips.groupby('time')['tip'].mean())

time
Lunch     2.728088
Dinner    3.102670
Name: tip, dtype: float64

# EXERCISE: Create a DataFrame of only non-smokers at tables of 4 or more
large_nonsmokers = tips[(tips['smoker'] == 'No') & (tips['size'] >= 4)]
print(large_nonsmokers.head())

    total_bill   tip     sex smoker  day    time  size visit_date
4        24.59  3.61  Female     No  Sun  Dinner     4 2021-01-05
5        25.29  4.71    Male     No  Sun  Dinner     4 2021-01-06
7        26.88  3.12    Male     No  Sun  Dinner     4 2021-01-08
11       35.26  5.00  Female     No  Sun  Dinner     4 2021-01-12
13       18.43  3.00    Male     No  Sun  Dinner     4 2021-01-14

Recap - What Have We Learned?#

We covered a lot about Pandas today!

Data loading, exploring, and cleaning
Conditional selection and grouping
Joining, reshaping, visualizing, and more

Keep practicing, and these skills will soon feel natural.

See you in the next lesson!

Thank You!#

If you enjoyed this video and want more tutorials, please like and subscribe to our channel.

See you next time!

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.

☕ Buy me a coffee Other ways to help