Mathew K AnalyticsLesson 1 · Pandas Full Course
Learn Pandas Fast: Answer a Real Data Question in 20 Minutes | Pandas Course #1
In this lesson you'll be able to: Explain what pandas is and why analysts use it Load a real CSV file and inspect it in seconds Find and handle missing…
- CoursePandas Full Course
- Lesson1 of 65
- Video23 min
- FormatJupyter notebook · 46 code cells
- Data1 dataset
What you'll learn
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
- titanic.csv preview58.9 KB
📓 Full notebook
Download .ipynbPandas 01: Introduction to Pandas and its Role in Data Analysis#
In this lesson you'll be able to:
- Explain what pandas is and why analysts use it
- Load a real CSV file and inspect it in seconds
- Find and handle missing values
- Filter, group and chart data to answer a question Dataset: the Titanic passenger list (data/pandas/titanic.csv)
Warm-up: the same question in plain Python#
What share of passengers survived? No pandas yet.
import csv
with open("data/pandas/titanic.csv") as f:
rows = list(csv.DictReader(f))
print(len(rows), "passengers")
print(rows[0]["Name"], "| age:", rows[0]["Age"])
survived = [int(r["Survived"]) for r in rows]
print(sum(survived) / len(survived))
Load a real table in one line#
pandas reads the file, works out each column's type and labels it.
import pandas as pd
df = pd.read_csv("data/pandas/titanic.csv")
print(type(df))
print(df.shape)
df.head()
df["Survived"].mean().round(3)
Where pandas fits#
Files in, then pandas to clean and summarise, then charts and models. Under the hood, each column is a fast NumPy array.
fares = df["Fare"].to_numpy()
print(type(fares))
print(fares[:4])
print(df["Fare"].sum().round(2))
print((df["Fare"] * 2).head(3).tolist())
Common mistake: giving read_csv just the file name
name = "titanic.csv"
pd.read_csv(name)
df = pd.read_csv("data/pandas/titanic.csv")
df.shape
Inspect: what is in this table?#
Check column names, types and value ranges before any analysis.
df.tail(3)
df.columns.tolist()
df.info()
df.describe().round(2)
Quick check: is
df["Age"]a table or something else?
ages = df["Age"]
print(type(ages))
ages.head(3)
print(df["Sex"].unique())
print(df["Ticket"].nunique(), "different tickets")
df["Pclass"].value_counts().sort_index()
df["Embarked"].value_counts()
Clean: find and handle missing values#
Real data has gaps. Count them, then drop or fill with care.
df.isna().sum()
Quick check: what share of all ages is missing?
df["Age"].isna().mean().round(3)
with_age = df.dropna(subset=["Age"])
print(df.shape, with_age.shape)
top_port = df["Embarked"].mode()[0]
df["Embarked"] = df["Embarked"].fillna(top_port)
print(top_port, df["Embarked"].isna().sum())
Your turn: how many passengers have no recorded cabin?
df["Cabin"].isna().sum()
Ask questions: select and filter#
A condition gives True or False per row; keep the True ones.
df[["Name", "Age", "Fare"]].head(3)
kids = df[df["Age"] < 18]
print(len(kids), "children")
kids[["Name", "Age", "Pclass"]].head(3)
top = df.sort_values("Fare", ascending=False)
top[["Name", "Pclass", "Fare"]].head(3)
ports = ["C", "Q"]
not_s = df[df["Embarked"].isin(ports)]
print(len(not_s), "boarded at C or Q")
Your turn: how many passengers paid a fare above 100?
rich = df[df["Fare"] > 100]
print(len(rich), "passengers paid over 100")
print(rich["Pclass"].unique())
Common mistake: combining conditions with
and
df[(df["Sex"] == "female") and (df["Pclass"] == 1)]
first_women = df[(df["Sex"] == "female") & (df["Pclass"] == 1)]
print(len(first_women), "women in first class")
first_women["Survived"].mean().round(2)
Summarise: group and compare#
groupby splits rows into groups, then summarises each one.
df.groupby("Sex")["Survived"].mean().round(2)
df.groupby("Pclass")["Fare"].mean().round(2)
Quick check: which class had the best survival rate?
df.groupby("Pclass")["Survived"].mean().round(2)
by_port = df.groupby("Embarked")["Survived"].mean()
by_port.sort_values(ascending=False).round(2)
df.groupby("Embarked")["Fare"].agg(["mean", "median"]).round(2)
Your turn: what was the median age in each class?
df.groupby("Pclass")["Age"].median()
table = df.pivot_table(values="Survived", index="Pclass",
columns="Sex", aggfunc="mean")
table.round(2)
Visualise: let a chart tell the story#
pandas hands its numbers straight to matplotlib for plotting.
import matplotlib.pyplot as plt
rates = df.groupby("Pclass")["Survived"].mean()
ax = rates.plot(kind="bar", color="steelblue", rot=0)
ax.set_title("Titanic survival rate by class")
ax.set_xlabel("Passenger class")
ax.set_ylabel("Share who survived")
plt.show()
ax = df.boxplot(column="Age", by="Pclass", grid=False)
ax.set_title("Age by passenger class")
plt.suptitle("")
ax.set_xlabel("Passenger class")
ax.set_ylabel("Age in years")
plt.show()
Common mistake: a column name with the wrong capital letter
df["age"].mean()
print([c for c in df.columns if c.lower() == "age"])
df["Age"].mean().round(1)
Mini-project: did women and children go first?#
Load, flag, group and compare, all with today's tools.
df["Group"] = "Man"
df.loc[df["Sex"] == "female", "Group"] = "Woman"
df.loc[df["Age"] < 16, "Group"] = "Child"
df["Group"].value_counts()
summary = df.groupby("Group")["Survived"].agg(["count", "mean"])
summary.round(2).sort_values("mean", ascending=False)
by_class = df.pivot_table(values="Survived", index="Group",
columns="Pclass", aggfunc="mean")
by_class.round(2)
pd.crosstab(df["Group"], df["Pclass"])
ax = by_class.T.plot(kind="bar", rot=0)
ax.set_title("Survival by group and class")
ax.set_xlabel("Passenger class")
ax.set_ylabel("Share who survived")
plt.show()
import os
os.makedirs("output/pandas", exist_ok=True)
path = "output/pandas/l01_group_survival.csv"
by_class.round(2).to_csv(path)
pd.read_csv(path, index_col=0)
Finding: women and children did go first, but mostly in first and second class. In third class, half of the women and children died.
Recap#
- pandas turns files into DataFrames: labelled rows and columns
head,infoanddescribeinspect a table in secondsisna,dropnaandfillnafind and handle missing values- Filter with conditions joined by
&, neverand groupby,pivot_tableandplotturn data into answers
Challenges#
- What was the survival rate at each port in
Embarked? - Find the oldest passenger who survived. Which class were they in?
- Chart the average fare for the Man, Woman and Child groups.
What's next#
Lesson 02: Installing Pandas and Setting Up the Environment. Check versions, fix paths and set up a project you can rerun.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.
Related lessons
26 minData Mining with Python: pandas & scikit-learn Crash Course
18 minData Cleaning in Python Pandas: Audit a Real Retail Dataset | Data Analytics #1
1 h 30 minPandas for Absolute Beginners (with Real Data)
18 min