Mathew K AnalyticsLesson 15 · Pandas Full Course
Pandas Mini Project: Analyse the Titanic Dataset From Scratch | Pandas Course #15
In this lesson you'll be able to: Inspect and tidy a real dataset before you analyse it Answer questions with filters, query and isin Sort, rank and slice…
- CoursePandas Full Course
- Lesson15 of 65
- FormatJupyter notebook · 47 code cells
- Data1 dataset
What you'll learn
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
- titanic.csv preview58.9 KB
📓 Full notebook
Download .ipynbPandas 15: Mini Project - Analysing Titanic Dataset Basics#
In this lesson you'll be able to:
- Inspect and tidy a real dataset before you analyse it
- Answer questions with filters,
queryandisin - Sort, rank and slice to find the stories in the data
- Test a famous claim: was it really women and children first? Data: the Titanic passenger list, 891 passengers
Warm-up: a cross-section from last lesson#
In lesson 14 we sliced a MultiIndex with xs. Same trick, new data.
import pandas as pd
df = pd.read_csv("data/pandas/titanic.csv")
by_sex = df.set_index(["Sex", "Pclass"]).sort_index()
women = by_sex.xs("female", level="Sex")
print(len(women), "women on board")
Step 1: get to know the data#
Before answering anything, check the size, the columns and the types.
print(df.shape)
df.dtypes
df.iloc[0]
df = df.rename(columns={"Pclass": "Class", "SibSp": "Siblings",
"Parch": "Parents"})
print(df.columns.tolist())
Quick check: what share of passengers do you think survived?
print(df["Survived"].value_counts())
print("Survival rate:", round(df["Survived"].mean(), 3))
df[["Age", "Fare", "Siblings", "Parents"]].describe().round(1)
df.isna().sum().sort_values(ascending=False).head(3)
Step 2: filter to answer questions#
Boolean masks, query and isin turn questions into code.
is_woman = df["Sex"] == "female"
print("Women:", round(df.loc[is_woman, "Survived"].mean(), 3))
print("Men: ", round(df.loc[~is_woman, "Survived"].mean(), 3))
is_child = df["Age"] < 16
print(is_child.sum(), "children under 16")
print("Children:", round(df.loc[is_child, "Survived"].mean(), 3))
twenties = df[df["Age"].between(20, 29)]
print(len(twenties), "passengers in their twenties")
print("Survival:", round(twenties["Survived"].mean(), 3))
Common mistake: combining masks without brackets.
df[df["Sex"] == "female" & df["Age"] < 16]
girls = df[(df["Sex"] == "female") & (df["Age"] < 16)]
print(len(girls), "girls")
print("Survival:", round(girls["Survived"].mean(), 3))
rich_women = df.query("Class == 1 and Sex == 'female'")
print(len(rich_women), "first class women")
print("Survival:", round(rich_women["Survived"].mean(), 3))
irish_french = df[df["Embarked"].isin(["C", "Q"])]
irish_french["Embarked"].value_counts()
for port in ["S", "C", "Q"]:
rate = df.loc[df["Embarked"] == port, "Survived"].mean()
print(port, round(rate, 3))
Your turn: what was the survival rate of men in first class?
first_men = df.query("Class == 1 and Sex == 'male'")
print(len(first_men), "men,",
round(first_men["Survived"].mean(), 3), "survived")
Step 3: build a tidy working table#
Keep the useful columns, set a meaningful index, add new columns.
keep = ["PassengerId", "Name", "Class", "Sex", "Age", "Siblings",
"Parents", "Fare", "Embarked", "Survived"]
tidy = df[keep].set_index("PassengerId")
print(tidy.shape)
tidy.iloc[:3, 1:6]
tidy.loc[[1, 2, 3], ["Class", "Sex", "Fare", "Survived"]]
tidy["Family"] = tidy["Siblings"] + tidy["Parents"] + 1
tidy["Family"].value_counts().sort_index().head(5)
bins = [0, 15, 59, 100]
labels = ["child", "adult", "senior"]
tidy["AgeGroup"] = pd.cut(tidy["Age"], bins=bins, labels=labels)
tidy["AgeGroup"].value_counts(dropna=False)
Common mistake: using an old column name after renaming.
tidy["Pclass"].value_counts()
tidy["Class"].value_counts().sort_index()
Your turn: what share of passengers travelled alone?
alone = tidy["Family"] == 1
print(round(alone.mean(), 3), "travelled alone")
print(round(tidy.loc[alone, "Survived"].mean(), 3), "of them survived")
Step 4: sort and rank to find stories#
Who paid the most, and did money buy a seat in a lifeboat?
cols = ["Name", "Class", "Fare", "Survived"]
tidy.nlargest(5, "Fare")[cols]
tidy["Fare_pct"] = tidy["Fare"].rank(pct=True)
top_q = tidy["Fare_pct"] > 0.75
print("Top quarter: ", round(tidy.loc[top_q, "Survived"].mean(), 3))
print("Bottom quarter:", round(tidy.loc[tidy["Fare_pct"] <= 0.25,
"Survived"].mean(), 3))
quarters = ["cheapest", "lower", "upper", "dearest"]
tidy["FareBand"] = pd.qcut(tidy["Fare"], 4, labels=quarters)
band = tidy.groupby("FareBand", observed=True)["Survived"]
band.mean().round(3)
by_age = tidy.sort_values("Age", ascending=False)
by_age[["Name", "Class", "Age", "Survived"]].head(5)
Common mistake: asking
nlargestfor a text column.
tidy.nlargest(3, "Name")
tidy.sort_values("Name", ascending=False)["Name"].head(3)
big = tidy.sort_values(["Family", "Fare"], ascending=False)
big[["Name", "Family", "Survived"]].head(4)
Quick check: do small families survive more often than solo travellers?
fam = tidy.groupby("Family")["Survived"].agg(["size", "mean"])
fam.round(2)
Quick check: among children, did the youngest survive more often?
kids = tidy[tidy["AgeGroup"] == "child"]
young = kids["Age"] <= 5
print("Age 5 and under:", round(kids.loc[young, "Survived"].mean(), 3))
print("Age 6 to 15: ", round(kids.loc[~young, "Survived"].mean(), 3))
Step 5: slice survival by class and sex#
A MultiIndex table lets us read any combination in one line.
rates = tidy.groupby(["Class", "Sex"])["Survived"].mean()
rates.round(2)
tidy.groupby(["Class", "Sex"]).size().unstack("Sex")
rates.xs("female", level="Sex").round(2)
Your turn: use
xsto get men's survival in each class.
rates.xs("male", level="Sex").round(2)
table = rates.unstack("Sex").round(2)
table["gap"] = table["female"] - table["male"]
table
Quick check: who survived more often, a 3rd class woman or a 1st class man?
print("3rd class women:", rates.loc[(3, "female")].round(2))
print("1st class men: ", rates.loc[(1, "male")].round(2))
import matplotlib.pyplot as plt
ax = table[["female", "male"]].plot.bar(figsize=(7, 4), rot=0)
ax.set_title("Titanic survival rate by class and sex")
ax.set_xlabel("Passenger class")
ax.set_ylabel("Share who survived")
plt.show()
Mini-project finale: women and children first?#
Did the famous rule hold, and did it hold equally in every class?
tidy["Group"] = "man"
tidy.loc[tidy["Sex"] == "female", "Group"] = "woman"
tidy.loc[tidy["AgeGroup"] == "child", "Group"] = "child"
tidy["Group"].value_counts()
tidy.groupby("Group")["Survived"].mean().sort_values().round(3)
tidy.groupby("Survived")["Age"].median()
grid = tidy.pivot_table(index="Class", columns="Group",
values="Survived", aggfunc="mean")
grid[["child", "woman", "man"]].round(2)
ax = grid[["child", "woman", "man"]].plot.bar(figsize=(7, 4), rot=0)
ax.set_title("Survival by class: children, women and men")
ax.set_xlabel("Passenger class")
ax.set_ylabel("Share who survived")
plt.show()
third_kids = tidy.query("Class == 3 and Group == 'child'")
print(len(third_kids), "third class children")
print(third_kids["Survived"].value_counts().sort_index())
kid_sex = third_kids.groupby("Sex")["Survived"].agg(["size", "mean"])
kid_sex.round(2)
import os
os.makedirs("output/pandas", exist_ok=True)
tidy.to_csv("output/pandas/l15_titanic_tidy.csv")
back = pd.read_csv("output/pandas/l15_titanic_tidy.csv",
index_col="PassengerId")
print(back.shape)
Finding: women and children first largely held, but class decided how well. In 1st and 2nd class nearly all women and children lived; in 3rd, fewer than half.
Recap#
- Start every project with
shape,dtypes,headandisna().sum() - Masks need brackets:
(a) & (b);queryandisinread clearly - A tidy table: chosen columns, a meaningful index, new columns
nlargestandrank(pct=True)reveal who paid most, and why it matteredxs,unstackand tuples slice a two-key table in any direction
Challenges#
- Repeat the finale for each port. Did Cherbourg passengers fare better?
- Rank fares within each class. Did top-fare passengers survive more?
- Take the first letter of
Cabinas the deck. Which deck survived best?
What's next#
Lesson 16: Identifying and Handling Missing Data. 177 ages were missing today. Next, we find and fix gaps properly.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.
Related lessons
18 minLesson 1
Analyse E-Commerce Orders with Python Pandas | Full Project Tutorial
13 minLesson 10
Introduction to Data Analysis with the Titanic Dataset Using Pandas
15 minLesson 7
Pandas Series & DataFrames from Scratch | Data Analytics #7
36 minLesson 69