Lesson 56 · Pandas Full Course
Explore a Real Dataset with Pandas: Netflix Data Analysis Project | Pandas Course #56
In this lesson you'll be able to: Run a first-look checklist on any new dataset Repair text dates and text durations so you can calculate with them Ask…
- CoursePandas Full Course
- Lesson56 of 65
- FormatJupyter notebook · 45 code cells
- Data1 dataset
What you'll learn
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
- netflix_titles.csv preview2.9 MB
📓 Full notebook
Download .ipynbPandas 56: Exploring Real Datasets with Pandas#
In this lesson you'll be able to:
- Run a first-look checklist on any new dataset
- Repair text dates and text durations so you can calculate with them
- Ask questions with groupby and catch odd rows along the way
- Split multi-value cells with str.split and explode
- Turn a question into a chart and a one-line finding Dataset: Netflix titles (data/pandas/netflix_titles.csv)
Warm-up: profile groups with groupby#
Lesson 55 profiled clusters with groupby. Same habit, new data.
import pandas as pd
df = pd.read_csv("data/pandas/netflix_titles.csv")
print(df.shape)
df[["title", "type", "country", "release_year"]].head(3)
df.groupby("type")["release_year"].agg(["size", "median", "min"])
Run a first-look checklist#
Problem: know what you're holding before you trust any answer.
df.dtypes.value_counts()
Quick check: only one column is numeric. Which one?
df.select_dtypes("number").columns.tolist()
df.nunique().sort_values().head(6)
missing = df.isna().mean().mul(100).round(1)
missing.sort_values(ascending=False).head(5)
print(df.duplicated().sum())
print(df["show_id"].is_unique, df["title"].is_unique)
df.sample(3, random_state=42)[["title", "type", "duration"]]
Your turn#
Rating has 14 values. Which five are the rarest?
df["rating"].value_counts().tail(5)
Repair the date column#
Problem: date_added is text, so we can't ask 'when?' questions.
Common mistake: trusting text to be perfectly clean
pd.to_datetime(df["date_added"], format="%B %d, %Y")
clean = df["date_added"].str.strip()
df["added"] = pd.to_datetime(clean, format="%B %d, %Y")
print(df["added"].isna().sum())
print(df["added"].min(), df["added"].max())
no_date = df[df["added"].isna()]
print(no_date["type"].value_counts())
no_date[["title", "type"]].head(3)
df["year_added"] = df["added"].dt.year.astype("Int64")
df["year_added"].value_counts().sort_index().tail(7)
Quick check: 2021 has only 117 titles. Did Netflix stop adding?
latest = df["added"].max()
print(latest.date(), (df["added"] >= "2021-01-01").sum())
Turn durations into numbers#
Problem: duration mixes '90 min' with '2 Seasons'.
Common mistake: converting mixed text straight to numbers
df["duration"].astype(int)
parts = df["duration"].str.extract(r"(\d+) (\w+)")
df["length"] = parts[0].astype(int)
df["unit"] = parts[1].str.replace("Seasons", "Season")
pd.crosstab(df["type"], df["unit"])
Your turn#
Describe film lengths in minutes. What's typical?
movies = df[df["type"] == "Movie"]
movies["length"].describe().round(1)
cols = ["title", "length", "release_year"]
print(movies.nlargest(3, "length")[cols])
print(movies.nsmallest(3, "length")[cols])
import matplotlib.pyplot as plt
ax = movies["length"].plot.hist(bins=40, figsize=(7, 3.5))
ax.set_title("How long are Netflix films?")
ax.set_xlabel("Length (minutes)")
ax.set_ylabel("Number of films")
plt.show()
shows = df[df["type"] == "TV Show"]
shows["length"].value_counts().head(4)
Ask questions with groupby#
Problem: turn curiosity into a calculation, and spot oddities.
df["added"].dt.month_name().value_counts().head(3)
df["added"].dt.day_name().value_counts().head(3)
df["lag"] = df["year_added"] - df["release_year"]
df.groupby("type")["lag"].median()
film_lag = df.loc[df["type"] == "Movie", "lag"]
print(film_lag.describe().round(1))
print(round((film_lag >= 10).mean(), 3))
df.nsmallest(3, "release_year")[["title", "type", "release_year"]]
Quick check: can a title arrive on Netflix before its release year?
odd = df[df["lag"] < 0]
print(len(odd))
odd[["title", "type", "release_year", "year_added"]].head()
Common mistake: averaging a text column
df.groupby("type")["duration"].mean()
df.groupby(["type", "unit"])["length"].mean().round(1)
Your turn#
Which three calendar months had the most additions? Hint: to_period.
months = df["added"].dt.to_period("M")
months.value_counts().head(3)
Split cells that hold several values#
Problem: 'United States, India' is one cell but two countries.
print(df["country"].str.contains(",").sum())
print(df["country"].nunique())
Quick check: what does explode do to the raw text column?
print(len(df), len(df["country"].explode()))
countries = df["country"].str.split(", ").explode().str.strip()
print(len(countries))
countries.value_counts().head(8)
n_countries = (df["country"].str.count(",") + 1).astype("Int64")
print(n_countries.value_counts().head(4))
print(n_countries.max(), df.loc[n_countries.idxmax(), "title"])
genres = df["listed_in"].str.split(", ").explode()
print(genres.nunique())
genres.value_counts().head(5)
Your turn#
Find the five most common genres among TV shows only.
tv_genres = shows["listed_in"].str.split(", ").explode()
tv_genres.value_counts().head(5)
Reshape and chart a trend#
Problem: see how the mix of films and shows changed over time.
full = df[df["year_added"].between(2015, 2020)]
mix = pd.crosstab(full["year_added"], full["type"])
mix
share = pd.crosstab(full["year_added"], full["type"],
normalize="index").round(3)
share
ax = share["TV Show"].plot(marker="o", figsize=(7, 4))
ax.set_title("Share of Netflix additions that were TV shows")
ax.set_xlabel("Year added")
ax.set_ylabel("Share of titles added")
plt.show()
Your turn#
Which rating was most common among titles added in 2020?
added_2020 = df[df["year_added"] == 2020]
added_2020["rating"].value_counts().head(3)
Mini-project: is Netflix mostly American?#
Share of titles added each year with no United States production.
known = full[full["country"].notna()]
us = known["country"].str.contains("United States")
known = known.assign(intl=~us)
print(len(full), len(known), round(known["intl"].mean(), 3))
by_year = known.groupby("year_added")["intl"].mean().round(3)
by_year
by_type = known.groupby(["year_added", "type"])["intl"].mean()
by_type.unstack().round(3)
ax = by_type.unstack().plot(marker="o", figsize=(7, 4))
ax.axhline(0.5, color="grey", linestyle="--")
ax.set_title("Share of Netflix additions with no US production")
ax.set_xlabel("Year added")
ax.set_ylabel("Non-US share")
plt.show()
k20 = known[known["year_added"] == 2020]
c20 = k20["country"].str.split(", ").explode().str.strip()
c20[c20 != "United States"].value_counts().head(5)
peak = by_year.idxmax()
print(f"Non-US share: {by_year.min():.0%} to {by_year.max():.0%},"
f" peaking in {peak}")
Recap#
- First look: dtypes, nunique, missing shares, duplicates, sample
- Strip text before parsing dates with an exact format
- Use str.extract to pull numbers and units out of text
- Split and explode cells that hold several values
- Compare shares, not raw counts, when group sizes differ
Challenges#
- Which director has the most titles? Watch out for shared credits.
- Has the median film length changed by release decade?
- Repeat the mini-project for India: what share of additions involve it?
What's next#
Lesson 57: Hands-on - Exploratory Data Analysis (EDA) Project A full EDA, from raw file to written conclusions.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.



