Lesson 57 · Pandas Full Course
Exploratory Data Analysis (EDA) in Pandas: Full Hands-on Project | Pandas Course #57
In this lesson you'll be able to: Audit a new dataset for gaps, caps and odd values Describe one column at a time with numbers and a histogram Compare…
- CoursePandas Full Course
- Lesson57 of 65
- FormatJupyter notebook · 40 code cells
- Data1 dataset
What you'll learn
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
- housing.csv preview1.4 MB
📓 Full notebook
Download .ipynbPandas 57: Hands-on - Exploratory Data Analysis (EDA) Project#
In this lesson you'll be able to:
- Audit a new dataset for gaps, caps and odd values
- Describe one column at a time with numbers and a histogram
- Compare groups and measure relationships between columns
- Engineer fairer features such as rooms per household
- Turn the exploration into one clear, checked finding Data: California housing, 20,640 districts from the 1990 census
Warm-up: the first-look routine#
Lesson 56 explored real datasets: shape, head, types, gaps.
import pandas as pd
housing = pd.read_csv("data/pandas/housing.csv")
print(housing.shape)
housing.head(3)
housing.info()
Our three questions
- What does a typical district cost? 2. Where is it dearest?
- Which numbers move with price, and why?
Step 1: Audit the data before trusting it#
Find gaps, limits and odd values before any analysis.
missing = housing.isna().sum()
print(missing[missing > 0])
share = housing["total_bedrooms"].isna().mean()
print(f"{share:.1%} of districts lack a bedroom count")
Quick check: are the gaps bunched in one kind of location?
gaps = housing["total_bedrooms"].isna()
gaps.groupby(housing["ocean_proximity"]).mean().mul(100).round(1)
print(housing.duplicated().sum())
print(sorted(housing["ocean_proximity"].unique()))
cols = ["housing_median_age", "median_income", "median_house_value"]
housing[cols].describe().round(1)
Quick check: why might the maximum price be exactly 500,001?
top = housing["median_house_value"].max()
at_top = (housing["median_house_value"] == top).sum()
print(top, at_top)
housing["housing_median_age"].value_counts().head(3)
Both columns are capped: real values above the limit were cut.
Common mistake: converting a column that still has gaps
housing["total_bedrooms"].astype(int)
fill = housing["total_bedrooms"].median()
housing["total_bedrooms"] = housing["total_bedrooms"].fillna(fill)
housing["total_bedrooms"] = housing["total_bedrooms"].astype(int)
print(fill, housing["total_bedrooms"].isna().sum())
Step 2: Describe one column at a time#
What does a typical district look like?
price = housing["median_house_value"]
print(f"Median {price.median():,.0f}, mean {price.mean():,.0f}")
print(price.quantile([0.1, 0.9]).round(0))
print(round(price.skew(), 2))
import matplotlib.pyplot as plt
ax = price.plot(kind="hist", bins=50, figsize=(8, 4))
ax.set_title("Median house value per district")
ax.set_xlabel("Median house value (USD)")
ax.set_ylabel("Number of districts")
plt.show()
places = housing["ocean_proximity"].value_counts()
print(places)
print(places.div(len(housing)).round(3))
Your turn: find the 25th, 50th and 75th percentile of median_income. Note: income is in tens of thousands of dollars.
income = housing["median_income"]
income.quantile([0.25, 0.5, 0.75]).round(2)
print(income.min(), income.max())
print((income == income.max()).sum())
age = housing["housing_median_age"]
ax = age.plot(kind="hist", bins=52, figsize=(8, 4))
ax.set_title("Median housing age per district")
ax.set_xlabel("Median age (years)")
ax.set_ylabel("Number of districts")
plt.show()
Step 3: Compare groups#
Where is housing dearest? Group by location and compare.
by_place = housing.groupby("ocean_proximity")["median_house_value"]
summary = by_place.agg(["count", "median", "mean"]).round(0)
summary.sort_values("median", ascending=False)
Quick check: should we trust the ISLAND median?
island = housing[housing["ocean_proximity"] == "ISLAND"]
island[["median_income", "median_house_value", "population"]]
extra = ["median_income", "housing_median_age"]
housing.groupby("ocean_proximity")[extra].median()
Your turn: which location has the highest share of capped prices?
housing["capped"] = housing["median_house_value"] == top
housing.groupby("ocean_proximity")["capped"].mean().round(3)
import seaborn as sns
order = ["INLAND", "<1H OCEAN", "NEAR OCEAN", "NEAR BAY"]
main = housing[housing["ocean_proximity"].isin(order)]
fig, ax = plt.subplots(figsize=(8, 4))
sns.boxplot(data=main, x="ocean_proximity",
y="median_house_value", order=order, ax=ax)
ax.set_title("House value by location")
ax.set_xlabel("Location")
ax.set_ylabel("Median house value (USD)")
plt.show()
Step 4: Find relationships between numbers#
Which columns move together with price?
Common mistake: correlating a table that contains text
housing.corr()
corr = housing.corr(numeric_only=True)
corr["median_house_value"].sort_values(ascending=False).round(2)
Quick check: does the price cap weaken the income link?
uncapped = housing[~housing["capped"]]
r = uncapped["median_income"].corr(uncapped["median_house_value"])
print(len(uncapped), round(r, 2))
ax = housing.plot(kind="scatter", x="median_income",
y="median_house_value", alpha=0.1,
figsize=(8, 4))
ax.set_title("Income against house value")
ax.set_xlabel("Median income (tens of thousands USD)")
ax.set_ylabel("Median house value (USD)")
plt.show()
bins = [0, 2, 3, 4, 5, 6, 16]
labels = ["<2", "2-3", "3-4", "4-5", "5-6", "6+"]
housing["income_band"] = pd.cut(income, bins=bins, labels=labels)
band = housing.groupby("income_band", observed=True)
band["median_house_value"].median()
housing["income_band"].value_counts().sort_index()
Step 5: Engineer fairer features#
Raw totals grow with district size. Ratios compare like with like.
housing["rooms_per_home"] = (housing["total_rooms"]
/ housing["households"])
housing["people_per_home"] = (housing["population"]
/ housing["households"])
new = ["rooms_per_home", "people_per_home"]
housing[new].describe().round(1)
show = ["rooms_per_home", "households", "population"]
housing.nlargest(3, "rooms_per_home")[show].round(1)
crowded = housing["people_per_home"] > 10
print(crowded.sum(), f"{crowded.mean():.2%}")
Quick check: which links more strongly to price: total_rooms or rooms_per_home?
check = ["total_rooms", "rooms_per_home", "median_house_value"]
print(housing[check].corr().round(3))
Common mistake: a typo in a new column name
housing["rooms_per_house"].median()
print([c for c in housing.columns if "per" in c])
print(round(housing["rooms_per_home"].median(), 2))
Your turn: add bedrooms_per_room and find its correlation with price.
housing["bedrooms_per_room"] = (housing["total_bedrooms"]
/ housing["total_rooms"])
bpr = housing["bedrooms_per_room"]
print(round(bpr.corr(housing["median_house_value"]), 3))
Mini-project: is the coast premium just about income?#
Compare inland and coastal prices for districts with the same income.
housing["inland"] = housing["ocean_proximity"] == "INLAND"
housing.groupby("inland")["median_income"].median().round(2)
table = housing.pivot_table(index="income_band", columns="inland",
values="median_house_value",
aggfunc="median", observed=True)
table.columns = ["coastal", "inland"]
table["inland_vs_coast"] = (table["inland"] / table["coastal"])
table.round(2)
ax = table[["coastal", "inland"]].plot(kind="bar", rot=0,
figsize=(8, 4))
ax.set_title("Same income, different price")
ax.set_xlabel("Median income band (tens of thousands USD)")
ax.set_ylabel("Median house value (USD)")
plt.show()
ax = housing.plot(kind="scatter", x="longitude", y="latitude",
c="median_house_value", cmap="viridis",
s=4, alpha=0.4, figsize=(7, 6))
ax.set_title("California districts by house value")
ax.set_xlabel("Longitude")
ax.set_ylabel("Latitude")
plt.show()
fair = housing[~housing["capped"]]
check = fair.pivot_table(index="income_band", columns="inland",
values="median_house_value",
aggfunc="median", observed=True)
(check[True] / check[False]).round(2)
ratio = table["inland_vs_coast"]
low, high = ratio.min(), ratio.max()
print(f"Same income: inland homes cost {low:.0%} to {high:.0%}")
print("of coastal homes, so income is not the whole story.")
import os
os.makedirs("output/pandas", exist_ok=True)
path = "output/pandas/l57_coast_premium.csv"
table.to_csv(path)
pd.read_csv(path, index_col="income_band").round(2).head(3)
Recap#
- Start with questions, then audit: gaps, caps and odd values
- Describe one column at a time: median, percentiles, histogram
- Compare groups with groupby, and check each group's count
- corr(numeric_only=True) and scatter plots show relationships
- Ratios beat raw totals; hold one factor fixed to test a story
Challenges#
- Hold housing age fixed instead of income. Does the inland gap remain?
- Which location type has the oldest housing, by median age?
- Plot a histogram of people_per_home below 6. What's typical?
What's next#
Lesson 58: Cleaning and Analysing Survey Data Real survey answers: free-text job titles, currencies and messy categories.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.



