Lesson 5 · Pandas Projects

Real Estate Data Analysis with Pandas: Clean, Explore, Visualise

A complete, standalone tutorial: build a messy synthetic listings dataset, clean it properly, then analyze pricing with pandas. No prior pandas experience…

⬇ Download notebook Open in Colab ↗

CoursePandas Projects
Lesson5 of 10
Video18 min
FormatJupyter notebook · 15 code cells

Matplotlib pandas NumPy

Data

No separate download needed — the notebook creates or downloads everything it uses.

📓 Full notebook

Download .ipynb

Pandas for Real Estate: Cleaning and Analyzing Housing Listings#

A complete, standalone tutorial: build a messy synthetic listings dataset, clean it properly, then analyze pricing with pandas.
No prior pandas experience needed. Let's jump straight in.

Before You Start#

Open a new Jupyter Notebook in VS Code and select your Python interpreter as the kernel.
If pandas isn't installed yet, open a terminal in VS Code and run: pip install pandas

Part 1: Building the Dataset#

import pandas as pd
import numpy as np
print(pd.__version__)

2.3.0

Generating Synthetic Listings#

rng = np.random.default_rng(seed=66)
n_listings = 250
neighborhoods = ['Riverside', 'Hillcrest', 'Oakwood', 'Lakeview', 'Downtown']

bedrooms = rng.integers(1, 6, size=n_listings)
bathrooms = np.clip(bedrooms - rng.integers(0, 2, size=n_listings), 1, None)
sqft = (bedrooms * rng.uniform(380, 520, size=n_listings) + rng.normal(0, 150, size=n_listings)).round(0)
year_built = rng.integers(1960, 2025, size=n_listings)
days_on_market = rng.integers(3, 180, size=n_listings)
print(sqft[:5])

[2129. 2040.  932. 1932. 1788.]

base_price = 220 * sqft + (2024 - year_built) * -80 + rng.normal(0, 15000, size=n_listings)
listing_price = np.clip(base_price, 80000, None).round(-2)

listings = pd.DataFrame({
    'listing_id': range(5001, 5001 + n_listings),
    'neighborhood': rng.choice(neighborhoods, size=n_listings),
    'bedrooms': bedrooms,
    'bathrooms': bathrooms,
    'sqft': sqft,
    'year_built': year_built,
    'listing_price': listing_price,
    'days_on_market': days_on_market
})
print(listings.shape)

(250, 8)

Injecting Missing Values and Outliers#

missing_idx = rng.choice(listings.index, size=15, replace=False)
listings.loc[missing_idx, 'sqft'] = np.nan

outlier_idx = rng.choice(listings.index, size=5, replace=False)
listings.loc[outlier_idx, 'listing_price'] = listings.loc[outlier_idx, 'listing_price'] * 10

listings.to_csv('housing_listings.csv', index=False)
listings = pd.read_csv('housing_listings.csv')
listings.head()

	listing_id	neighborhood	bedrooms	bathrooms	sqft	year_built	listing_price	days_on_market
0	5001	Downtown	5	5	2129.0	1972	451500.0	85
1	5002	Downtown	5	5	2040.0	2007	438500.0	132
2	5003	Downtown	2	2	932.0	1972	190700.0	18
3	5004	Riverside	4	4	1932.0	2016	424600.0	147
4	5005	Oakwood	4	3	1788.0	1973	371300.0	153

Part 2: First Look at the Data#

print(listings.shape)
listings.isna().sum()

(250, 8)

listing_id         0
neighborhood       0
bedrooms           0
bathrooms          0
sqft              15
year_built         0
listing_price      0
days_on_market     0
dtype: int64

listings['listing_price'].describe()

count    2.500000e+02
mean     3.479028e+05
std      3.759155e+05
min      8.000000e+04
25%      1.828500e+05
50%      3.237500e+05
75%      4.278000e+05
max      4.141000e+06
Name: listing_price, dtype: float64

Part 3: Cleaning the Data#

neighborhood_median_sqft = listings.groupby('neighborhood')['sqft'].transform('median')
listings['sqft'] = listings['sqft'].fillna(neighborhood_median_sqft)
listings['sqft'].isna().sum()

np.int64(0)

Detecting and Capping Price Outliers#

q1 = listings['listing_price'].quantile(0.25)
q3 = listings['listing_price'].quantile(0.75)
iqr = q3 - q1
upper_bound = q3 + 1.5 * iqr
outliers = listings[listings['listing_price'] > upper_bound]
print(len(outliers))
outliers[['listing_id', 'neighborhood', 'sqft', 'listing_price']]

	listing_id	neighborhood	sqft	listing_price
18	5019	Hillcrest	1218.0	2511000.0
78	5079	Hillcrest	408.0	1058000.0
127	5128	Riverside	1158.0	2402000.0
162	5163	Riverside	1918.0	4141000.0
242	5243	Lakeview	1376.0	2776000.0

listings['listing_price'] = listings['listing_price'].clip(upper=upper_bound)
listings.loc[outlier_idx, 'listing_price']

18     795225.0
242    795225.0
162    795225.0
127    795225.0
78     795225.0
Name: listing_price, dtype: float64

Part 4: Price Per Square Foot#

listings['price_per_sqft'] = (listings['listing_price'] / listings['sqft']).round(2)
listings[['neighborhood', 'sqft', 'listing_price', 'price_per_sqft']].head()

	neighborhood	sqft	listing_price	price_per_sqft
0	Downtown	2129.0	451500.0	212.07
1	Downtown	2040.0	438500.0	214.95
2	Downtown	932.0	190700.0	204.61
3	Riverside	1932.0	424600.0	219.77
4	Oakwood	1788.0	371300.0	207.66

Part 5: Correlation Analysis#

numeric_cols = ['bedrooms', 'bathrooms', 'sqft', 'year_built', 'listing_price', 'days_on_market']
correlations = listings[numeric_cols].corr()
correlations['listing_price'].sort_values(ascending=False)

listing_price     1.000000
bedrooms          0.840525
sqft              0.836053
bathrooms         0.802220
days_on_market   -0.026956
year_built       -0.055238
Name: listing_price, dtype: float64

Part 6: Price Tiers and Neighborhood Summary#

listings['price_tier'] = pd.qcut(listings['listing_price'], q=4, labels=['Budget', 'Mid-Range', 'Upper-Mid', 'Premium'])
listings['price_tier'].value_counts().sort_index()

price_tier
Budget       63
Mid-Range    62
Upper-Mid    62
Premium      63
Name: count, dtype: int64

neighborhood_summary = listings.groupby('neighborhood').agg(
    listings_count=('listing_id', 'count'),
    avg_price=('listing_price', 'mean'),
    avg_price_per_sqft=('price_per_sqft', 'mean'),
    avg_days_on_market=('days_on_market', 'mean')
).round(1).sort_values('avg_price', ascending=False)
neighborhood_summary

	listings_count	avg_price	avg_price_per_sqft	avg_days_on_market
neighborhood
Hillcrest	48	339486.5	262.7	77.0
Riverside	47	327028.7	231.8	82.8
Downtown	51	308935.3	209.7	100.2
Lakeview	48	300275.5	227.7	84.0
Oakwood	56	289807.1	216.8	82.2

Part 7: Visualizing the Results#

import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=(8, 5))
ax.scatter(listings['sqft'], listings['listing_price'], alpha=0.5, color='teal')
ax.set_title('Listing Price vs. Square Footage')
ax.set_xlabel('Square Feet')
ax.set_ylabel('Listing Price ($)')
plt.tight_layout()
plt.savefig('price_vs_sqft.png', dpi=150)
plt.close(fig)
print('Saved price_vs_sqft.png')

Saved price_vs_sqft.png

fig, ax = plt.subplots(figsize=(8, 5))
neighborhood_summary['avg_price'].plot(kind='bar', ax=ax, color='steelblue')
ax.set_title('Average Listing Price by Neighborhood')
ax.set_ylabel('Average Price ($)')
plt.tight_layout()
plt.savefig('avg_price_by_neighborhood.png', dpi=150)
plt.close(fig)
print('Saved avg_price_by_neighborhood.png')

Saved avg_price_by_neighborhood.png

Wrap-Up: What You Learned#

Generating a realistic, deliberately messy synthetic listings dataset, then saving and reloading with to_csv and read_csv.
Detecting missing values with isna, and filling them intelligently with groupby plus transform.
Detecting price outliers with the IQR method, and capping them with clip instead of deleting rows.
Deriving a normalized metric, price per square foot, and running a full correlation analysis with corr.
Bucketing with qcut for balanced quartile tiers, and a multi-statistic neighborhood summary.
A scatter plot and a bar chart, both presentation-ready, with matplotlib.
You went from a messy synthetic listings feed to a fully cleaned, analyzed housing report. If you want the next dataset in this series to land in your feed automatically, subscribing is the move see you in the next one.

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.

☕ Buy me a coffee Other ways to help