Lesson 2 · Data Mining Full Course
Data Mining Full Course Part 2 (2 Hours): Data Cleaning, Preprocessing & PCA in Python
All 5 lesson notebooks from Part 2 of the Data Mining Full Course in Python, in teaching order. Each lesson starts with a heading; the timestamps below jump…
- CourseData Mining Full Course
- Lesson2 of 6
- Video1 h 58 min
- FormatJupyter notebook · 186 code cells
- Data9 datasets
What you'll learn
- Data Mining 09: Data Cleaning- Handling Missing Values, Noise, and Outliers
- Warm-up: explore before you clean
- Find missing values, including disguised ones
- Drop or fill? Strategies for missing values
- Detect outliers with the IQR rule and z-scores
- Smooth noise by binning
- Mini-project: does cleaning change the story?
- Challenges
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
- pima_diabetes.csv preview22.7 KB
- housing.csv preview1.4 MB
- diamonds.csv preview2.6 MB
- superstore.csv preview1.7 MB
- superstore_api_page1.json preview76.1 KB
- wine.csv preview13.0 KB
- breast_cancer.csv preview121.9 KB
- iris.csv preview3.8 KB
- titanic.csv preview58.9 KB
📓 Full notebook
Download .ipynbData Mining Full Course, Part 2: Data Cleaning, Preprocessing and PCA#
All 5 lesson notebooks from Part 2 of the Data Mining Full Course in Python, in teaching order. Each lesson starts with a heading; the timestamps below jump to it in the video.
- 0:00 Lesson 9: Data Cleaning- Handling Missing Values, Noise, and Outliers
- 0:23:53 Lesson 10: Data Transformation- Normalization, Encoding, Feature Scaling
- 0:47:14 Lesson 11: Data Integration and Feature Engineering
- 1:10:04 Lesson 12: Dimensionality Reduction with PCA
- 1:33:56 Lesson 13: Hands-on Preprocessing with Titanic and Iris Datasets
Data files: save the datasets from the lesson page in the same folder as this notebook. Files the lessons create are written to that folder too.
Data Mining 09: Data Cleaning- Handling Missing Values, Noise, and Outliers#
In this lesson you'll be able to:
- Uncover missing values disguised as zeros, and measure them
- Choose between dropping and filling, and see what each costs
- Detect outliers with the IQR rule and z-scores, then cap them
- Smooth noisy values by binning
- Wrap every step into one reusable cleaning function Dataset: Pima Indians diabetes, 768 women (data/mining/pima_diabetes.csv)
Warm-up: explore before you clean#
Lesson 08 explored a dataset step by step. Start the same way here.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
df = pd.read_csv("pima_diabetes.csv")
print(df.shape)
df.head()
df.describe().round(1).loc[["mean", "min", "max"]]
Find missing values, including disguised ones#
Problem: blanks are easy to count, but zeros can hide missing data too.
print(df.isna().sum().sum())
Quick check: can a living patient have a BMI or blood pressure of 0?
cols = ["glucose", "blood_pressure", "skin_thickness", "insulin",
"bmi"]
(df[cols] == 0).sum()
Zeros are real in pregnancies and outcome, so leave those columns alone.
df[cols] = df[cols].replace(0, np.nan)
missing = df.isna().sum()
missing[missing > 0]
share = df.isna().mean().sort_values()
ax = share[share > 0].plot(kind="barh", figsize=(7, 3.5))
ax.set_title("Share of values missing per column")
ax.set_xlabel("Share missing")
ax.set_ylabel("Column")
print(round(df.isna().any(axis=1).mean(), 2))
import seaborn as sns
plt.figure(figsize=(7, 4))
sns.heatmap(df[cols].isna().head(80), cbar=False, cmap="Greys")
plt.title("Where the gaps are: first 80 patients")
plt.xlabel("Measurement")
plt.ylabel("Patient row")
plt.show()
gap = df["insulin"].isna()
gap.groupby(df["outcome"]).mean().round(3)
Your turn#
How many patients are missing BOTH skin_thickness and insulin?
both = df["skin_thickness"].isna() & df["insulin"].isna()
print(both.sum())
Drop or fill? Strategies for missing values#
Problem: every strategy has a cost. Measure it before you choose.
dropped = df.dropna()
print(len(df), len(dropped))
print(round(df["outcome"].mean(), 3),
round(dropped["outcome"].mean(), 3))
thin = df.drop(columns=["insulin", "skin_thickness"]).dropna()
print(thin.shape)
filled = df.copy()
medians = filled[cols].median()
filled[cols] = filled[cols].fillna(medians)
print(medians)
print(filled.isna().sum().sum())
before = df["insulin"].std()
after = filled["insulin"].std()
print(round(before, 1), round(after, 1))
print((filled["insulin"] == medians["insulin"]).sum())
fig, axes = plt.subplots(1, 2, figsize=(10, 3.5), sharey=True)
df["insulin"].plot(kind="hist", bins=40, ax=axes[0])
filled["insulin"].plot(kind="hist", bins=40, ax=axes[1])
axes[0].set_title("Insulin, gaps left out")
axes[1].set_title("Insulin, gaps filled with the median")
for ax in axes:
ax.set_xlabel("Insulin (mu U/ml)")
axes[0].set_ylabel("Patients")
age_band = pd.cut(df["age"], bins=[20, 30, 40, 50, 81],
labels=["21-30", "31-40", "41-50", "51+"])
by_age = df.groupby(age_band, observed=True)["blood_pressure"]
by_age.agg(["size", "count", "median"])
bp = df["blood_pressure"].fillna(by_age.transform("median"))
print(bp.isna().sum())
print(bp[df["blood_pressure"].isna()].value_counts())
Common mistake: converting a column with gaps to whole numbers
df["glucose"].astype(int)
glucose = df["glucose"].astype("Int64")
print(glucose.dtype, glucose.isna().sum())
glucose[glucose.isna()]
from sklearn.impute import KNNImputer
knn = KNNImputer(n_neighbors=5)
knn_vals = knn.fit_transform(df[cols])
knn_df = pd.DataFrame(knn_vals, columns=cols)
print(round(knn_df["insulin"].std(), 1))
Your turn#
Compare the BMI median of filled and knn_df. Which moved more?
print(df["bmi"].median(), filled["bmi"].median(),
knn_df["bmi"].median())
Detect outliers with the IQR rule and z-scores#
Problem: a few extreme values can drag means and models off course.
ax = sns.boxplot(data=df[["insulin"]], orient="h")
ax.set_title("Measured insulin: box and outliers")
ax.set_xlabel("Insulin (mu U/ml)")
IQR rule: an outlier lies more than 1.5 IQR below Q1 or above Q3.
ins = df["insulin"]
q1, q3 = ins.quantile([0.25, 0.75])
iqr = q3 - q1
low, high = q1 - 1.5 * iqr, q3 + 1.5 * iqr
print(q1, q3, iqr)
print(low, high)
Common mistake: using or between two conditions
outliers = df[ins < low or ins > high]
outliers = df[(ins < low) | (ins > high)]
print(len(outliers))
outliers["insulin"].describe().round(1)
z = (ins - ins.mean()) / ins.std()
print((z.abs() > 3).sum())
print(df.loc[z.abs() > 3, "insulin"].min())
Quick check: which rule flags more outliers here, and why?
print(round(ins.skew(), 2))
print(round(ins.mean(), 1), round(ins.median(), 1))
Your turn#
Use the IQR rule to count BMI outliers. Print the fences too.
b1, b3 = df["bmi"].quantile([0.25, 0.75])
b_low, b_high = b1 - 1.5 * (b3 - b1), b3 + 1.5 * (b3 - b1)
print(round(b_low, 2), round(b_high, 2))
print(((df["bmi"] < b_low) | (df["bmi"] > b_high)).sum())
df["insulin_cap"] = ins.clip(upper=high)
df[["insulin", "insulin_cap"]].describe().round(1)
f_q1, f_q3 = filled["insulin"].quantile([0.25, 0.75])
print(f_q3 - f_q1, q3 - q1)
Smooth noise by binning#
Problem: tiny random wobbles in measurements hide the real pattern.
Smoothing by bin means:
- Sort values into equal-frequency bins
- Replace each value with its bin's mean
Common mistake: a label count that doesn't match the bins
pd.qcut(filled["bmi"], q=4, labels=["low", "mid", "high"])
names = ["low", "mid", "high", "very high"]
filled["bmi_bin"] = pd.qcut(filled["bmi"], q=4, labels=names)
filled["bmi_bin"].value_counts(sort=False)
by_bin = filled.groupby("bmi_bin", observed=True)["bmi"]
filled["bmi_smooth"] = by_bin.transform("mean").round(1)
filled[["bmi", "bmi_bin", "bmi_smooth"]].head()
rate = filled.groupby("bmi_bin", observed=True)["outcome"].mean()
rate.round(3)
Your turn#
Make 4 equal-WIDTH bins of bmi with pd.cut. How full is each bin?
wide = pd.cut(filled["bmi"], bins=4)
wide.value_counts(sort=False)
Mini-project: does cleaning change the story?#
Build one cleaning function, save the clean file, and compare raw vs clean.
def clean(raw):
out = raw.copy()
out[cols] = out[cols].replace(0, np.nan)
q1, q3 = out["insulin"].quantile([0.25, 0.75])
cap = q3 + 1.5 * (q3 - q1)
out[cols] = out[cols].fillna(out[cols].median())
out["insulin"] = out["insulin"].clip(upper=cap)
return out
raw = pd.read_csv("pima_diabetes.csv")
tidy = clean(raw)
print(tidy.isna().sum().sum(), (tidy[cols] == 0).sum().sum())
import os
tidy.to_csv("m09_pima_clean.csv", index=False)
back = pd.read_csv("m09_pima_clean.csv")
print(back.shape)
show = ["glucose", "blood_pressure", "insulin", "bmi"]
raw_means = raw.groupby("outcome")[show].mean()
clean_means = back.groupby("outcome")[show].mean()
pd.concat({"raw": raw_means, "clean": clean_means}).round(1)
raw_gap = raw_means.loc[1] - raw_means.loc[0]
clean_gap = clean_means.loc[1] - clean_means.loc[0]
pd.DataFrame({"raw_gap": raw_gap, "clean_gap": clean_gap}).round(1)
Recap#
- isna() only sees blanks; check for impossible zeros too
- Dropping rows is simple but can throw away half the data
- Median filling keeps rows but shrinks spread; KNN fills smarter
- IQR fences and z-scores flag outliers; clip caps them
- Binning by bin means smooths noise; wrap it all in one function
Challenges#
- Count blood_pressure outliers with the IQR rule, then cap them.
- Compare KNN and median filling for skin_thickness spread.
- Smooth glucose by bin means with 5 equal-frequency bins.
What's next#
Lesson 10: Data Transformation- Normalization, Encoding, Feature Scaling Put clean columns on the same scale and turn text into numbers.
Data Mining 10: Data Transformation: Normalization, Encoding, Feature Scaling#
In this lesson you'll be able to:
- Explain why features on different scales mislead a model
- Rescale columns with min-max, standard and robust scalers
- Tame skewed columns with a log transform
- Encode categories with one-hot and ordinal encoders
- Fit transformations on training data only, inside a pipeline Datasets: California housing districts (data/pandas/housing.csv) and diamonds (data/pandas/diamonds.csv)
Warm-up: clean before you transform#
Lesson 09 handled missing values. One column here still has gaps.
import pandas as pd
import numpy as np
df = pd.read_csv("housing.csv")
print(df.shape)
df.isna().sum()
fill = df["total_bedrooms"].median()
df["total_bedrooms"] = df["total_bedrooms"].fillna(fill)
print(fill, df["total_bedrooms"].isna().sum())
Spot the scale problem#
Problem: some columns are in the thousands, others are tiny.
cols = ["housing_median_age", "total_rooms", "population",
"median_income", "median_house_value"]
df[cols].describe().loc[["min", "max", "mean"]].round(1)
Quick check: two districts differ by 1 in income and 100 in rooms. Which gap matters more to a distance?
feats = ["total_rooms", "median_income"]
a, b = df.loc[0, feats], df.loc[3, feats]
gaps = (a - b) ** 2
print(gaps.round(4))
print("share from rooms:", round(gaps.iloc[0] / gaps.sum(), 4))
Normalise to the 0-1 range#
Min-max scaling: (x - min) / (max - min). Every value lands between 0 and 1.
rooms = df["total_rooms"]
rooms_mm = (rooms - rooms.min()) / (rooms.max() - rooms.min())
print(rooms.head(3).to_list())
print(rooms_mm.head(3).round(4).to_list())
print(rooms_mm.min(), rooms_mm.max())
Common mistake: passing a single column (a Series) to a scikit-learn scaler.
from sklearn.preprocessing import MinMaxScaler
mm = MinMaxScaler()
mm.fit_transform(df["total_rooms"])
scaled = mm.fit_transform(df[["total_rooms"]])
print(scaled[:3].ravel().round(4))
print(mm.data_min_, mm.data_max_)
mm_all = MinMaxScaler().fit_transform(df[feats])
a2, b2 = mm_all[0], mm_all[3]
gaps2 = (a2 - b2) ** 2
print("share from rooms:", round(gaps2[0] / gaps2.sum(), 4))
Standardise, resist outliers, tame skew#
Standard scaling: (x - mean) / std. Robust scaling uses the median and IQR.
pop = df["population"]
print("max:", pop.max(), "median:", pop.median())
pop_mm = (pop - pop.min()) / (pop.max() - pop.min())
print("median after min-max:", round(pop_mm.median(), 4))
from sklearn.preprocessing import StandardScaler, RobustScaler
std = StandardScaler().fit_transform(df[["population"]])
rob = RobustScaler().fit_transform(df[["population"]])
print("standard mean, std:", std.mean().round(3), std.std().round(3))
print("standard median:", np.median(std).round(3))
print("robust median:", np.median(rob).round(3))
comp = pd.DataFrame({"min-max": pop_mm.to_numpy(),
"standard": std.ravel(),
"robust": rob.ravel()})
comp.describe().loc[["min", "50%", "max"]].round(2)
Your turn: standardise median_income by hand with the mean and std, then check the first value against StandardScaler.
inc = df["median_income"]
z = (inc - inc.mean()) / inc.std(ddof=0)
z_sk = StandardScaler().fit_transform(df[["median_income"]])
print(round(z.iloc[0], 4), z_sk[0, 0].round(4))
A long right tail hides most districts in one bar. log1p pulls the tail in.
print("skew before:", round(pop.skew(), 2))
log_pop = np.log1p(pop)
print("skew after:", round(log_pop.skew(), 2))
import matplotlib.pyplot as plt
fig, axes = plt.subplots(1, 2, figsize=(10, 4))
axes[0].hist(pop, bins=50)
axes[0].set_title("Population (raw)")
axes[0].set_xlabel("People in district")
axes[1].hist(log_pop, bins=50)
axes[1].set_title("Population (log1p)")
axes[1].set_xlabel("log(1 + people)")
axes[0].set_ylabel("Number of districts")
plt.show()
Encode categories as numbers#
Models need numbers. One-hot gives each category its own 0/1 column.
df["ocean_proximity"].value_counts()
dummies = pd.get_dummies(df["ocean_proximity"], dtype=int)
print(dummies.shape)
dummies.head(3)
Your turn: add the dummies to df with the prefix ocean, and count the columns.
ocean = pd.get_dummies(df["ocean_proximity"], prefix="ocean",
dtype=int)
wide = df.join(ocean)
print(df.shape[1], wide.shape[1])
print(wide.columns[-5:].to_list())
Common mistake: fitting an encoder on data that is missing a rare category, then meeting that category later.
from sklearn.preprocessing import OneHotEncoder
island = df["ocean_proximity"] == "ISLAND"
seen = df.loc[~island, ["ocean_proximity"]]
ohe = OneHotEncoder().fit(seen)
ohe.transform(df.loc[island, ["ocean_proximity"]])
ohe = OneHotEncoder(handle_unknown="ignore", sparse_output=False)
ohe.fit(seen)
new = ohe.transform(df.loc[island, ["ocean_proximity"]])
print(ohe.get_feature_names_out())
print(new[:2])
Keep the order with ordinal encoding#
Some categories have a natural order. Encode that order explicitly.
gems = pd.read_csv("diamonds.csv")
gems[["carat", "cut", "price"]].head(3)
Quick check: if we sort the cut names alphabetically and number them, where does Ideal land?
alpha = sorted(gems["cut"].unique())
print(dict(zip(alpha, range(5))))
from sklearn.preprocessing import OrdinalEncoder
order = ["Fair", "Good", "Very Good", "Premium", "Ideal"]
oe_cut = OrdinalEncoder(categories=[order])
gems["cut_code"] = oe_cut.fit_transform(gems[["cut"]]).ravel()
gems.groupby("cut_code")["cut"].first()
by_cut = gems.groupby("cut_code")[["carat", "price"]].mean()
by_cut.round(2)
Fit on training data only#
Scalers learn from data. Let them learn from the training rows alone.
Quick check: if the scaler is fitted on training rows, will the scaled test column have a mean of exactly 0?
from sklearn.model_selection import train_test_split
train, test = train_test_split(df, test_size=0.2, random_state=42)
sc = StandardScaler().fit(train[["median_income"]])
test_z = sc.transform(test[["median_income"]])
print(sc.mean_.round(3), test_z.mean().round(3))
from sklearn.compose import ColumnTransformer
num_cols = ["housing_median_age", "total_rooms", "total_bedrooms",
"population", "households", "median_income"]
prep = ColumnTransformer([
("num", StandardScaler(), num_cols),
("cat", OneHotEncoder(handle_unknown="ignore"),
["ocean_proximity"]),
])
X_train = prep.fit_transform(train)
print(X_train.shape)
print(prep.get_feature_names_out()[[0, 6, 10]])
Your turn: transform the test rows with prep and check the shape.
X_test = prep.transform(test)
print(X_test.shape)
print(X_test[:, :6].mean(axis=0).round(2))
Mini-project: does scaling change a model?#
Predict house value with k-nearest neighbours, with and without scaling.
from sklearn.neighbors import KNeighborsRegressor
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_absolute_error
feats = num_cols + ["longitude", "latitude"]
target = "median_house_value"
raw = KNeighborsRegressor().fit(train[feats], train[target])
pred_raw = raw.predict(test[feats])
mae_raw = mean_absolute_error(test[target], pred_raw)
print(f"raw MAE: ${mae_raw:,.0f}")
knn = make_pipeline(StandardScaler(), KNeighborsRegressor())
knn.fit(train[feats], train[target])
pred_sc = knn.predict(test[feats])
mae_sc = mean_absolute_error(test[target], pred_sc)
print(f"scaled MAE: ${mae_sc:,.0f}")
one = test.iloc[[0]]
print("actual:", one[target].iloc[0])
print("raw kNN:", raw.predict(one[feats])[0])
print("scaled kNN:", knn.predict(one[feats])[0])
from sklearn.metrics import r2_score
r2_raw = r2_score(test[target], pred_raw)
r2_sc = r2_score(test[target], pred_sc)
print(round(r2_raw, 3), round(r2_sc, 3))
maes = pd.Series({"Raw": mae_raw, "Scaled": mae_sc})
ax = maes.plot(kind="bar", rot=0, figsize=(6, 4))
ax.set_title("kNN error on test districts")
ax.set_xlabel("Features")
ax.set_ylabel("Mean absolute error ($)")
print(f"error cut by {1 - mae_sc / mae_raw:.0%}")
Recap#
- Distance-based models are dominated by large-scale columns
- Min-max maps to 0-1; standard gives mean 0, std 1; robust resists outliers
- log1p pulls in long right tails
- One-hot for unordered categories, OrdinalEncoder with an explicit order
- Fit on training rows only: use ColumnTransformer and pipelines
Challenges#
- Swap StandardScaler for MinMaxScaler in the kNN pipeline. Which wins?
- Add log1p of population as a feature and re-measure the error.
- Ordinal-encode diamond clarity (I1 worst, IF best).
What's next#
Lesson 11: Data Integration and Feature Engineering Combine data from several sources and build new, more useful features.
Data Mining 11: Data Integration and Feature Engineering#
In this lesson you'll be able to:
- Stack paged API responses into one dataset
- Flatten nested JSON records into a table
- Match schemas and merge in a lookup table safely
- Engineer date, ratio, band and group features
- Package features in a function and reuse it on new data Data: Superstore orders as a saved API response (data/pandas/superstore_api_page1-3.json) and as a CSV export (data/pandas/superstore.csv)
Warm-up: encode a category#
Lesson 10 turned categories into numbers. Here it is on the store data.
import pandas as pd
shop = pd.read_csv("superstore.csv")
print(shop.shape)
pd.get_dummies(shop["Ship Mode"], dtype=int).sum()
money = shop[["Sales", "Shipping Cost", "Profit"]]
money.describe().loc[["min", "50%", "max"]].round(2)
Stack the pages of an API response#
Problem: an API sends 100 orders per page. We need all of them in one place.
import json
with open("superstore_api_page1.json") as f:
page = json.load(f)
print(list(page.keys()))
print(page["total_pages"], page["total_orders"], page["next"])
Quick check: three pages of up to 100 orders. How many orders should we end up with?
orders = []
for i in range(1, 4):
name = f"superstore_api_page{i}.json"
with open(name) as f:
orders += json.load(f)["results"]
print(len(orders))
ids = [o["order_id"] for o in orders]
print(len(ids), len(set(ids)))
first = orders[0]
print(list(first.keys()))
print(first["customer"])
print(len(first["items"]), "item(s)")
Flatten nested records into a table#
json_normalize turns nested dictionaries into columns, and lists into rows.
flat = pd.json_normalize(orders)
print(flat.shape)
print(flat.columns.to_list())
Common mistake: passing a nested field to meta as a dotted string.
items = pd.json_normalize(orders, record_path="items",
meta=["order_id", "customer.name"])
meta = ["order_id", "order_date", ["customer", "name"],
["customer", "segment"], ["shipping", "mode"],
["shipping", "ship_date"]]
items = pd.json_normalize(orders, record_path="items", meta=meta)
print(items.shape)
items.iloc[0]
Quick check: 300 orders became 477 item rows. Why?
per_order = items["order_id"].value_counts()
per_order.value_counts().sort_index()
Match schemas and merge a lookup table#
The API lacks each product's container and base margin. The CSV export has them.
schema = {"customer.name": "customer",
"customer.segment": "segment",
"shipping.mode": "ship_mode",
"shipping.ship_date": "ship_date"}
items = items.rename(columns=schema)
print(items.columns.to_list())
cols = ["Product Name", "Product Container", "Product Base Margin"]
catalog = shop[cols].drop_duplicates()
catalog.columns = ["product", "container", "base_margin"]
print(len(catalog), catalog["product"].nunique())
in_csv = items["order_id"].isin(shop["Order ID"])
print(in_csv.mean())
Common mistake: merging without checking that the lookup keys are unique.
items.merge(catalog, on="product", how="left",
validate="many_to_one")
dupes = catalog[catalog["product"].duplicated(keep=False)]
dupes.sort_values("product").head(4)
catalog = catalog.drop_duplicates("product")
enriched = items.merge(catalog, on="product", how="left",
validate="many_to_one", indicator=True)
print(len(items), len(enriched))
enriched["_merge"].value_counts()
import os
path = "m11_items.csv"
enriched.drop(columns="_merge").to_csv(path, index=False)
items = pd.read_csv(path)
print(items.shape)
Your turn: bring the order-level shipping cost from flat onto each item, and check the row count.
costs = flat[["order_id", "shipping.cost"]]
costs = costs.rename(columns={"shipping.cost": "order_ship_cost"})
items = items.merge(costs, on="order_id", how="left",
validate="many_to_one")
print(items.shape, items["order_ship_cost"].isna().sum())
Engineer features from dates#
A date hides several useful features: gaps, months, weekdays.
Common mistake: doing date maths on dates stored as text.
items["ship_date"] - items["order_date"]
fmt = "%m/%d/%Y"
for col in ["order_date", "ship_date"]:
items[col] = pd.to_datetime(items[col], format=fmt)
gap = items["ship_date"] - items["order_date"]
items["days_to_ship"] = gap.dt.days
items["days_to_ship"].describe().round(2)
items["month"] = items["order_date"].dt.month
print(items["month"].unique())
items["weekday"] = items["order_date"].dt.day_name()
items["weekday"].value_counts().head(3)
per_day = items["days_to_ship"].value_counts().sort_index()
ax = per_day.plot(kind="bar", rot=0, figsize=(8, 4))
ax.set_title("API order lines by days to ship")
ax.set_xlabel("Days from order to shipping")
ax.set_ylabel("Order lines")
print(per_day.idxmax(), per_day.max())
by_mode = items.groupby("ship_mode")["days_to_ship"]
by_mode.agg(["size", "mean"]).round(2)
Build ratio, band and group features#
Ratios compare columns, bands simplify them, and groups add context.
items["margin"] = items["profit"] / items["sales"]
items["margin"].describe().round(3)
print(items["discount"].min(), items["discount"].max())
edges = [0, 0.03, 0.07, 0.25]
items["disc_band"] = pd.cut(items["discount"], bins=edges,
labels=["low", "mid", "high"],
include_lowest=True)
items["disc_band"].value_counts()
by_band = items.groupby("disc_band", observed=True)["margin"]
by_band.median().round(3)
by_cust = items.groupby("customer")["product"]
items["cust_lines"] = by_cust.transform("size")
items[["customer", "cust_lines"]].head(3)
Your turn: add order_total (the sum of sales in each order) and each item's share of it.
by_order = items.groupby("order_id")["sales"]
items["order_total"] = by_order.transform("sum")
items["share"] = items["sales"] / items["order_total"]
items.sort_values("order_total", ascending=False)[
["order_id", "sales", "order_total", "share"]].head(3).round(2)
Mini-project: which feature flags loss-making lines?#
Reuse our features on all 8,399 CSV lines and find the strongest warning sign.
def add_features(d):
d = d.copy()
for col in ["order_date", "ship_date"]:
d[col] = pd.to_datetime(d[col], format=fmt)
d["days_to_ship"] = (d["ship_date"] - d["order_date"]).dt.days
d["disc_band"] = pd.cut(d["discount"], bins=edges,
labels=["low", "mid", "high"],
include_lowest=True)
d["ship_share"] = d["shipping_cost"] / d["sales"]
return d
csv_schema = {"Order Date": "order_date", "Ship Date": "ship_date",
"Discount": "discount", "Sales": "sales",
"Shipping Cost": "shipping_cost", "Profit": "profit"}
full = add_features(shop.rename(columns=csv_schema))
full["loss"] = full["profit"] < 0
print(full.shape, round(full["loss"].mean(), 3))
Why not use margin here? It is built from profit, so it would leak the answer.
Quick check: which will track losses more closely, raw shipping cost or shipping cost as a share of the sale?
check = ["shipping_cost", "discount", "days_to_ship", "ship_share"]
full[check].corrwith(full["loss"]).round(3)
labels = ["Q1", "Q2", "Q3", "Q4"]
quart = pd.qcut(full["ship_share"], 4, labels=labels)
loss_rate = full.groupby(quart, observed=True)["loss"].mean()
loss_rate.round(3)
by_disc = full.groupby("disc_band", observed=True)["loss"].mean()
by_disc.round(3)
cut_off = full["ship_share"].quantile(0.75)
print(f"top quarter: shipping above {cut_off:.1%} of the sale")
ax = loss_rate.plot(kind="bar", rot=0, figsize=(7, 4))
ax.set_title("Loss-making lines by shipping share of sale")
ax.set_xlabel("Shipping cost / sales (quarters, low to high)")
ax.set_ylabel("Share of lines with a loss")
print(f"Q1: {loss_rate.iloc[0]:.0%} Q4: {loss_rate.iloc[-1]:.0%}")
Recap#
- Collect paged API results in a loop, then flatten with json_normalize
- Nested meta fields are lists of keys, not dotted strings
- Agree on one schema, then merge with validate and indicator
- Convert text dates with to_datetime before doing date maths
- Ratios, bands and group features often beat raw columns
Challenges#
- Add an order-level feature: the number of items in each order.
- Find the loss rate by ship mode and by discount band.
- Add a 'weekend order' flag and check whether it matters.
What's next#
Lesson 12: Dimensionality Reduction with PCA Many features? PCA squeezes them into a few that keep most of the information.
Data Mining 12: Dimensionality Reduction with PCA#
In this lesson you'll be able to:
- Explain why many correlated features are a problem
- Describe what PCA does, in plain words and one picture
- Scale features and fit PCA with scikit-learn
- Choose how many components to keep from explained variance
- Read components and use PCA inside a model pipeline Datasets: wine chemistry (data/mining/wine.csv) and breast cancer diagnostics (data/mining/breast_cancer.csv)
Warm-up: engineer one feature#
Lesson 11 built new features. Each one adds a column, and columns add up.
import pandas as pd
import numpy as np
wine = pd.read_csv("wine.csv")
print(wine.shape)
wine["cultivar"].value_counts()
ratio = wine["flavanoids"] / wine["total_phenols"]
print(ratio.groupby(wine["cultivar"]).mean().round(2))
See why too many features hurt#
Problem: 13 features are hard to plot, and many say the same thing.
X = wine.drop(columns="cultivar")
y = wine["cultivar"]
print(X.shape[1], "features")
Quick check: how many different two-feature scatter plots would we need to see every pair?
n = X.shape[1]
print(n * (n - 1) // 2)
corr = X.corr().abs()
upper = np.triu(np.ones(corr.shape, dtype=bool), k=1)
pairs = corr.where(upper).stack()
pairs.sort_values(ascending=False).head(3).round(2)
import matplotlib.pyplot as plt
import seaborn as sns
fig, ax = plt.subplots(figsize=(8, 6))
sns.heatmap(X.corr(), cmap="coolwarm", center=0, ax=ax)
ax.set_title("Wine: correlation between 13 features")
ax.set_xlabel("Feature")
ax.set_ylabel("Feature")
Picture PCA with two features#
PCA finds new axes: the direction with the most spread first, then the next, at right angles.
from sklearn.preprocessing import StandardScaler
two = ["total_phenols", "flavanoids"]
Z2 = StandardScaler().fit_transform(wine[two])
print(Z2[:3].round(2))
from sklearn.decomposition import PCA
pca2 = PCA().fit(Z2)
print(pca2.explained_variance_ratio_.round(3))
print(pca2.components_.round(2))
fig, ax = plt.subplots(figsize=(6, 6))
ax.scatter(Z2[:, 0], Z2[:, 1], alpha=0.6)
d = pca2.components_[0] * 3
ax.annotate("", xy=d, xytext=-d, arrowprops={"color": "red"})
ax.set_title("First principal component of two wine features")
ax.set_xlabel("Total phenols (scaled)")
ax.set_ylabel("Flavanoids (scaled)")
plt.show()
s1 = pca2.transform(Z2)[:3, 0]
by_hand = Z2[:3] @ pca2.components_[0]
print(s1.round(3), by_hand.round(3))
Your turn: keep only PC1 of these two features. What share of their spread survives?
one = PCA(n_components=1).fit(Z2)
print(one.transform(Z2).shape)
print(one.explained_variance_ratio_.round(3))
Scale first, then fit PCA#
PCA chases spread. Unscaled, the biggest numbers win.
print(X.var().sort_values().round(3).tail(3))
raw_pca = PCA().fit(X)
print(raw_pca.explained_variance_ratio_[:3].round(3))
Quick check: one component keeps almost everything. Good news, or a warning sign?
w1 = pd.Series(raw_pca.components_[0], index=X.columns)
print(w1.abs().sort_values(ascending=False).head(3).round(3))
print(X["proline"].var().round(0), X["hue"].var().round(3))
Z = StandardScaler().fit_transform(X)
pca = PCA().fit(Z)
ratios = pca.explained_variance_ratio_
print(ratios[:3].round(3))
Choose how many components to keep#
Keep enough components to explain most of the variance, often 90 or 95 percent.
cum = np.cumsum(ratios)
print(cum.round(3))
print(np.argmax(cum >= 0.9) + 1)
k = np.arange(1, len(ratios) + 1)
fig, ax = plt.subplots(figsize=(8, 4))
ax.bar(k, ratios, label="each component")
ax.plot(k, cum, marker="o", color="red", label="cumulative")
ax.axhline(0.9, linestyle="--", color="grey")
ax.set_title("Wine: variance explained by each component")
ax.set_xlabel("Principal component")
ax.set_ylabel("Share of variance")
ax.legend()
Common mistake: asking for more components than there are features.
PCA(n_components=20).fit(Z)
pca95 = PCA(n_components=0.95).fit(Z)
print(pca95.n_components_)
print(pca95.explained_variance_ratio_.sum().round(3))
Z8 = PCA(n_components=8).fit_transform(Z)
print(Z.shape, "->", Z8.shape)
Your turn: how many components keep at least 80 percent of the wine variance? Let PCA tell you.
p80 = PCA(n_components=0.8).fit(Z)
print(p80.n_components_, p80.explained_variance_ratio_.sum().round(3))
Read the components and plot the wines#
Each component is a weighted mix of the original features.
w = pd.DataFrame(pca.components_[:2].T, index=X.columns,
columns=["PC1", "PC2"])
top = w["PC1"].abs().sort_values(ascending=False).index[:4]
w.loc[top].round(2)
top2 = w["PC2"].abs().sort_values(ascending=False).index[:3]
w.loc[top2].round(2)
pcs = PCA(n_components=2).fit_transform(Z)
scores = pd.DataFrame(pcs, columns=["PC1", "PC2"])
scores["cultivar"] = y
scores.groupby("cultivar")[["PC1", "PC2"]].mean().round(2)
fig, ax = plt.subplots(figsize=(7, 5))
for name, grp in scores.groupby("cultivar"):
ax.scatter(grp["PC1"], grp["PC2"], label=name, alpha=0.7)
ax.set_title("178 wines: 13 measurements squeezed into 2")
ax.set_xlabel(f"PC1 ({ratios[0]:.0%} of variance)")
ax.set_ylabel(f"PC2 ({ratios[1]:.0%} of variance)")
ax.legend(title="Cultivar")
A new wine arrives: scale and project it with what we learned, never refit.
from sklearn.pipeline import make_pipeline
new = X.tail(3)
proj = make_pipeline(StandardScaler(), PCA(n_components=2))
proj.fit(X.iloc[:-3])
print(y.tail(3).to_list())
print(proj.transform(new).round(2))
known = proj.transform(X.iloc[:-3])
ref = pd.DataFrame(known, columns=["PC1", "PC2"])
ref.groupby(y.iloc[:-3].to_numpy()).mean().round(2)
Your turn: rebuild the 13 features from 2 components with inverse_transform. How much error is left?
p2 = PCA(n_components=2).fit(Z)
back = p2.inverse_transform(p2.transform(Z))
err = ((Z - back) ** 2).mean()
print(err.round(3), (1 - p2.explained_variance_ratio_.sum()).round(3))
Mini-project: fewer features, same accuracy?#
Classify 569 tumours from 30 measurements, with and without PCA.
cancer = pd.read_csv("breast_cancer.csv")
Xc = cancer.drop(columns="diagnosis")
yc = cancer["diagnosis"]
print(Xc.shape, yc.value_counts().to_dict())
cp = PCA().fit(StandardScaler().fit_transform(Xc))
print(np.cumsum(cp.explained_variance_ratio_)[:3].round(3))
from sklearn.model_selection import train_test_split
Xtr, Xte, ytr, yte = train_test_split(Xc, yc, test_size=0.25,
stratify=yc, random_state=42)
print(len(Xtr), len(Xte))
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
lr = LogisticRegression(max_iter=1000)
full_model = make_pipeline(StandardScaler(), lr)
pca_model = make_pipeline(StandardScaler(), PCA(n_components=0.95),
LogisticRegression(max_iter=1000))
for name, m in [("all 30", full_model), ("PCA 95%", pca_model)]:
cv = cross_val_score(m, Xtr, ytr, cv=5).mean()
print(name, "CV accuracy:", round(cv, 3))
sizes = [1, 2, 3, 5, 10, 20]
cv_acc = {}
for c in sizes:
m = make_pipeline(StandardScaler(), PCA(n_components=c),
LogisticRegression(max_iter=1000))
cv_acc[c] = cross_val_score(m, Xtr, ytr, cv=5).mean()
cv_acc = pd.Series(cv_acc)
print(cv_acc.round(3))
ax = cv_acc.plot(marker="o", figsize=(7, 4))
ax.set_title("Tumour classifier: accuracy vs PCA components")
ax.set_xlabel("Components kept (of 30)")
ax.set_ylabel("Cross-validated accuracy")
pca_model.fit(Xtr, ytr)
print("components kept:", pca_model[1].n_components_)
print("test accuracy:", round(pca_model.score(Xte, yte), 3))
full_model.fit(Xtr, ytr)
print("all 30 test:", round(full_model.score(Xte, yte), 3))
Quick check: for a cancer screen, which mistake is worse: a false alarm or a missed tumour?
from sklearn.metrics import confusion_matrix
pred = pca_model.predict(Xte)
labels = ["benign", "malignant"]
cm = confusion_matrix(yte, pred, labels=labels)
pd.DataFrame(cm, index=labels, columns=labels)
Recap#
- Correlated features repeat information; PCA combines them
- PCA finds new axes ordered by how much spread they capture
- Always scale first, or the biggest units take over
- Pick components with the cumulative explained variance
- Put PCA inside a pipeline and judge it on held-out data
Challenges#
- Plot the breast cancer tumours on PC1 and PC2, coloured by diagnosis.
- Try n_components of 2, 5 and 10 in the pipeline. Where does accuracy level off?
- Find the three features with the biggest PC1 weights for the cancer data.
What's next#
Lesson 13: Hands-on Preprocessing with Titanic and Iris Datasets Clean, transform, engineer and reduce, from raw file to model-ready table.
Data Mining 13: Hands-on Preprocessing with Titanic and Iris Datasets#
In this lesson you'll be able to:
- Audit a raw table for gaps, odd values and text columns
- Split first, then fill missing values without leaking test data
- Encode text columns and engineer new features
- Scale measurements so they share one yardstick
- Wrap every step in one reusable pipeline Datasets: Titanic passengers (data/pandas/titanic.csv) and Iris flowers (data/pandas/iris.csv)
Warm-up: PCA on the Iris flowers#
Lesson 12 squeezed four measurements into two components.
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
iris = pd.read_csv("iris.csv")
features = iris.drop(columns="species")
scaled = StandardScaler().fit_transform(features)
pca = PCA(n_components=2).fit(scaled)
print(pca.explained_variance_ratio_.round(3))
print(pca.explained_variance_ratio_.sum().round(3))
Audit the raw Titanic data#
Problem: find out what is broken before fixing anything.
titanic = pd.read_csv("titanic.csv")
print(titanic.shape)
titanic.head(3)
titanic.dtypes
missing = titanic.isna().sum()
missing[missing > 0]
share = titanic.isna().mean() * 100
share.round(1).sort_values(ascending=False).head(3)
Quick check: how many passengers have a recorded cabin?
print(titanic["Cabin"].notna().sum())
titanic[["Age", "Fare"]].describe().round(2)
zero_fare = titanic[titanic["Fare"] == 0]
print(len(zero_fare))
zero_fare[["Name", "Pclass", "Ticket"]].head(4)
Split first, then fill the gaps#
Problem: fill missing values without peeking at the test rows.
from sklearn.model_selection import train_test_split
X = titanic.drop(columns="Survived")
y = titanic["Survived"]
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, stratify=y, random_state=42)
print(X_train.shape, X_test.shape)
print(y_train.mean().round(3), y_test.mean().round(3))
Common mistake: handing a model columns that still have gaps.
from sklearn.linear_model import LogisticRegression
cols = ["Age", "Fare"]
model = LogisticRegression()
model.fit(X_train[cols], y_train)
from sklearn.impute import SimpleImputer
imputer = SimpleImputer(strategy="median")
train_num = imputer.fit_transform(X_train[cols])
test_num = imputer.transform(X_test[cols])
print(imputer.statistics_)
model.fit(train_num, y_train)
print(round(model.score(test_num, y_test), 3))
filled_age = pd.Series(train_num[:, 0])
print((filled_age == 28.5).sum())
Quick check: is one median age fair to every class?
class_age = X_train.groupby("Pclass")["Age"].median()
class_age
train = X_train.copy()
test = X_test.copy()
train_fill = train["Pclass"].map(class_age)
train["Age"] = train["Age"].fillna(train_fill)
test_fill = test["Pclass"].map(class_age)
test["Age"] = test["Age"].fillna(test_fill)
print(train["Age"].isna().sum(), test["Age"].isna().sum())
port = train["Embarked"].mode()[0]
train["Embarked"] = train["Embarked"].fillna(port)
test["Embarked"] = test["Embarked"].fillna(port)
print(port)
train["HasCabin"] = train["Cabin"].notna().astype(int)
test["HasCabin"] = test["Cabin"].notna().astype(int)
y_train.groupby(train["HasCabin"]).mean().round(3)
Turn text columns into numbers#
Problem: models and scalers only understand numbers.
Common mistake: scaling a table that still holds words.
scaler = StandardScaler()
scaler.fit(train[["Age", "Fare", "Sex"]])
sex_map = {"male": 0, "female": 1}
train["IsFemale"] = train["Sex"].map(sex_map)
test["IsFemale"] = test["Sex"].map(sex_map)
train[["Sex", "IsFemale"]].head(3)
y_train.groupby(train["IsFemale"]).mean().round(3)
Ports have no natural order, so each port gets its own 0/1 column.
from sklearn.preprocessing import OneHotEncoder
enc = OneHotEncoder(handle_unknown="ignore", sparse_output=False)
enc.fit(train[["Embarked"]])
print(enc.get_feature_names_out())
enc.transform(test[["Embarked"]])[:3]
Quick check: what does pd.get_dummies make from 3 test rows?
pd.get_dummies(test["Embarked"].head(3), dtype=int)
Engineer features that carry signal#
Problem: useful patterns hide inside raw columns.
train["FamilySize"] = train["SibSp"] + train["Parch"] + 1
test["FamilySize"] = test["SibSp"] + test["Parch"] + 1
rate = y_train.groupby(train["FamilySize"]).mean()
rate.round(2).head(6)
import matplotlib.pyplot as plt
ax = rate.plot(kind="bar", color="steelblue")
ax.set_title("Survival rate by family size (training rows)")
ax.set_xlabel("Family size")
ax.set_ylabel("Share who survived")
plt.show()
pattern = r",\s*([^.]+)\."
train["Title"] = train["Name"].str.extract(pattern)[0]
test["Title"] = test["Name"].str.extract(pattern)[0]
train["Title"].value_counts().head(6)
common = ["Mr", "Miss", "Mrs", "Master"]
keep = train["Title"].isin(common)
train["Title"] = train["Title"].where(keep, "Rare")
keep_test = test["Title"].isin(common)
test["Title"] = test["Title"].where(keep_test, "Rare")
train["Title"].value_counts()
Your turn: find the survival rate for each title in the training rows.
by_title = y_train.groupby(train["Title"]).mean()
by_title.round(2).sort_values()
Scale and split the Iris measurements#
Problem: put features on one yardstick and keep classes balanced.
print(iris.isna().sum().sum())
features.agg(["min", "max", "std"]).round(2)
ax = features.plot(kind="box", figsize=(8, 4))
ax.set_title("Iris measurements before scaling")
ax.set_ylabel("Centimetres")
plt.show()
species = iris["species"]
Xi_train, Xi_test, yi_train, yi_test = train_test_split(
features, species, test_size=0.2, stratify=species,
random_state=42)
yi_test.value_counts()
Quick check: after scaling, what mean will each training column have?
iris_scaler = StandardScaler().fit(Xi_train)
train_sc = pd.DataFrame(iris_scaler.transform(Xi_train),
columns=features.columns)
train_sc.agg(["mean", "std"]).round(2)
test_sc = pd.DataFrame(iris_scaler.transform(Xi_test),
columns=features.columns)
test_sc.agg(["mean", "std"]).round(2)
ax = train_sc.plot(kind="box", figsize=(8, 4))
ax.set_title("Iris measurements after standard scaling")
ax.set_ylabel("Standard deviations from the mean")
plt.show()
Your turn: turn the species names into numbers 0, 1 and 2.
from sklearn.preprocessing import LabelEncoder
le = LabelEncoder().fit(yi_train)
print(le.classes_)
pd.Series(le.transform(yi_test)).value_counts().sort_index()
Mini-project: does preprocessing pay off?#
Bundle every Titanic step into one pipeline, then score it on the test set.
def add_features(df):
out = df.copy()
out["FamilySize"] = out["SibSp"] + out["Parch"] + 1
out["HasCabin"] = out["Cabin"].notna().astype(int)
title = out["Name"].str.extract(pattern)[0]
out["Title"] = title.where(title.isin(common), "Rare")
return out
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import make_pipeline
num_cols = ["Age", "Fare", "FamilySize", "HasCabin", "Pclass"]
cat_cols = ["Sex", "Embarked", "Title"]
num_steps = make_pipeline(SimpleImputer(strategy="median"),
StandardScaler())
cat_steps = make_pipeline(SimpleImputer(strategy="most_frequent"),
OneHotEncoder(handle_unknown="ignore"))
prep = ColumnTransformer([("num", num_steps, num_cols),
("cat", cat_steps, cat_cols)])
clf = make_pipeline(prep, LogisticRegression(max_iter=1000))
clf.fit(add_features(X_train), y_train)
acc = clf.score(add_features(X_test), y_test)
print(round(acc, 3))
five = X_test.head(5)
guess = clf.predict(add_features(five))
pd.DataFrame({"guess": guess, "actual": y_test.head(5)})
import os
clean = prep.transform(add_features(X_train))
names = prep.get_feature_names_out()
clean_df = pd.DataFrame(clean, columns=names)
clean_df.to_csv("m13_titanic_train_clean.csv",
index=False)
back = pd.read_csv("m13_titanic_train_clean.csv")
print(back.shape)
print(int(back.isna().sum().sum()))
back.iloc[:3, :6].round(2)
gain = acc - model.score(test_num, y_test)
print(f"Pipeline: {acc:.1%}, gain: {gain * 100:.1f} points")
Recap#
- Audit first: gaps, odd values like zero fares, and text columns
- Split before you learn any fill value or scale
- Map two-level text, one-hot encode unordered categories
- Engineered features like Title and FamilySize carry real signal
- A ColumnTransformer pipeline repeats every step safely on new data
Challenges#
- Fill Age with the median for each Title instead of each class.
- Add IsAlone (FamilySize == 1) to the pipeline. Does accuracy move?
- Run PCA on the scaled Iris training data and plot the two components.
What's next#
Lesson 14: Decision Trees - ID3, C4.5, CART Our first real classifier, built on the data we just cleaned.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.



