Lesson 55 · Pandas Full Course
Clustering and Outlier Detection in Python: KMeans + Pandas | Pandas Course #55
In this lesson you'll be able to: Prepare and scale numeric features for clustering Group similar rows with KMeans and profile each cluster in pandas Choose…
- CoursePandas Full Course
- Lesson55 of 65
- FormatJupyter notebook · 39 code cells
- Data1 dataset
What you'll learn
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
- penguins.csv preview13.2 KB
📓 Full notebook
Download .ipynbPandas 55: Clustering outlier detection#
In this lesson you'll be able to:
- Prepare and scale numeric features for clustering
- Group similar rows with KMeans and profile each cluster in pandas
- Choose the number of clusters with inertia and silhouette
- Flag unusual rows with z-scores, IQR and Local Outlier Factor
- Check clusters and outliers against what you already know Dataset: Palmer penguins (data/pandas/penguins.csv)
Warm-up: features ready for scikit-learn#
Lesson 54 prepared data for scikit-learn. Same habits, new data.
import pandas as pd
df = pd.read_csv("data/pandas/penguins.csv")
print(df.shape)
df.head()
df.isna().sum()
Prepare the features#
Problem: clustering needs complete numbers on similar scales.
num = ["bill_length_mm", "bill_depth_mm", "flipper_length_mm",
"body_mass_g"]
df[num].describe().round(1)
Common mistake: clustering a table that still has gaps
from sklearn.cluster import KMeans
KMeans(n_clusters=3, n_init=10, random_state=42).fit(df[num])
p = df.dropna(subset=num).copy()
print(len(df), len(p))
Quick check: body mass runs into thousands, bill depth stays near 17. Which one would dominate a distance?
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler().set_output(transform="pandas")
X = scaler.fit_transform(p[num])
X.describe().loc[["mean", "std", "min", "max"]].round(2)
Group penguins with KMeans#
Problem: find natural groups without being told the answer.
km = KMeans(n_clusters=3, n_init=10, random_state=42)
p["cluster"] = km.fit_predict(X)
p["cluster"].value_counts().sort_index()
order = p.groupby("cluster")["flipper_length_mm"].mean().rank()
p["cluster"] = p["cluster"].map(order.astype(int))
p.groupby("cluster")[num].mean().round(1)
p.groupby("cluster")["island"].value_counts().unstack(fill_value=0)
Quick check: we never showed KMeans the species. Did it find them anyway?
pd.crosstab(p["species"], p["cluster"])
match = pd.crosstab(p["species"], p["cluster"]).max(axis=1)
print(match.sum(), len(p), round(match.sum() / len(p), 3))
raw = KMeans(n_clusters=3, n_init=10, random_state=42)
p["raw_cluster"] = raw.fit_predict(p[num])
pd.crosstab(p["species"], p["raw_cluster"])
Choose the number of clusters#
Problem: in real work nobody tells you k.
from sklearn.metrics import silhouette_score
rows = []
for k in range(2, 8):
model = KMeans(n_clusters=k, n_init=10, random_state=42)
labels = model.fit_predict(X)
sil = silhouette_score(X, labels)
rows.append({"k": k, "inertia": model.inertia_, "silhouette": sil})
scores = pd.DataFrame(rows).set_index("k").round(3)
scores
import matplotlib.pyplot as plt
fig, axes = plt.subplots(1, 2, figsize=(9, 3.5))
scores["inertia"].plot(ax=axes[0], marker="o")
axes[0].set_title("Inertia: lower is tighter")
scores["silhouette"].plot(ax=axes[1], marker="o", color="green")
axes[1].set_title("Silhouette: higher is better")
for ax in axes:
ax.set_xlabel("Number of clusters (k)")
axes[0].set_ylabel("Score")
plt.tight_layout()
from sklearn.metrics import silhouette_samples
p["sil"] = silhouette_samples(X, p["cluster"])
print(p.groupby("cluster")["sil"].agg(["mean", "min"]).round(3))
print((p["sil"] < 0).sum())
Your turn#
Silhouette prefers k = 2. Fit it and crosstab against species. What merged?
two = KMeans(n_clusters=2, n_init=10, random_state=42)
labels2 = two.fit_predict(X)
pd.crosstab(p["species"], labels2, colnames=["cluster"])
Scores guide the choice of k; domain knowledge makes the final call.
See the clusters in two dimensions with PCA#
Problem: four measurements can't be drawn on a flat screen.
from sklearn.decomposition import PCA
pca = PCA(n_components=2).set_output(transform="pandas")
proj = pca.fit_transform(X)
proj.head().round(2)
Quick check: how much of the original spread do two axes keep?
share = pd.Series(pca.explained_variance_ratio_,
index=["PC1", "PC2"])
print(share.round(3), round(share.sum(), 3))
ax = proj.plot.scatter(x="pca0", y="pca1", c=p["cluster"],
cmap="viridis", figsize=(7, 4.5))
ax.set_title("Penguin clusters on the first two PCA axes")
ax.set_xlabel("First principal component")
ax.set_ylabel("Second principal component")
plt.show()
Your turn#
Draw the same PCA scatter, coloured by species instead of cluster.
palette = {"Adelie": "darkorange", "Chinstrap": "purple",
"Gentoo": "teal"}
ax = proj.plot.scatter(x="pca0", y="pca1",
c=p["species"].map(palette), figsize=(7, 4.5))
ax.set_title("Penguin species on the first two PCA axes")
ax.set_xlabel("First principal component")
ax.set_ylabel("Second principal component")
plt.show()
Flag outliers one column at a time#
Problem: find measurements that are far from typical.
mass = p["body_mass_g"]
z = (mass - mass.mean()) / mass.std(ddof=0)
print((z - X["body_mass_g"]).abs().max().round(6))
z_out = (X.abs() > 3).sum()
print(z_out)
q1, q3 = p[num].quantile(0.25), p[num].quantile(0.75)
iqr = q3 - q1
low, high = q1 - 1.5 * iqr, q3 + 1.5 * iqr
((p[num] < low) | (p[num] > high)).sum()
Your turn#
Loosen the rule to 2.5 standard units. Which penguins break it in any column?
flag = (X.abs() > 2.5).any(axis=1)
print(flag.sum())
p.loc[flag, ["species"] + num]
Flag outliers with Local Outlier Factor#
Problem: a penguin can be normal in every column but odd overall.
from sklearn.neighbors import LocalOutlierFactor
lof = LocalOutlierFactor(n_neighbors=20, contamination=0.03)
p["outlier"] = lof.fit_predict(X) == -1
p["lof_score"] = -lof.negative_outlier_factor_
print(p["outlier"].sum())
p["lof_score"].describe().round(2)
ax = p["lof_score"].plot.hist(bins=30, figsize=(7, 3.5))
ax.set_title("Local Outlier Factor scores for 342 penguins")
ax.set_xlabel("LOF score (around 1 is normal)")
ax.set_ylabel("Number of penguins")
plt.show()
cols = ["species", "bill_length_mm", "flipper_length_mm",
"body_mass_g", "lof_score"]
top = p.sort_values("lof_score", ascending=False)
top[cols].head().round(2)
X.loc[169].round(2)
Common mistake: calling predict on a fitted LOF
lof.predict(X)
new_lof = LocalOutlierFactor(n_neighbors=20, novelty=True)
new_lof.fit(X.iloc[:300].to_numpy())
flags = new_lof.predict(X.iloc[300:].to_numpy())
print(len(flags), (flags == -1).sum())
Your turn#
Which species has the most LOF outliers, and which cluster?
by_species = p.groupby("species")["outlier"].sum()
by_cluster = p.groupby("cluster")["outlier"].sum()
print(by_species, by_cluster, sep="\n\n")
Mini-project: which penguins don't fit their species?#
Misfit = a penguin outside its species' main cluster.
table = pd.crosstab(p["species"], p["cluster"])
home = table.idxmax(axis=1)
p["misfit"] = p["cluster"] != p["species"].map(home)
p.groupby("species")["misfit"].agg(["sum", "mean"]).round(3)
adelie = p[p["species"] == "Adelie"]
adelie.groupby("misfit")[num].mean().round(1)
adelie.groupby("misfit")["sex"].value_counts().unstack(fill_value=0)
adelie.groupby("misfit")["island"].value_counts().unstack()
pd.crosstab(p["misfit"], p["outlier"])
colors = p["misfit"].map({False: "lightgrey", True: "crimson"})
ax = p.plot.scatter(x="bill_length_mm", y="bill_depth_mm",
c=colors, figsize=(7, 4.5))
ax.set_title("Penguins outside their species' main cluster")
ax.set_xlabel("Bill length (mm)")
ax.set_ylabel("Bill depth (mm)")
plt.show()
n_mis = p["misfit"].sum()
both = (p["misfit"] & p["outlier"]).sum()
print(f"{n_mis} misfits, of which {both} also flagged by LOF")
Recap#
- Clustering needs complete, scaled numeric features
- KMeans groups rows; profile clusters with groupby and crosstab
- Inertia and silhouette guide k; domain knowledge decides
- z-scores and IQR check one column; LOF checks all together
- LOF needs novelty=True to score new rows
Challenges#
- Cluster on bill length and bill depth only. Does agreement improve?
- Try LOF with n_neighbors=10 and 40. Which penguins stay flagged?
- Cluster each sex separately with k = 3 and compare the crosstabs.
What's next#
Lesson 56: Exploring Real Datasets with Pandas A full first look at a messy real dataset, start to finish.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.



