Lesson 4 · Data Mining Full Course
Data Mining Full Course Part 4 (4 Hours): Random Forest, XGBoost & Clustering in Python
All 10 lesson notebooks from Part 4 of the Data Mining Full Course in Python, in teaching order. Each lesson starts with a heading; the timestamps below…
- CourseData Mining Full Course
- Lesson4 of 6
- Video4 h 0 min
- FormatJupyter notebook · 367 code cells
- Data8 datasets
What you'll learn
- Data Mining 23: Introduction to Ensemble Learning
- Warm-up: set the baseline, as in lesson 22
- Why a crowd of models can beat one
- Let scikit-learn hold the vote
- Bagging: many trees on resampled rows
- Boosting: each model fixes the last one's mistakes
- Stacking: a model that learns whom to trust
- Judge ensembles fairly with cross-validation
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
- titanic.csv preview58.9 KB
- housing.csv preview1.4 MB
- breast_cancer.csv preview121.9 KB
- pima_diabetes.csv preview22.7 KB
- penguins.csv preview13.2 KB
- mpg.csv preview20.7 KB
- wine.csv preview13.0 KB
- gapminder.csv preview80.2 KB
📓 Full notebook
Download .ipynbData Mining Full Course, Part 4: Ensembles, Random Forest, XGBoost and Clustering#
All 10 lesson notebooks from Part 4 of the Data Mining Full Course in Python, in teaching order. Each lesson starts with a heading; the timestamps below jump to it in the video.
- 0:00 Lesson 23: Introduction to Ensemble Learning
- 0:23:30 Lesson 24: Bagging and Random Forests
- 0:47:10 Lesson 25: Boosting Techniques- AdaBoost and XGBoost
- 1:10:51 Lesson 26: Hands-on Project- Predicting Diabetes with Ensemble Models
- 1:35:13 Lesson 27: Introduction to Clustering
- 1:58:45 Lesson 28: k-Means Clustering Algorithm
- 2:23:20 Lesson 29: Hierarchical Clustering Techniques
- 2:46:39 Lesson 30: Density-Based Clustering with DBSCAN
- 3:11:20 Lesson 31: Cluster Evaluation Metrics and Interpretation
- 3:35:59 Lesson 32: Hands-on Clustering Project- Socio-Economic Segmentation
Data files: save the datasets from the lesson page in the same folder as this notebook. Files the lessons create are written to that folder too.
Data Mining 23: Introduction to Ensemble Learning#
In this lesson you'll be able to:
- Explain why a vote of several models can beat any one of them
- Build hard and soft voting ensembles in scikit-learn
- Recognise bagging, boosting and stacking, and try each one
- Judge ensembles fairly with cross-validation Dataset: the Titanic passenger list, 891 people (data/pandas/titanic.csv)
Warm-up: set the baseline, as in lesson 22#
Lesson 22 compared classifiers against a do-nothing baseline. Same habit, new data.
import pandas as pd
df = pd.read_csv("titanic.csv")
print(df.shape)
print(round(df["Survived"].mean(), 3))
cols = ["Pclass", "Sex", "Age", "SibSp", "Parch", "Fare", "Embarked"]
X = df[cols].copy()
X["Sex"] = X["Sex"].eq("male").astype(int)
X = pd.get_dummies(X, columns=["Embarked"], drop_first=True,
dtype=int)
y = df["Survived"]
X.head(3)
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, stratify=y, random_state=42)
age_fill = X_train["Age"].median()
X_train = X_train.fillna({"Age": age_fill})
X_test = X_test.fillna({"Age": age_fill})
print(age_fill, len(X_train), len(X_test))
from sklearn.dummy import DummyClassifier
base = DummyClassifier(strategy="most_frequent")
base.fit(X_train, y_train)
print(round(base.score(X_test, y_test), 3))
Why a crowd of models can beat one#
Problem: one model's mistakes are hard to fix; several can outvote them.
- Three models, each right 70% of the time, mistakes independent
- The vote is right whenever at least 2 of the 3 are right
from scipy.stats import binom
print(round(binom.sf(1, 3, 0.7), 3))
print(round(binom.sf(5, 11, 0.7), 3))
Real models are not independent, so the gain is smaller. Let's measure it.
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
models = {
"Logistic": make_pipeline(StandardScaler(),
LogisticRegression()),
"kNN": make_pipeline(StandardScaler(),
KNeighborsClassifier(n_neighbors=7)),
"Tree": DecisionTreeClassifier(max_depth=4, random_state=42),
}
preds = {}
for name, model in models.items():
model.fit(X_train, y_train)
preds[name] = model.predict(X_test)
print(f"{name:9} {model.score(X_test, y_test):.3f}")
Quick check: do the three models get the same passengers wrong?
votes = pd.DataFrame(preds, index=X_test.index)
wrong = votes.ne(y_test, axis=0)
print(wrong.sum().to_dict())
print(wrong.any(axis=1).sum(), wrong.all(axis=1).sum())
majority = (votes.sum(axis=1) >= 2).astype(int)
vote_acc = (majority == y_test).mean()
print(round(vote_acc, 3))
import matplotlib.pyplot as plt
acc = votes.eq(y_test, axis=0).mean()
acc["Majority vote"] = vote_acc
ax = acc.plot(kind="bar", rot=0, figsize=(7, 4), color="tab:blue")
ax.set_ylim(0.7, 0.82)
ax.set_title("Test accuracy: three models and their vote")
ax.set_xlabel("Model")
ax.set_ylabel("Accuracy on 223 test passengers")
plt.show()
Let scikit-learn hold the vote#
Problem: a hand-made vote can't be saved, tuned or cross-validated.
- Hard voting: each model casts one vote; the majority wins
- Soft voting: average the predicted probabilities, then decide
Common mistake: handing VotingClassifier bare models without names
from sklearn.ensemble import VotingClassifier
VotingClassifier(list(models.values())).fit(X_train, y_train)
members = list(models.items())
hard = VotingClassifier(members, voting="hard")
hard.fit(X_train, y_train)
print(round(hard.score(X_test, y_test), 3))
soft = VotingClassifier(members, voting="soft")
soft.fit(X_train, y_train)
print(round(soft.score(X_test, y_test), 3))
one = X_test.head(1)
for name, member in soft.named_estimators_.items():
p = member.predict_proba(one)[0, 1]
print(f"{name:9} {p:.2f}")
print(f"{'Average':9} {soft.predict_proba(one)[0, 1]:.2f}")
Your turn#
Build a hard vote of three identical depth-4 trees. Does it beat one tree?
clones = [(f"tree{i}", DecisionTreeClassifier(max_depth=4,
random_state=42))
for i in range(3)]
same = VotingClassifier(clones, voting="hard").fit(X_train, y_train)
print(round(same.score(X_test, y_test), 3))
Bagging: many trees on resampled rows#
Problem: a deep tree changes a lot when the data changes a little.
- Draw a bootstrap sample: rows picked at random, with replacement
- Train one tree per sample, then let all the trees vote
Quick check: a bootstrap sample of 668 rows. How many different passengers?
boot = X_train.sample(frac=1, replace=True, random_state=42)
print(len(boot), boot.index.nunique())
print(round(boot.index.nunique() / len(boot), 3))
deep = DecisionTreeClassifier(random_state=42)
deep.fit(X_train, y_train)
print(round(deep.score(X_train, y_train), 3))
print(round(deep.score(X_test, y_test), 3))
for seed in range(5):
rows = X_train.sample(frac=1, replace=True, random_state=seed)
t = DecisionTreeClassifier(random_state=42)
t.fit(rows, y_train.loc[rows.index])
print(seed, round(t.score(X_test, y_test), 3))
from sklearn.ensemble import BaggingClassifier
bag = BaggingClassifier(DecisionTreeClassifier(random_state=42),
n_estimators=100, random_state=42)
bag.fit(X_train, y_train)
print(round(bag.score(X_test, y_test), 3))
Boosting: each model fixes the last one's mistakes#
Problem: simple models miss the hard cases; boosting focuses on them.
- AdaBoost: give extra weight to the rows the last model missed
- Gradient boosting: each new tree predicts the errors left over
from sklearn.ensemble import AdaBoostClassifier
from sklearn.ensemble import GradientBoostingClassifier
ada = AdaBoostClassifier(n_estimators=100, random_state=42)
ada.fit(X_train, y_train)
print(round(ada.score(X_test, y_test), 3))
errors = pd.Series(ada.estimator_errors_)
print(errors.head(4).round(3).to_list())
print(round(errors.mean(), 3))
gb = GradientBoostingClassifier(random_state=42)
gb.fit(X_train, y_train)
print(round(gb.score(X_test, y_test), 3))
steps = [(p == y_test).mean() for p in gb.staged_predict(X_test)]
ax = pd.Series(steps, index=range(1, 101)).plot(figsize=(7, 4))
ax.set_title("Gradient boosting: accuracy as trees are added")
ax.set_xlabel("Number of trees")
ax.set_ylabel("Test accuracy")
plt.show()
print(round(steps[0], 3), round(max(steps), 3))
Your turn#
Train gradient boosting with only 10 trees (n_estimators=10). Better or worse?
gb10 = GradientBoostingClassifier(n_estimators=10, random_state=42)
gb10.fit(X_train, y_train)
print(round(gb10.score(X_test, y_test), 3))
Stacking: a model that learns whom to trust#
Problem: equal votes ignore that some members are more reliable.
from sklearn.ensemble import StackingClassifier
stack = StackingClassifier(members,
final_estimator=LogisticRegression(),
cv=5)
stack.fit(X_train, y_train)
print(round(stack.score(X_test, y_test), 3))
Quick check: which member will the top model trust most?
trust = stack.final_estimator_.coef_[0]
pd.Series(trust, index=list(models)).round(2)
Judge ensembles fairly with cross-validation#
Problem: on 223 test passengers, one person moves accuracy by almost half a point.
one_person = 1 / len(y_test)
print(round(one_person * 100, 2))
- Cross-validation splits the training rows into 5 folds
- Train on 4, test on the 5th, rotate, then average the 5 scores
Common mistake: cross-validating the raw table that still has gaps
from sklearn.model_selection import cross_val_score
cross_val_score(soft, X, y, cv=5)
cv = cross_val_score(soft, X_train, y_train, cv=5)
print(cv.round(3))
print(round(cv.mean(), 3), round(cv.std(), 3))
Quick check: soft voting lost to hard voting on the test set. Does it lose across 5 folds too?
cv_hard = cross_val_score(hard, X_train, y_train, cv=5)
print(round(cv_hard.mean(), 3), round(cv_hard.std(), 3))
Mini-project: which approach should we trust?#
Rank three single models and five ensembles by 5-fold cross-validation.
contenders = dict(models)
contenders["Voting (soft)"] = soft
contenders["Bagging"] = bag
contenders["AdaBoost"] = ada
contenders["Gradient boosting"] = gb
contenders["Stacking"] = stack
print(len(contenders))
rows = {}
for name, model in contenders.items():
s = cross_val_score(model, X_train, y_train, cv=5)
rows[name] = {"mean": s.mean(), "std": s.std()}
results = pd.DataFrame(rows).T.sort_values("mean")
results.round(3)
ax = results["mean"].plot(kind="barh", xerr=results["std"],
figsize=(7, 5), color="tab:green")
ax.set_xlim(0.7, 0.87)
ax.set_title("5-fold cross-validated accuracy on Titanic")
ax.set_xlabel("Mean accuracy (bars show one standard deviation)")
ax.set_ylabel("Model")
plt.show()
singles = results.loc[list(models), "mean"]
ensembles = results.drop(index=list(models))["mean"]
print(f"Best single: {singles.idxmax()} {singles.max():.3f}")
print(f"Best ensemble: {ensembles.idxmax()} {ensembles.max():.3f}")
print((ensembles > singles.max()).sum(), "of", len(ensembles))
print(round(gb.score(X_test, y_test), 3))
Recap#
- Ensembles combine several models so their mistakes can cancel out
- Voting only helps when members make different mistakes
- Bagging averages many trees on bootstrap samples
- Boosting adds models one by one to fix earlier errors
- Stacking learns how much to trust each member
- Compare with cross-validation; a small test set is noisy
Challenges#
- Add GaussianNB as a fourth voting member. Does the CV score rise?
- Give soft voting weights=[1, 2, 1]. What changes?
- Use 10-fold cross-validation for the top three. Does the ranking hold?
What's next#
Lesson 24: Bagging and Random Forests Go deeper into bagging and meet the random forest.
Data Mining 24: Bagging and Random Forests#
In this lesson you'll be able to:
- Explain why averaging many bootstrap trees steadies a jumpy tree
- Build bagged trees and random forests in scikit-learn
- Use the out-of-bag score to tune a forest without the test set
- Read feature importances, and check them with permutation importance Dataset: California housing, 1990 census districts (data/pandas/housing.csv)
Warm-up: from lesson 23 to house prices#
Lesson 23 met bagging as one of three ensemble families. Today it gets a whole lesson.
import pandas as pd
housing = pd.read_csv("housing.csv")
print(housing.shape)
housing["median_house_value"].describe().round(0)
df = housing.sample(6000, random_state=42)
capped = (df["median_house_value"] >= 500001).sum()
print(len(df), capped)
y = df["median_house_value"]
X = pd.get_dummies(df.drop(columns="median_house_value"),
columns=["ocean_proximity"], dtype=int)
print(X.shape)
X.columns[-5:].tolist()
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=42)
fill = X_train["total_bedrooms"].median()
X_train = X_train.fillna({"total_bedrooms": fill})
X_test = X_test.fillna({"total_bedrooms": fill})
print(len(X_train), len(X_test))
Why one deep tree is not enough#
Problem: a deep tree memorises its rows and changes with every sample.
- R squared: share of the price variation explained (1 is perfect)
- MAE: the average miss, in dollars
from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import mean_absolute_error
tree = DecisionTreeRegressor(random_state=42)
tree.fit(X_train, y_train)
print(round(tree.score(X_train, y_train), 3))
print(round(tree.score(X_test, y_test), 3))
print(round(mean_absolute_error(y_test, tree.predict(X_test))))
print(tree.get_depth(), tree.get_n_leaves())
Quick check: three trees, three bootstrap samples. Will they agree on one district?
district = X_test.head(1)
for seed in range(3):
rows = X_train.sample(frac=1, replace=True, random_state=seed)
t = DecisionTreeRegressor(random_state=42)
t.fit(rows, y_train.loc[rows.index])
print(seed, round(t.predict(district)[0]))
print("actual", y_test.iloc[0])
Bagging: average many bootstrap trees#
Problem: each tree is noisy, but their noise points in different directions.
- Grow each tree on its own bootstrap sample of the rows
- Average their predictions (or take a vote, for classes)
Common mistake: copying old code that says base_estimator
from sklearn.ensemble import BaggingRegressor
BaggingRegressor(base_estimator=DecisionTreeRegressor())
bag = BaggingRegressor(estimator=DecisionTreeRegressor(),
n_estimators=50, random_state=42)
bag.fit(X_train, y_train)
print(round(bag.score(X_test, y_test), 3))
print(round(mean_absolute_error(y_test, bag.predict(X_test))))
import numpy as np
X_test_np = X_test.to_numpy()
per_tree = np.array([t.predict(X_test_np) for t in bag.estimators_])
counts = np.arange(1, len(per_tree) + 1)[:, None]
running = per_tree.cumsum(axis=0) / counts
maes = [mean_absolute_error(y_test, p) for p in running]
print(round(maes[0]), round(maes[9]), round(maes[-1]))
import matplotlib.pyplot as plt
ax = pd.Series(maes, index=counts.ravel()).plot(figsize=(7, 4))
ax.set_title("Bagging: the average miss falls as trees are added")
ax.set_xlabel("Number of trees averaged")
ax.set_ylabel("Mean absolute error (dollars)")
plt.show()
Your turn#
Bag 50 trees of max_depth=3. Does bagging rescue a shallow tree?
short = DecisionTreeRegressor(max_depth=3, random_state=42)
short.fit(X_train, y_train)
print(round(mean_absolute_error(y_test, short.predict(X_test))))
bag3 = BaggingRegressor(estimator=short, n_estimators=50,
random_state=42).fit(X_train, y_train)
print(round(mean_absolute_error(y_test, bag3.predict(X_test))))
Random forests: bagging plus random feature choices#
Problem: bagged trees still look alike, because they all split on the same strong column.
- At each split, a tree may only look at a random subset of columns
- max_features sets the subset size, e.g. 0.5 = half the columns
- For regression, scikit-learn's default (1.0) is plain bagging
from sklearn.ensemble import RandomForestRegressor
forest = RandomForestRegressor(n_estimators=100, max_features=0.5,
random_state=42, n_jobs=-1)
forest.fit(X_train, y_train)
print(round(forest.score(X_test, y_test), 3))
print(round(mean_absolute_error(y_test, forest.predict(X_test))))
Quick check: do forest trees agree with each other more or less than bagged trees?
def agreement(model):
first20 = model.estimators_[:20]
preds = np.array([t.predict(X_test_np) for t in first20])
return np.corrcoef(preds).mean()
print(round(agreement(bag), 3), round(agreement(forest), 3))
models = {"One tree": tree, "Bagging": bag, "Forest": forest}
mae = pd.Series({name: mean_absolute_error(y_test, m.predict(X_test))
for name, m in models.items()})
ax = mae.plot(kind="bar", rot=0, figsize=(7, 4), color="tab:green")
ax.set_title("Average miss on 1,500 test districts")
ax.set_xlabel("Model")
ax.set_ylabel("Mean absolute error (dollars)")
plt.show()
The out-of-bag score: a free validation check#
Problem: we want to tune the forest without spending the test set.
- Each bootstrap sample leaves out about a third of the rows
- Those left-out rows grade the tree: that's the out-of-bag score
used = np.unique(bag.estimators_samples_[0]).size
print(used, round(1 - used / len(X_train), 3))
Common mistake: asking for oob_score_ without switching it on
forest.oob_score_
forest = RandomForestRegressor(n_estimators=100, max_features=0.5,
oob_score=True, random_state=42,
n_jobs=-1)
forest.fit(X_train, y_train)
print(round(forest.oob_score_, 3))
print(round(forest.score(X_test, y_test), 3))
Tune the forest without touching the test set#
Problem: pick max_features using out-of-bag scores only.
oob = {}
for share in [0.2, 0.33, 0.5, 1.0]:
rf = RandomForestRegressor(n_estimators=100, max_features=share,
oob_score=True, random_state=42,
n_jobs=-1).fit(X_train, y_train)
oob[share] = rf.oob_score_
pd.Series(oob).round(3)
ax = pd.Series(oob).plot(kind="bar", rot=0, figsize=(7, 4))
ax.set_ylim(0.74, 0.78)
ax.set_title("Out-of-bag R squared by max_features")
ax.set_xlabel("Share of columns tried at each split")
ax.set_ylabel("Out-of-bag R squared")
plt.show()
How many trees? Grow one forest step by step with warm_start.
grow = RandomForestRegressor(max_features=0.5, oob_score=True,
warm_start=True, random_state=42,
n_jobs=-1)
by_size = {}
for n in [25, 50, 100, 200]:
grow.set_params(n_estimators=n).fit(X_train, y_train)
by_size[n] = grow.oob_score_
pd.Series(by_size).round(3)
Your turn#
Keep max_features=0.5 and set min_samples_leaf=5. Does the OOB score rise or fall?
leafy = RandomForestRegressor(n_estimators=100, max_features=0.5,
min_samples_leaf=5, oob_score=True,
random_state=42, n_jobs=-1)
leafy.fit(X_train, y_train)
print(round(leafy.oob_score_, 3))
Read what the forest learned#
Problem: a valuation tool needs reasons, not just numbers.
Quick check: which column do you think matters most to district values?
imp = pd.Series(forest.feature_importances_, index=X.columns)
imp.nlargest(6).round(3)
Built-in importances favour columns with many distinct values. Check them.
from sklearn.inspection import permutation_importance
perm = permutation_importance(forest, X_test, y_test, n_repeats=5,
random_state=42, n_jobs=-1)
drops = pd.Series(perm.importances_mean, index=X.columns)
drops.nlargest(6).round(3)
Quick check: which three columns could we drop with almost no loss?
drops.nsmallest(3).round(4)
ax = drops.nlargest(8).sort_values().plot(kind="barh", figsize=(7, 4))
ax.set_title("What the forest relies on (permutation importance)")
ax.set_xlabel("Drop in test R squared when shuffled")
ax.set_ylabel("Feature")
plt.show()
Mini-project: could this forest value a district?#
How often is the forest within 20% of the true value, and where does it struggle?
check = df.loc[X_test.index, ["ocean_proximity"]].copy()
check["actual"] = y_test
check["predicted"] = forest.predict(X_test)
gap = (check["predicted"] - check["actual"]).abs()
check["pct_error"] = gap / check["actual"]
check.head().round(2)
within = (check["pct_error"] <= 0.2).mean()
median_err = check["pct_error"].median()
print(round(within, 3), round(median_err, 3))
by_area = check.groupby("ocean_proximity")["pct_error"]
by_area.agg(["count", "median"]).round(3)
ax = check.plot.scatter(x="actual", y="predicted", s=8, alpha=0.4,
figsize=(6, 6))
ax.plot([0, 500001], [0, 500001], "k--")
ax.set_title("Forest estimates against real district values")
ax.set_xlabel("Real median value (dollars)")
ax.set_ylabel("Predicted value (dollars)")
plt.show()
top = check["actual"] >= 500001
print(top.sum(), round(check.loc[top, "predicted"].mean()))
new = X_test.iloc[[4]]
trees = [t.predict(new.to_numpy())[0] for t in forest.estimators_]
low, high = np.percentile(trees, [10, 90])
print(round(low), round(high), y_test.iloc[4])
print(f"{within:.0%} of districts valued within 20%; "
f"typical miss {median_err:.0%}")
Recap#
- One deep tree memorises its data and changes with every sample
- Bagging averages trees grown on bootstrap samples, cutting the error
- Random forests also limit the columns per split (max_features), so trees differ more
- The out-of-bag score tunes the forest without touching the test set
- Check built-in importances with permutation importance
Challenges#
- Try max_features='sqrt' and compare its OOB score with the table.
- Drop latitude and longitude. How much does the OOB score fall?
- Fit a RandomForestClassifier on the Titanic data from lesson 23.
What's next#
Lesson 25: Boosting Techniques- AdaBoost and XGBoost Trees that learn one after another, each fixing earlier mistakes.
Data Mining 25: Boosting Techniques- AdaBoost and XGBoost#
In this lesson you'll be able to:
- Explain how boosting learns from the mistakes of earlier models
- Train and inspect an AdaBoost model built from tree stumps
- Use gradient boosting and see what the learning rate does
- Train XGBoost with early stopping on a validation set
- Compare boosted models fairly and read them in plain terms Dataset: Wisconsin breast cancer biopsies (data/mining/breast_cancer.csv)
Warm-up: a random forest as our baseline#
Lesson 24 grew many trees side by side and let them vote.
import pandas as pd
df = pd.read_csv("breast_cancer.csv")
print(df.shape)
df["diagnosis"].value_counts()
from sklearn.model_selection import train_test_split
X = df.drop(columns="diagnosis")
y = df["diagnosis"]
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, stratify=y, random_state=42)
print(X_train.shape, X_test.shape)
from sklearn.ensemble import RandomForestClassifier
forest = RandomForestClassifier(n_estimators=200, random_state=42)
forest.fit(X_train, y_train)
print(round(forest.score(X_test, y_test), 3))
Boosting in plain words: learn from mistakes#
Problem: a forest's trees never learn from each other.
- Bagging: many strong trees, trained side by side, then a vote
- Boosting: many weak models, trained one after another
- Each new model focuses on the cases the chain still gets wrong
from sklearn.tree import DecisionTreeClassifier
stump = DecisionTreeClassifier(max_depth=1, random_state=42)
stump.fit(X_train, y_train)
print(round(stump.score(X_test, y_test), 3))
Quick check: which single measurement would you split on?
feature = X.columns[stump.tree_.feature[0]]
cut = stump.tree_.threshold[0]
print(feature, round(cut, 1))
wrong = stump.predict(X_train) != y_train
print(wrong.sum())
y_train[wrong].value_counts()
AdaBoost: give the hard cases more weight#
Problem: turn many weak stumps into one strong classifier.
- Start with every training row weighted equally
- Fit a stump, then raise the weight of the rows it got wrong
- The next stump must try harder on those heavy rows
- Final answer: a weighted vote, better stumps get a bigger say
from sklearn.ensemble import AdaBoostClassifier
ada = AdaBoostClassifier(n_estimators=50, random_state=42)
ada.fit(X_train, y_train)
print(round(ada.score(X_test, y_test), 3))
firsts = ada.estimators_[:5]
names = [X.columns[s.tree_.feature[0]] for s in firsts]
print(names)
print(ada.estimator_weights_[:5].round(2))
Common mistake: copying
algorithm="SAMME.R"from an old tutorial
AdaBoostClassifier(algorithm="SAMME.R", random_state=42)
ada = AdaBoostClassifier(n_estimators=50, random_state=42)
print("algorithm" in ada.get_params())
How many rounds? Ask a validation set#
Never tune on the test set: carve a validation set from training.
X_fit, X_val, y_fit, y_val = train_test_split(
X_train, y_train, test_size=0.25, stratify=y_train,
random_state=42)
print(X_fit.shape, X_val.shape)
ada_big = AdaBoostClassifier(n_estimators=300, random_state=42)
A_fit, A_val = X_fit.to_numpy(), X_val.to_numpy()
ada_big.fit(A_fit, y_fit)
fit_acc = list(ada_big.staged_score(A_fit, y_fit))
val_acc = list(ada_big.staged_score(A_val, y_val))
best = max(val_acc)
print(val_acc.index(best) + 1, round(best, 3))
import matplotlib.pyplot as plt
rounds = range(1, len(val_acc) + 1)
plt.plot(rounds, fit_acc, label="fitting rows")
plt.plot(rounds, val_acc, label="validation rows")
plt.title("AdaBoost accuracy as stumps are added")
plt.xlabel("Number of boosting rounds")
plt.ylabel("Accuracy")
plt.legend()
plt.show()
Your turn#
Train AdaBoost with learning_rate=0.5 on the fitting rows and score it on validation.
slow = AdaBoostClassifier(n_estimators=100, learning_rate=0.5,
random_state=42)
slow.fit(X_fit, y_fit)
print(round(slow.score(X_val, y_val), 3))
Gradient boosting: fit what is still wrong#
Problem: correct the chain's errors directly, not via weights.
- Start from a simple guess: the overall malignant rate
- Each small tree predicts the gap between guess and truth
- Add a small step of that correction: the learning rate
from sklearn.ensemble import GradientBoostingClassifier
gb = GradientBoostingClassifier(n_estimators=100,
learning_rate=0.1, max_depth=3,
random_state=42)
gb.fit(X_fit, y_fit)
print(round(gb.score(X_val, y_val), 3))
patient = X_val.iloc[[0]]
print(y_val.iloc[0])
steps = list(gb.staged_predict_proba(patient))
for r in [1, 5, 20, 100]:
print(r, round(steps[r - 1][0, 1], 3))
Quick check: will a learning rate of 1.0 beat 0.1 on validation?
for lr in [0.01, 0.1, 1.0]:
m = GradientBoostingClassifier(learning_rate=lr,
random_state=42)
m.fit(X_fit, y_fit)
tr = round(m.score(X_fit, y_fit), 3)
va = round(m.score(X_val, y_val), 3)
print(lr, tr, va)
Watch out: deep trees do not make a better booster
deep = GradientBoostingClassifier(max_depth=None, random_state=42)
deep.fit(X_fit, y_fit)
print(len(deep.estimators_))
print(round(deep.score(X_val, y_val), 3))
XGBoost: gradient boosting with extra brakes#
Problem: boost fast, avoid overfitting, and stop at the right time.
- Same chain of small trees correcting each other
- Penalties on complex trees: regularisation
- Early stopping: halt when the validation score stops improving
Common mistake: giving XGBoost text labels
from xgboost import XGBClassifier
xgb = XGBClassifier(random_state=42)
xgb.fit(X_train, y_train)
yn_train = (y_train == "malignant").astype(int)
yn_test = (y_test == "malignant").astype(int)
yn_fit = (y_fit == "malignant").astype(int)
yn_val = (y_val == "malignant").astype(int)
print(yn_train.value_counts())
xgb = XGBClassifier(n_estimators=200, learning_rate=0.1,
max_depth=3, random_state=42)
xgb.fit(X_fit, yn_fit)
print(round(xgb.score(X_val, yn_val), 3))
xgb_es = XGBClassifier(n_estimators=500, learning_rate=0.1,
max_depth=3, early_stopping_rounds=20,
eval_metric="logloss", random_state=42)
xgb_es.fit(X_fit, yn_fit, eval_set=[(X_val, yn_val)],
verbose=False)
print(xgb_es.best_iteration)
print(round(xgb_es.score(X_val, yn_val), 3))
Your turn#
Repeat early stopping with max_depth=1 (stumps). Does it need more rounds?
stumps = XGBClassifier(n_estimators=500, learning_rate=0.1,
max_depth=1, early_stopping_rounds=20,
eval_metric="logloss", random_state=42)
stumps.fit(X_fit, yn_fit, eval_set=[(X_val, yn_val)],
verbose=False)
print(stumps.best_iteration)
Compare the models fairly, then test once#
Problem: one split is noisy. Cross-validate, then test once.
from sklearn.model_selection import cross_val_score
models = {
"forest": RandomForestClassifier(random_state=42),
"adaboost": AdaBoostClassifier(random_state=42),
"grad boost": GradientBoostingClassifier(random_state=42),
"xgboost": XGBClassifier(n_estimators=xgb_es.best_iteration + 1,
max_depth=3, random_state=42),
}
for name, model in models.items():
s = cross_val_score(model, X_train, yn_train, cv=5)
print(f"{name:11} {s.mean():.3f} +/- {s.std():.3f}")
from sklearn.metrics import recall_score
rows = []
for name, model in models.items():
pred = model.fit(X_train, yn_train).predict(X_test)
missed = int(((pred == 0) & (yn_test == 1)).sum())
rows.append({"model": name,
"accuracy": (pred == yn_test).mean(),
"recall": recall_score(yn_test, pred),
"missed": missed})
results = pd.DataFrame(rows).round(3)
results
Quick check: accuracies are close. Which column matters most to a doctor?
from sklearn.metrics import ConfusionMatrixDisplay
xgb_final = models["xgboost"]
ConfusionMatrixDisplay.from_predictions(
yn_test, xgb_final.predict(X_test),
display_labels=["benign", "malignant"])
plt.title("XGBoost on the test tumours")
plt.show()
imp = pd.Series(xgb_final.feature_importances_, index=X.columns)
top = imp.nlargest(10).sort_values()
ax = top.plot.barh(figsize=(7, 4))
ax.set_title("XGBoost: top 10 measurements")
ax.set_xlabel("Importance (share of total)")
ax.set_ylabel("Measurement")
plt.show()
Mini-project: can 5 measurements do the work of 30?#
Fewer measurements means a cheaper, simpler test for a clinic.
top5 = imp.nlargest(5).index.to_list()
print(top5)
small = XGBClassifier(n_estimators=xgb_es.best_iteration + 1,
max_depth=3, random_state=42)
s5 = cross_val_score(small, X_train[top5], yn_train, cv=5)
s30 = cross_val_score(models["xgboost"], X_train, yn_train, cv=5)
print(round(s5.mean(), 3), round(s30.mean(), 3))
small.fit(X_train[top5], yn_train)
pred5 = small.predict(X_test[top5])
acc5 = (pred5 == yn_test).mean()
missed5 = int(((pred5 == 0) & (yn_test == 1)).sum())
print(round(acc5, 3), missed5)
full = results.set_index("model").loc["xgboost"]
print(f"30 measurements: {full.accuracy:.1%}, missed {full.missed:.0f}")
print(f" 5 measurements: {acc5:.1%}, missed {missed5}")
Recap#
- Boosting trains weak models in a chain, each fixing earlier mistakes
- AdaBoost reweights hard rows; better stumps get a bigger vote
- Gradient boosting fits the remaining error, one small step at a time
- XGBoost needs 0/1 labels and can stop early on a validation set
- Tune on validation or cross-validation; test once at the end
Challenges#
- Plot XGBoost's validation log loss per round (
evals_result()). - Cross-validate AdaBoost with 50, 200 and 500 rounds. Which is best?
- Find the 5-measurement model's missed tumours and inspect their values.
What's next#
Lesson 26: Hands-on Project- Predicting Diabetes with Ensemble Models Clean real medical data, then compare forests, boosting and voting.
Data Mining 26: Hands-on Project- Predicting Diabetes with Ensemble Models#
In this lesson you'll be able to:
- Find missing values hiding as zeros in medical data
- Fill gaps inside a pipeline without leaking test information
- Compare simple models and ensembles with cross-validation
- Combine models with soft voting and stacking
- Test a final model once and tune it for screening Dataset: Pima Indians diabetes study, 768 women (data/mining/pima_diabetes.csv)
Warm-up: boosting on the raw data#
Lesson 25 boosted tree stumps with AdaBoost. Same tool, new data.
import numpy as np
import pandas as pd
df = pd.read_csv("pima_diabetes.csv")
print(df.shape)
df.head(3)
from sklearn.ensemble import AdaBoostClassifier
from sklearn.model_selection import cross_val_score
X_raw = df.drop(columns="outcome")
ada = AdaBoostClassifier(random_state=42)
s = cross_val_score(ada, X_raw, df["outcome"], cv=5)
print(round(s.mean(), 3))
Step 1: find the missing values hiding as zeros#
Problem: some zeros are not measurements, they are blanks.
print(df.isna().sum().sum())
(df == 0).sum()
Quick check: which of these columns can honestly be zero?
zero_cols = ["glucose", "blood_pressure", "skin_thickness",
"insulin", "bmi"]
(df[zero_cols] == 0).mean().round(3)
Common mistake: forcing gaps back into whole numbers
df["glucose"].replace(0, np.nan).astype(int)
clean = df.copy()
clean[zero_cols] = clean[zero_cols].replace(0, np.nan)
clean[zero_cols].isna().sum()
import matplotlib.pyplot as plt
import seaborn as sns
ax = sns.histplot(data=clean, x="glucose", hue="outcome",
bins=30, element="step")
ax.set_title("Glucose by diabetes outcome")
ax.set_xlabel("Plasma glucose (mg/dL)")
ax.set_ylabel("Number of women")
plt.show()
clean.groupby("outcome")[["glucose", "bmi", "age"]].median()
Step 2: split first, then fill the gaps#
Problem: filling with medians from all rows leaks the test set.
y = clean["outcome"]
y.value_counts(normalize=True).round(3)
from sklearn.model_selection import train_test_split
X = clean.drop(columns="outcome")
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, stratify=y, random_state=42)
print(X_train.shape, X_test.shape)
from sklearn.impute import SimpleImputer
imputer = SimpleImputer(strategy="median")
imputer.fit(X_train)
learned = pd.Series(imputer.statistics_, index=X.columns)
learned.round(1)
Quick check: do the test rows have the same medians as the training rows?
gap = pd.DataFrame({"train": X_train[zero_cols].median(),
"test": X_test[zero_cols].median()})
gap
from sklearn.pipeline import make_pipeline
from sklearn.tree import DecisionTreeClassifier
tree = make_pipeline(SimpleImputer(strategy="median"),
DecisionTreeClassifier(random_state=42))
tree
Step 3: set the bar with simple models#
Problem: an ensemble is only worth it if it beats simpler models.
- Accuracy: share of women labelled correctly
- Recall: share of diabetic women the model finds
- ROC AUC: how well it ranks patients by risk (0.5 = coin flip)
from sklearn.model_selection import StratifiedKFold, cross_validate
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
metrics = ["accuracy", "recall", "roc_auc"]
scores, models = {}, {}
def evaluate(name, model):
r = cross_validate(model, X_train, y_train, cv=cv,
scoring=metrics)
scores[name] = {m: r["test_" + m].mean() for m in metrics}
models[name] = model
return pd.Series(scores[name]).round(3)
from sklearn.dummy import DummyClassifier
dummy = make_pipeline(SimpleImputer(strategy="median"),
DummyClassifier())
evaluate("dummy", dummy)
evaluate("tree", tree)
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
logreg = make_pipeline(SimpleImputer(strategy="median"),
StandardScaler(), LogisticRegression())
evaluate("logreg", logreg)
Your turn#
Evaluate a tree limited to max_depth=4. Does it beat the full tree?
tree4 = make_pipeline(SimpleImputer(strategy="median"),
DecisionTreeClassifier(max_depth=4,
random_state=42))
evaluate("tree depth 4", tree4)
Step 4: ensembles head to head#
Problem: which ensemble, if any, beats logistic regression?
from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(n_estimators=100, min_samples_leaf=3,
random_state=42)
forest = make_pipeline(SimpleImputer(strategy="median"), rf)
evaluate("forest", forest)
adaboost = make_pipeline(SimpleImputer(strategy="median"),
AdaBoostClassifier(random_state=42))
evaluate("adaboost", adaboost)
from xgboost import XGBClassifier
xgb = XGBClassifier(n_estimators=200, learning_rate=0.05,
max_depth=3, n_jobs=1, random_state=42)
boost = make_pipeline(SimpleImputer(strategy="median"), xgb)
evaluate("xgboost", boost)
Common mistake: naming a metric after its function
cross_validate(tree, X_train, y_train, cv=cv,
scoring="recall_score")
r = cross_validate(tree, X_train, y_train, cv=cv,
scoring="recall")
print(r["test_score"].round(3))
board = pd.DataFrame(scores).T
board.sort_values("roc_auc", ascending=False).round(3)
Quick check: which models find the most diabetic women?
board["recall"].sort_values(ascending=False).head(3).round(3)
Step 5: combine models with voting and stacking#
Problem: different models make different mistakes. Pool them.
- Soft voting: average each model's probability of diabetes
- Stacking: a small referee model learns how to weigh them
from sklearn.model_selection import cross_val_predict
members = {"logreg": logreg, "forest": forest, "xgboost": boost}
probs = pd.DataFrame({
name: cross_val_predict(m, X_train, y_train, cv=cv,
method="predict_proba")[:, 1]
for name, m in members.items()})
probs.corr().round(2)
from sklearn.ensemble import VotingClassifier
team = [("logreg", logreg), ("forest", forest), ("xgboost", boost)]
vote = VotingClassifier(team, voting="soft")
evaluate("soft vote", vote)
from sklearn.ensemble import StackingClassifier
stack = StackingClassifier(team, cv=3,
final_estimator=LogisticRegression())
evaluate("stacking", stack)
Common mistake: asking a hard-voting model for probabilities
hard = VotingClassifier(team, voting="hard")
hard.fit(X_train, y_train)
hard.predict_proba(X_train.head(3))
vote.fit(X_train, y_train)
vote.predict_proba(X_train.head(3)).round(3)
board = pd.DataFrame(scores).T
board = board.sort_values("roc_auc")
ax = board["roc_auc"].plot.barh(figsize=(7, 4), xlim=(0.5, 0.9))
ax.set_title("Cross-validated ROC AUC on the training set")
ax.set_xlabel("ROC AUC (0.5 = coin flip)")
ax.set_ylabel("Model")
plt.show()
Final check: screen the test patients#
Pick the winner by cross-validation, then test it exactly once.
top = board.sort_values("roc_auc", ascending=False)
print(top["roc_auc"].head(3).round(3))
best = "soft vote"
final = models[best].fit(X_train, y_train)
print(final.named_estimators_.keys())
from sklearn.metrics import recall_score, roc_auc_score
proba = final.predict_proba(X_test)[:, 1]
pred = (proba >= 0.5).astype(int)
print(round((pred == y_test).mean(), 3))
print(round(recall_score(y_test, pred), 3))
print(round(roc_auc_score(y_test, proba), 3))
oof = cross_val_predict(models[best], X_train, y_train, cv=cv,
method="predict_proba")[:, 1]
for cut in [0.5, 0.4, 0.3, 0.2]:
flag = oof >= cut
found = flag[y_train == 1].mean()
sent = flag.mean()
print(cut, round(found, 3), round(sent, 3))
screen = (proba >= 0.3).astype(int)
print(round(recall_score(y_test, screen), 3))
print(round(screen.mean(), 3))
from sklearn.metrics import ConfusionMatrixDisplay
ConfusionMatrixDisplay.from_predictions(
y_test, screen, display_labels=["no diabetes", "diabetes"])
plt.title("Screening the test patients at a 0.3 cut-off")
plt.show()
from sklearn.inspection import permutation_importance
pi = permutation_importance(final, X_test, y_test, n_repeats=5,
scoring="roc_auc", random_state=42)
drops = pd.Series(pi.importances_mean, index=X.columns)
drops.sort_values(ascending=False).round(3)
new = X_test.head(3)
risk = final.predict_proba(new)[:, 1].round(3)
print(pd.DataFrame({"risk": risk, "flag": risk >= 0.3,
"actual": y_test.head(3).to_numpy()}))
Recap#
- Zeros in medical columns were missing values in disguise
- Imputing inside a pipeline keeps the test set out of training
- Judge every model with the same cross-validation folds
- Ensembles help a little here; a simple model came close
- Pick the cut-off on training data to match the clinic's goal
Challenges#
- Add
class_weight="balanced"to the forest. What happens to recall? - Add the depth-4 tree to the voting team. Does ROC AUC improve?
- Find the cut-off that flags 90% of diabetic women in training.
What's next#
Lesson 27: Introduction to Clustering No labels at all: let the data reveal its own groups.
Data Mining 27: Introduction to Clustering#
In this lesson you'll be able to:
- Explain what clustering is and when you'd use it
- Measure similarity as distance, and scale features fairly
- Group data with k-means and describe each cluster
- Compare choices of k and three clustering families
- Check clusters against known groups Dataset: Palmer penguins, 344 birds from Antarctica (data/pandas/penguins.csv)
Warm-up: classification needs labels#
Lesson 26 predicted a known label with ensembles. Here's that idea once more.
import pandas as pd
df = pd.read_csv("penguins.csv")
features = ["bill_length_mm", "bill_depth_mm",
"flipper_length_mm", "body_mass_g"]
peng = df.dropna(subset=features).reset_index(drop=True)
print(len(df), len(peng))
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
forest = RandomForestClassifier(random_state=42)
s = cross_val_score(forest, peng[features], peng["species"], cv=5)
print(round(s.mean(), 3))
Clustering: finding groups without labels#
Problem: no answer column, but we still want meaningful groups.
- Classification: learn from labelled examples, predict the label
- Clustering: no labels; put similar rows in the same group
- Uses: customer segments, patient types, grouping documents
X = peng[features]
X.describe().round(1)
import matplotlib.pyplot as plt
plt.scatter(X["bill_length_mm"], X["flipper_length_mm"],
color="grey", alpha=0.6)
plt.title("Penguins with no labels")
plt.xlabel("Bill length (mm)")
plt.ylabel("Flipper length (mm)")
plt.show()
Quick check: how many groups can you see in that cloud?
for name, grp in peng.groupby("species"):
plt.scatter(grp["bill_length_mm"], grp["flipper_length_mm"],
label=name, alpha=0.6)
plt.title("The same penguins, coloured by species")
plt.xlabel("Bill length (mm)")
plt.ylabel("Flipper length (mm)")
plt.legend()
plt.show()
Similarity is distance#
Problem: a computer needs a number for 'these two are alike'.
trio = peng.loc[[0, 1, 200], ["species"] + features]
trio
Common mistake: measuring distance with a text column included
from sklearn.metrics.pairwise import euclidean_distances
euclidean_distances(trio)
d = euclidean_distances(trio[features])
pd.DataFrame(d, index=trio.index, columns=trio.index).round(1)
Quick check: which column is dominating those distances?
X.std().round(1)
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
ds = euclidean_distances(X_scaled[[0, 1, 200]])
pd.DataFrame(ds, index=trio.index, columns=trio.index).round(2)
import numpy as np
gap = X_scaled[0] - X_scaled[1]
print(gap.round(2))
print(round(np.sqrt((gap ** 2).sum()), 2))
A first clustering with k-means#
Problem: assign every penguin to a group automatically.
- Choose k, the number of clusters, and drop k centres
- Give each bird to its nearest centre
- Move each centre to the middle of its birds; repeat until stable
Common mistake: feeding k-means rows with missing values
from sklearn.cluster import KMeans
km = KMeans(n_clusters=3, n_init=10, random_state=42)
km.fit(df[features])
km.fit(X_scaled)
peng["raw_label"] = km.labels_
peng["raw_label"].value_counts().sort_index()
order = peng.groupby("raw_label")["flipper_length_mm"].mean()
rank = order.rank().astype(int)
peng["cluster"] = peng["raw_label"].map(rank)
peng["cluster"].value_counts().sort_index()
peng.groupby("cluster")[features].mean().round(1)
centres = scaler.inverse_transform(km.cluster_centers_)
centres = pd.DataFrame(centres, columns=features)
centres.index = rank.to_list()
centres.sort_index().round(1)
for c, grp in peng.groupby("cluster"):
plt.scatter(grp["bill_length_mm"], grp["flipper_length_mm"],
label=f"cluster {c}", alpha=0.6)
plt.title("k-means clusters (k = 3, scaled data)")
plt.xlabel("Bill length (mm)")
plt.ylabel("Flipper length (mm)")
plt.legend()
plt.show()
near = km.transform(X_scaled[[0, 200]])
near = pd.DataFrame(near, columns=rank.to_list(), index=[0, 200])
near.sort_index(axis=1).round(2)
Quick check: which species do you expect in cluster 2?
pd.crosstab(peng["cluster"], peng["species"])
Your turn#
Run the same k-means on the unscaled measurements. How many birds per cluster?
km_raw = KMeans(n_clusters=3, n_init=10, random_state=42)
raw = km_raw.fit_predict(X)
peng.assign(raw=raw).groupby("raw")["body_mass_g"].agg(
["size", "min", "max"])
How many clusters? Let the data advise#
Problem: k-means needs k, but nobody tells us the right number.
inertia = []
for k in range(1, 9):
m = KMeans(n_clusters=k, n_init=10, random_state=42)
inertia.append(m.fit(X_scaled).inertia_)
print([round(v) for v in inertia])
plt.plot(range(1, 9), inertia, marker="o")
plt.title("Elbow chart: inertia by number of clusters")
plt.xlabel("Number of clusters (k)")
plt.ylabel("Inertia (lower = tighter)")
plt.show()
Quick check: silhouette scores run from -1 to 1. Which k will score best?
from sklearn.metrics import silhouette_score
for k in range(2, 7):
m = KMeans(n_clusters=k, n_init=10, random_state=42)
lab = m.fit_predict(X_scaled)
print(k, round(silhouette_score(X_scaled, lab), 3))
km2 = KMeans(n_clusters=2, n_init=10, random_state=42)
peng["k2"] = km2.fit_predict(X_scaled)
pd.crosstab(peng["k2"], peng["species"])
Your turn#
Fit k-means with k = 4. Which species gets split, and by what?
km4 = KMeans(n_clusters=4, n_init=10, random_state=42)
peng["k4"] = km4.fit_predict(X_scaled)
pd.crosstab(peng["k4"], [peng["species"], peng["sex"]])
Three families of clustering#
Problem: k-means likes round blobs. Other shapes need other methods.
- Centroid-based (k-means): groups around centre points
- Hierarchical: merge the closest groups step by step
- Density-based (DBSCAN): dense regions; sparse points are noise
from sklearn.cluster import AgglomerativeClustering
ward = AgglomerativeClustering(n_clusters=3, linkage="ward")
peng["ward"] = ward.fit_predict(X_scaled)
pd.crosstab(peng["ward"], peng["species"])
from sklearn.cluster import DBSCAN
db = DBSCAN(eps=0.6, min_samples=5)
peng["dbscan"] = db.fit_predict(X_scaled)
peng["dbscan"].value_counts().sort_index()
noise = peng["dbscan"] == -1
plt.scatter(peng.loc[~noise, "bill_length_mm"],
peng.loc[~noise, "flipper_length_mm"],
c=peng.loc[~noise, "dbscan"], alpha=0.6)
plt.scatter(peng.loc[noise, "bill_length_mm"],
peng.loc[noise, "flipper_length_mm"],
color="black", marker="x", label="noise")
plt.title("DBSCAN: dense groups and noise")
plt.xlabel("Bill length (mm)")
plt.ylabel("Flipper length (mm)")
plt.legend()
Your turn#
Repeat the hierarchical clustering with linkage="single". What goes wrong?
single = AgglomerativeClustering(n_clusters=3, linkage="single")
lab = single.fit_predict(X_scaled)
pd.Series(lab).value_counts()
Quick check: by silhouette, is Ward's 3-cluster answer better than k-means'?
print(round(silhouette_score(X_scaled, peng["cluster"]), 3))
print(round(silhouette_score(X_scaled, peng["ward"]), 3))
Mini-project: which measurements reveal the species?#
Cluster with different feature sets; score agreement with the hidden species.
from sklearn.metrics import adjusted_rand_score
truth = peng["species"]
print(round(adjusted_rand_score(truth, peng["cluster"]), 3))
print(round(adjusted_rand_score(truth, raw), 3))
print(round(adjusted_rand_score(truth, peng["ward"]), 3))
def cluster_score(cols):
Z = StandardScaler().fit_transform(peng[cols])
m = KMeans(n_clusters=3, n_init=10, random_state=42)
return round(adjusted_rand_score(truth, m.fit_predict(Z)), 3)
print(cluster_score(["bill_length_mm", "bill_depth_mm"]))
print(cluster_score(["flipper_length_mm", "body_mass_g"]))
print(cluster_score(["bill_length_mm", "flipper_length_mm"]))
print(cluster_score(features))
best_cols = ["bill_length_mm", "flipper_length_mm"]
Z = StandardScaler().fit_transform(peng[best_cols])
m = KMeans(n_clusters=3, n_init=10, random_state=42)
peng["pair"] = m.fit_predict(Z)
pd.crosstab(peng["pair"], peng["species"])
match = pd.crosstab(peng["pair"], peng["species"]).max(axis=1)
share = match.sum() / len(peng)
print(f"{share:.1%} of penguins share a cluster with their species")
Recap#
- Clustering groups similar rows without any labels
- Similarity is distance, so scale features first
- k-means: choose k, assign to nearest centre, move centres, repeat
- Elbow and silhouette advise on k; domain sense decides
- Hierarchical and DBSCAN suit other shapes of data
Challenges#
- Cluster with k = 3 using bill depth and body mass. Which species mix up?
- Try DBSCAN with eps 0.5 and 0.8. How do noise and cluster counts change?
- Cross-tab the k = 3 clusters against
sex. Do any clusters split by sex?
What's next#
Lesson 28: k-Means Clustering Algorithm Watch k-means step by step, pick k with care, and avoid its traps.
Data Mining 28: k-Means Clustering Algorithm#
In this lesson you'll be able to:
- Run the two steps of k-means by hand and watch them converge
- Explain why starting points matter, and how n_init helps
- Choose k with inertia, silhouette and business sense
- Name segments and place brand-new rows into them Dataset: 398 real cars from 1970-1982 (data/pandas/mpg.csv)
Warm-up: k-means on penguins#
Lesson 27 grouped penguins with k-means. One quick replay.
import pandas as pd
from sklearn.preprocessing import StandardScaler
peng = pd.read_csv("penguins.csv")
pair = ["bill_length_mm", "flipper_length_mm"]
peng = peng.dropna(subset=pair)
Zp = StandardScaler().fit_transform(peng[pair])
from sklearn.cluster import KMeans
km = KMeans(n_clusters=3, n_init=10, random_state=42)
peng["cluster"] = km.fit_predict(Zp)
pd.crosstab(peng["cluster"], peng["species"])
Getting cars ready for k-means#
Problem: 398 cars and no segment label. Can the data suggest some?
df = pd.read_csv("mpg.csv")
print(df.shape)
df[["name", "mpg", "horsepower", "weight", "model_year"]].head(3)
df.isna().sum()
df.loc[df["horsepower"].isna(), ["name", "model_year"]]
features = ["mpg", "horsepower", "weight", "acceleration"]
cars = df.dropna(subset=features).reset_index(drop=True)
cars[features].describe().round(1)
Quick check: which feature would swamp the distances if we left the units alone?
cars[features].std().round(1)
scaler = StandardScaler()
X = scaler.fit_transform(cars[features])
print(X.mean(axis=0).round(2), X.std(axis=0).round(2))
Watching k-means work, step by step#
Problem: k-means feels like magic until you run its steps yourself.
- Start: pick k rows as the first centres
- Assign: each car joins its nearest centre
- Update: move each centre to the mean of its cars; repeat 2-3
import matplotlib.pyplot as plt
plt.scatter(cars["weight"], cars["mpg"], color="grey", alpha=0.5)
plt.title("392 cars before clustering")
plt.xlabel("Weight (lb)")
plt.ylabel("Miles per gallon")
plt.show()
import numpy as np
two = ["weight", "mpg"]
Z = StandardScaler().fit_transform(cars[two])
start = cars.sample(3, random_state=42).index
centres = Z[start]
cars.loc[start, ["name"] + two]
from sklearn.metrics.pairwise import euclidean_distances
d = euclidean_distances(Z, centres)
labels = d.argmin(axis=1)
np.bincount(labels)
centres = np.array([Z[labels == j].mean(axis=0) for j in range(3)])
labels = euclidean_distances(Z, centres).argmin(axis=1)
np.bincount(labels)
Quick check: if we keep repeating, when should we stop?
def my_kmeans(Z, start, rounds=100):
centres, labels = Z[start], None
for step in range(1, rounds + 1):
new = euclidean_distances(Z, centres).argmin(axis=1)
if labels is not None and (new == labels).all():
break
labels = new
k = len(centres)
centres = np.array([Z[labels == j].mean(axis=0)
for j in range(k)])
return labels, centres, step
labels, centres, step = my_kmeans(Z, start)
inertia = ((Z - centres[labels]) ** 2).sum()
print("stopped at round", step, "sizes", np.bincount(labels))
print(round(inertia, 2))
plt.scatter(cars["weight"], cars["mpg"], c=labels, alpha=0.6)
sd = cars[two].std(ddof=0).to_numpy()
cent = centres * sd + cars[two].mean().to_numpy()
plt.scatter(cent[:, 0], cent[:, 1], c="red", marker="X", s=200)
plt.title("Our hand-made k-means: weight vs mpg")
plt.xlabel("Weight (lb)")
plt.ylabel("Miles per gallon")
plt.show()
Common mistake: giving KMeans starting centres that don't match k
check = KMeans(n_clusters=4, init=Z[start], n_init=1)
check.fit(Z)
check = KMeans(n_clusters=3, init=Z[start], n_init=1)
check.fit(Z)
print(np.bincount(check.labels_), round(check.inertia_, 2))
Starting points matter: k-means++ and n_init#
Problem: an unlucky start can trap k-means in a poorer answer.
for seed in range(5):
m = KMeans(n_clusters=6, init="random", n_init=1,
random_state=seed)
print(seed, round(m.fit(X).inertia_, 1))
- k-means++ (the default) spreads the first centres far apart
- n_init runs several starts and keeps the lowest inertia
for n in [1, 10]:
m = KMeans(n_clusters=6, n_init=n, random_state=42).fit(X)
print(n, round(m.inertia_, 1))
Your turn#
Run my_kmeans from 3 cars picked with random_state=7. Same clusters as before?
start7 = cars.sample(3, random_state=7).index
lab7, cen7, step7 = my_kmeans(Z, start7)
in7 = ((Z - cen7[lab7]) ** 2).sum()
print(step7, np.bincount(lab7), round(in7, 2))
Choosing k for the car market#
Problem: k-means needs k up front. The data can advise; we decide.
Common mistake: asking for a silhouette score with one cluster
from sklearn.metrics import silhouette_score
for k in range(1, 8):
m = KMeans(n_clusters=k, n_init=10, random_state=42)
print(k, silhouette_score(X, m.fit_predict(X)))
rows = []
for k in range(2, 8):
m = KMeans(n_clusters=k, n_init=10, random_state=42).fit(X)
sil = silhouette_score(X, m.labels_)
rows.append({"k": k, "inertia": m.inertia_, "silhouette": sil})
scores = pd.DataFrame(rows).set_index("k")
scores.round(3)
scores["inertia"].plot(marker="o")
plt.title("Inertia by number of car segments")
plt.xlabel("Number of clusters (k)")
plt.ylabel("Inertia (lower = tighter)")
plt.show()
scores["silhouette"].plot(kind="bar", color="steelblue")
plt.title("Silhouette score by number of car segments")
plt.xlabel("Number of clusters (k)")
plt.ylabel("Silhouette (higher = better separated)")
plt.show()
Quick check: k = 2 scores best. Is two segments useful to a car maker?
m2 = KMeans(n_clusters=2, n_init=10, random_state=42).fit(X)
cars.groupby(m2.labels_)[features].mean().round(1)
Naming the segments#
Problem: cluster numbers mean nothing to a sales team.
train = cars[cars["model_year"] < 82]
new = cars[cars["model_year"] == 82]
scaler = StandardScaler().fit(train[features])
Xtr = scaler.transform(train[features])
km = KMeans(n_clusters=3, n_init=10, random_state=42).fit(Xtr)
print(len(train), len(new))
heavy = train.groupby(km.labels_)["weight"].mean().rank()
names = {1: "economy", 2: "mid-size", 3: "heavy"}
seg_of = {c: names[int(r)] for c, r in heavy.items()}
train = train.assign(segment=[seg_of[c] for c in km.labels_])
train["segment"].value_counts()
train.groupby("segment")[features].mean().round(1)
centres = scaler.inverse_transform(km.cluster_centers_)
centres = pd.DataFrame(centres, columns=features)
centres.index = [seg_of[c] for c in range(3)]
centres.sort_index().round(1)
pd.crosstab(train["segment"], train["cylinders"])
Your turn#
Which share of each segment comes from the USA, Europe and Japan?
pd.crosstab(train["segment"], train["origin"],
normalize="index").round(2)
Placing new cars into segments#
Problem: the 1982 models arrive after the model was built.
Common mistake: new rows with the columns in a different order
shuffled = ["weight", "mpg", "horsepower", "acceleration"]
scaler.transform(new[shuffled])
Xnew = scaler.transform(new[features])
pred = [seg_of[c] for c in km.predict(Xnew)]
new = new.assign(segment=pred)
new[["name", "mpg", "weight", "segment"]].head(6)
Quick check: will any 1982 car land in the heavy segment?
new["segment"].value_counts()
car = (new["name"] == "ford mustang gl").to_numpy()
dist = km.transform(Xnew[car])[0]
centre_names = [seg_of[c] for c in range(3)]
pd.Series(dist, index=centre_names).round(2).sort_values()
d_all = np.sort(km.transform(Xnew), axis=1)
margin = d_all[:, 1] - d_all[:, 0]
new.assign(margin=margin.round(2)).nsmallest(3, "margin")[
["name", "mpg", "weight", "segment", "margin"]]
Mini-project: did buyers shift to economy cars?#
The 1970s brought two oil crises. Did the economy segment grow?
all_X = scaler.transform(cars[features])
cars = cars.assign(segment=[seg_of[c] for c in km.predict(all_X)])
share = pd.crosstab(cars["model_year"], cars["segment"],
normalize="index")
share.round(2).iloc[[0, 3, 6, 9, 12]]
share.plot(marker="o")
plt.title("Share of new car models by segment, 1970-1982")
plt.xlabel("Model year (19xx)")
plt.ylabel("Share of models")
plt.show()
eco = cars[cars["segment"] == "economy"]
eco.groupby("model_year")["mpg"].mean().round(1).iloc[[0, 6, 12]]
early = share.loc[70:72, "economy"].mean()
late = share.loc[80:82, "economy"].mean()
print(f"Economy share: {early:.0%} in 1970-72, {late:.0%} in 1980-82")
Recap#
- k-means repeats two steps: assign to the nearest centre, move centres
- It stops when no row changes cluster; inertia measures tightness
- Starts matter: k-means++ and n_init guard against bad answers
- Silhouette advises on k; business sense makes the final call
- Rank clusters to name them; predict new rows with the same scaler
Challenges#
- Add
displacementto the features. Do the three segments change? - Try k = 4. Which segment splits, and does the split make sense?
- Run
my_kmeanson all four features and compare withKMeans.
What's next#
Lesson 29: Hierarchical Clustering Techniques Build a family tree of clusters and choose k by cutting it.
Data Mining 29: Hierarchical Clustering Techniques#
In this lesson you'll be able to:
- Build a cluster tree from the bottom up with
linkage - Read a dendrogram and cut it into clusters
- Compare single, complete, average and Ward linkage
- Use AgglomerativeClustering, and place new rows afterwards Dataset: 398 real cars and their makers (data/pandas/mpg.csv)
Warm-up: three segments with k-means#
Lesson 28 found three car segments with k-means. One quick replay.
import pandas as pd
from sklearn.preprocessing import StandardScaler
df = pd.read_csv("mpg.csv")
features = ["mpg", "horsepower", "weight", "acceleration"]
cars = df.dropna(subset=features).reset_index(drop=True)
X = StandardScaler().fit_transform(cars[features])
from sklearn.cluster import KMeans
km = KMeans(n_clusters=3, n_init=10, random_state=42).fit(X)
rank = cars.groupby(km.labels_)["weight"].mean().rank().astype(int)
cars["km"] = [rank[c] for c in km.labels_]
cars["km"].value_counts().sort_index()
From cars to brands: tidying the names#
Problem: we want to compare car makers, but the brand is hidden in the name.
cars["brand"] = cars["name"].str.split().str[0]
counts = cars["brand"].value_counts()
print(len(counts))
counts.tail(12)
odd = cars["brand"].isin(["hi", "capri"])
cars.loc[odd, ["name", "model_year"]]
Quick check: which of those rare brands are really typos or nicknames?
fix = {"chevy": "chevrolet", "chevroelt": "chevrolet",
"vw": "volkswagen", "vokswagen": "volkswagen",
"maxda": "mazda", "toyouta": "toyota",
"mercedes": "mercedes-benz"}
cars["brand"] = cars["brand"].replace(fix)
print(cars["brand"].nunique())
counts = cars["brand"].value_counts()
keep = counts[counts >= 4].index
mask = cars["brand"].isin(keep)
brands = cars[mask].groupby("brand")[features].mean()
print(len(brands))
brands.round(1).head()
Quick check: which three brands have the best fuel economy?
brands["mpg"].nlargest(3).round(1)
Xb = StandardScaler().fit_transform(brands)
names = brands.index.to_list()
print(Xb.shape, names[:4])
Building the tree from the bottom up#
Problem: k-means needs k first. A tree lets us decide later.
- Start with every brand in its own cluster
- Merge the two closest clusters; repeat until one is left
- Ward picks the merge that adds the least spread
from scipy.cluster.hierarchy import linkage
Z = linkage(Xb, method="ward")
print(Z.shape)
Z[:3].round(3)
Each row of Z: cluster a, cluster b, merge distance, size Numbers 0-21 are brands; 22 and up are clusters made earlier
def label(i):
i = int(i)
return names[i] if i < len(names) else f"group {i}"
for a, b, height, size in Z[:6]:
print(label(a), "+", label(b), round(height, 2))
Quick check: why might Chevrolet and Ford merge first?
brands.loc[["chevrolet", "ford", "dodge", "plymouth"]].round(1)
Reading a dendrogram#
Problem: a merge table is hard to read. A dendrogram draws it.
Common mistake: labelling the leaves with the wrong list
from scipy.cluster.hierarchy import dendrogram
dendrogram(Z, labels=cars["brand"].to_list())
import matplotlib.pyplot as plt
plt.figure(figsize=(10, 4))
dendrogram(Z, labels=names, leaf_rotation=90)
plt.title("Ward dendrogram of 22 car brands")
plt.ylabel("Merge distance")
plt.show()
- Leaves at the bottom are brands; each join is a merge
- The height of a join is how different the two groups were
print(Z[-4:, 2].round(2))
Cutting the tree into clusters#
Problem: we still need flat labels. Cut by count or by height.
from scipy.cluster.hierarchy import fcluster
brands["tree3"] = fcluster(Z, t=3, criterion="maxclust")
for g, grp in brands.groupby("tree3"):
print(g, grp.index.to_list())
cut = fcluster(Z, t=6, criterion="distance")
print(cut.max(), "clusters")
plt.figure(figsize=(10, 4))
dendrogram(Z, labels=names, leaf_rotation=90, color_threshold=6)
plt.axhline(6, color="grey", linestyle="--")
plt.title("Cutting the brand tree at height 6")
plt.ylabel("Merge distance")
plt.show()
brands.groupby("tree3")[features].mean().round(1)
from sklearn.metrics import silhouette_score
for k in range(2, 7):
lab = fcluster(Z, t=k, criterion="maxclust")
print(k, round(silhouette_score(Xb, lab), 3))
Your turn#
Cut the tree into 4 clusters. Which group splits, and who leaves?
brands["tree4"] = fcluster(Z, t=4, criterion="maxclust")
pd.crosstab(brands["tree3"], brands["tree4"])
brands[brands["tree4"] == 1].index.to_list()
Linkage methods change the tree#
Problem: 'closest clusters' can be measured in several ways.
- single: the closest pair of members
- complete: the furthest pair of members
- average: the average over all pairs
- ward: the merge that adds the least spread
from scipy.cluster.hierarchy import cophenet
from scipy.spatial.distance import pdist
d = pdist(Xb)
for m in ["single", "complete", "average", "ward"]:
Zm = linkage(Xb, method=m)
r = cophenet(Zm, d)[0]
sizes = pd.Series(fcluster(Zm, 5, "maxclust")).value_counts()
print(m, round(r, 3), sorted(sizes, reverse=True))
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
for ax, m in zip(axes, ["single", "ward"]):
dendrogram(linkage(Xb, method=m), labels=names, ax=ax,
leaf_rotation=90)
ax.set_title(f"{m} linkage")
ax.set_ylabel("Merge distance")
plt.show()
Common mistake: setting both n_clusters and distance_threshold
from sklearn.cluster import AgglomerativeClustering
AgglomerativeClustering(n_clusters=3, distance_threshold=6).fit(Xb)
cut6 = AgglomerativeClustering(n_clusters=None, distance_threshold=6)
cut6.fit(Xb)
print(cut6.n_clusters_)
Clustering every car with scikit-learn#
Problem: 392 cars make an unreadable tree, but the clusters still help.
train = cars["model_year"] < 82
scaler = StandardScaler().fit(cars.loc[train, features])
Xtr = scaler.transform(cars.loc[train, features])
Xnew = scaler.transform(cars.loc[~train, features])
print(len(Xtr), len(Xnew))
agg = AgglomerativeClustering(n_clusters=3, linkage="ward")
cars["agg"] = 0
raw = agg.fit_predict(Xtr)
w = cars.loc[train, "weight"].groupby(raw).mean().rank().astype(int)
cars.loc[train, "agg"] = [w[c] for c in raw]
cars.loc[train, "agg"].value_counts().sort_index()
from sklearn.metrics import adjusted_rand_score
tr = cars[train]
print(pd.crosstab(tr["agg"], tr["km"]))
print(round(adjusted_rand_score(tr["agg"], tr["km"]), 3))
plt.scatter(tr["weight"], tr["mpg"], c=tr["agg"], alpha=0.6)
plt.title("Ward clusters of 362 cars")
plt.xlabel("Weight (lb)")
plt.ylabel("Miles per gallon")
plt.show()
Zc = linkage(Xtr, method="ward")
plt.figure(figsize=(10, 4))
dendrogram(Zc, truncate_mode="lastp", p=12)
plt.title("Top of the tree for 362 cars")
plt.ylabel("Merge distance")
plt.show()
Your turn#
Re-fit Ward on the training cars with 4 clusters. Which group splits?
agg4 = AgglomerativeClustering(n_clusters=4, linkage="ward")
lab4 = agg4.fit_predict(Xtr)
pd.crosstab(tr["agg"], lab4)
Common mistake: Ward linkage with a non-Euclidean distance
AgglomerativeClustering(n_clusters=3, linkage="ward",
metric="manhattan").fit(Xtr)
man = AgglomerativeClustering(n_clusters=3, linkage="average",
metric="manhattan").fit(Xtr)
pd.Series(man.labels_).value_counts().to_list()
Common mistake: asking agglomerative clustering to predict
agg.predict(Xnew)
from sklearn.neighbors import NearestCentroid
nc = NearestCentroid().fit(Xtr, cars.loc[train, "agg"])
cars.loc[~train, "agg"] = nc.predict(Xnew)
cars.loc[~train, "agg"].value_counts().sort_index()
Mini-project: do brands cluster by country?#
Our brand tree never saw origin. Does it split by country anyway?
origin = cars.groupby("brand")["origin"].first()
brands["origin"] = origin
pd.crosstab(brands["tree3"], brands["origin"])
euro = brands[brands["origin"] == "europe"]
euro[["tree3", "mpg", "weight", "acceleration"]].round(1)
from scipy.spatial.distance import squareform
grid = pd.DataFrame(squareform(d), index=names, columns=names)
grid["toyota"].drop("toyota").nsmallest(3).round(2)
home = brands.groupby("origin")["tree3"].agg(lambda s: s.mode()[0])
fits = (brands["tree3"] == brands["origin"].map(home)).mean()
print(f"{fits:.0%} of brands sit in their country's usual cluster")
Recap#
- Agglomerative clustering merges the closest groups, bottom up
- A dendrogram shows every merge; big jumps suggest where to cut
fclustercuts by number of clusters or by height- Linkage matters: single chains, Ward makes compact groups
- No
predict: use NearestCentroid to place new rows
Challenges#
- Keep brands with at least 2 cars. Where do BMW and Cadillac land?
- Build the brand tree with
completelinkage. Is Peugeot still alone? - Add
displacementto the features. Do the American brands still group?
What's next#
Lesson 30: Density-Based Clustering with DBSCAN Find clusters of any shape, and flag the points that fit nowhere.
Data Mining 30: Density-Based Clustering with DBSCAN#
In this lesson you'll be able to:
- Explain core points, border points and noise
- Cluster map coordinates with DBSCAN and haversine distance
- Choose eps with a k-distance plot, then tune min_samples
- Profile clusters and noise to answer a business question Dataset: 20,640 California districts, 1990 census (data/pandas/housing.csv)
Warm-up: Ward clusters on cars#
Lesson 29 built cluster trees. One quick replay on the cars.
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import AgglomerativeClustering
cars = pd.read_csv("mpg.csv")
features = ["mpg", "horsepower", "weight", "acceleration"]
cars = cars.dropna(subset=features)
X = StandardScaler().fit_transform(cars[features])
ward = AgglomerativeClustering(n_clusters=3, linkage="ward")
print(sorted(ward.fit_predict(X).tolist().count(c) for c in range(3)))
Every car got a cluster, even the oddballs. Today: a method that can say 'none'.
Where k-means falls short#
Problem: real groups have odd shapes, and some points belong nowhere.
housing = pd.read_csv("housing.csv")
print(housing.shape)
cols = ["longitude", "latitude", "median_income",
"median_house_value"]
housing[cols].head(3)
top = housing["median_house_value"].max()
print(top, (housing["median_house_value"] == top).sum())
import matplotlib.pyplot as plt
plt.scatter(housing["longitude"], housing["latitude"], s=1,
color="grey")
plt.title("20,640 California districts")
plt.xlabel("Longitude")
plt.ylabel("Latitude")
plt.show()
import numpy as np
from sklearn.cluster import KMeans
where = housing[["latitude", "longitude"]]
km = KMeans(n_clusters=4, n_init=10, random_state=42).fit(where)
plt.scatter(where["longitude"], where["latitude"], s=1,
c=km.labels_)
plt.title("k-means: four clusters of districts")
plt.xlabel("Longitude")
plt.ylabel("Latitude")
plt.show()
Quick check: what's wrong with putting a lonely desert district in the LA cluster?
print(sorted(km.labels_.tolist().count(c) for c in range(4)))
Your turn#
Try k-means with k = 8. Are any lonely districts left out now?
km8 = KMeans(n_clusters=8, n_init=10, random_state=42).fit(where)
sizes8 = np.bincount(km8.labels_)
print(sorted(sizes8.tolist()), sizes8.sum())
How DBSCAN thinks: core, border and noise#
Problem: we want clusters to grow only where points are crowded.
- eps: the radius of each point's neighbourhood
- min_samples: points needed inside that radius to be a core point
- Border: near a core point but not core itself; noise: neither
coords = np.radians(housing[["latitude", "longitude"]])
EARTH_KM = 6371
eps = 8 / EARTH_KM
print(round(eps, 5))
from sklearn.neighbors import NearestNeighbors
nn = NearestNeighbors(radius=eps, metric="haversine").fit(coords)
rows = [0, housing["longitude"].idxmax()]
found = nn.radius_neighbors(coords.iloc[rows],
return_distance=False)
for r, f in zip(rows, found):
print(housing.loc[r, ["longitude", "latitude"]].to_list(), len(f))
Common mistake: giving haversine more than two columns
from sklearn.cluster import DBSCAN
extra = np.radians(housing[["latitude", "longitude", "median_income"]])
DBSCAN(eps=eps, min_samples=50, metric="haversine").fit(extra)
db = DBSCAN(eps=eps, min_samples=50, metric="haversine")
labels = db.fit_predict(coords)
print(labels.max() + 1, "clusters")
print((labels == -1).sum(), "noise districts")
core = np.zeros(len(labels), dtype=bool)
core[db.core_sample_indices_] = True
kind = np.where(core, "core", np.where(labels == -1, "noise",
"border"))
pd.Series(kind).value_counts()
b = np.flatnonzero(kind == "border")[0]
near = nn.radius_neighbors(coords.iloc[[b]], return_distance=False)
print(len(near[0]), labels[b] >= 0)
la = housing["latitude"].between(33.5, 34.5) & housing[
"longitude"].between(-119, -117)
for name, colour in [("core", "tab:blue"), ("border", "orange"),
("noise", "black")]:
pick = la.to_numpy() & (kind == name)
plt.scatter(housing.loc[pick, "longitude"],
housing.loc[pick, "latitude"], s=2, c=colour,
label=name)
plt.title("Core, border and noise around Los Angeles")
plt.xlabel("Longitude")
plt.ylabel("Latitude")
plt.legend()
Choosing eps with a k-distance plot#
Problem: 8 km was a guess. The data can suggest a radius.
- For each point, find the distance to its 50th nearest neighbour
- Sort them and plot: the bend marks where crowds end
nn50 = NearestNeighbors(n_neighbors=50, metric="haversine")
dist, _ = nn50.fit(coords).kneighbors(coords)
kdist = np.sort(dist[:, -1]) * EARTH_KM
print(np.percentile(kdist, [25, 50, 75, 90]).round(1))
plt.plot(kdist)
plt.axhline(8, color="red", linestyle="--")
plt.ylim(0, 60)
plt.title("k-distance plot: 50th nearest neighbour")
plt.xlabel("Districts, sorted")
plt.ylabel("Distance to 50th neighbour (km)")
plt.show()
Quick check: what share of districts would be core with eps = 8 km?
print(round((kdist <= 8).mean(), 3))
Tuning eps and min_samples#
Problem: small changes can merge cities or shatter them.
def summary(eps_km, min_samples):
m = DBSCAN(eps=eps_km / EARTH_KM, min_samples=min_samples,
metric="haversine")
lab = m.fit_predict(coords)
return {"eps_km": eps_km, "min_samples": min_samples,
"clusters": lab.max() + 1,
"noise": round((lab == -1).mean(), 3)}
Quick check: as eps grows, does noise go up or down?
tests = [summary(e, 50) for e in [3, 5, 8, 12]]
pd.DataFrame(tests)
tests = [summary(8, m) for m in [10, 50, 200]]
pd.DataFrame(tests)
sweep = pd.DataFrame([summary(e, 50) for e in range(2, 21, 2)])
fig, ax = plt.subplots(1, 2, figsize=(10, 3.5))
sweep.plot(x="eps_km", y="clusters", ax=ax[0], legend=False)
sweep.plot(x="eps_km", y="noise", ax=ax[1], legend=False)
ax[0].set_title("Clusters by eps")
ax[1].set_title("Noise share by eps")
ax[0].set_ylabel("Clusters")
ax[1].set_ylabel("Share of districts")
plt.show()
Your turn#
With min_samples=20, find an eps that leaves under 5% noise. How many clusters?
pd.DataFrame([summary(e, 20) for e in [8, 12, 15]])
Reading the clusters#
Problem: cluster numbers mean nothing until we profile them.
sizes = pd.Series(labels[labels >= 0]).value_counts()
rank = {c: i for i, c in enumerate(sizes.index, start=1)}
housing["cluster"] = [rank.get(c, -1) for c in labels]
housing["cluster"].value_counts().head(6)
profile = housing.groupby("cluster").agg(
districts=("cluster", "size"),
lat=("latitude", "mean"), lon=("longitude", "mean"),
income=("median_income", "median"),
value=("median_house_value", "median"))
profile.loc[[-1, 1, 2, 3, 4, 5]].round(2)
Quick check: using the map centres, which cities are clusters 1 to 5?
big = housing[housing["cluster"].between(1, 5)]
noise = housing[housing["cluster"] == -1]
plt.scatter(noise["longitude"], noise["latitude"], s=1, c="grey")
plt.scatter(big["longitude"], big["latitude"], s=1, c=big["cluster"],
cmap="tab10")
plt.title("DBSCAN: five biggest clusters, noise in grey")
plt.xlabel("Longitude")
plt.ylabel("Latitude")
plt.show()
Your turn#
Which three real clusters have the highest typical income?
profile.drop(-1).nlargest(3, "income")[["districts", "lat", "lon",
"income"]].round(2)
Watch out: -1 is a label for noise, not a cluster
print(housing["cluster"].nunique())
print(housing.loc[housing["cluster"] >= 0, "cluster"].nunique())
real = profile.drop(-1)
cheap = real.nsmallest(3, "value")
cheap[["districts", "lat", "lon", "value"]].round(2)
Placing new districts#
Problem: a new district arrives. Which cluster, or is it noise?
Common mistake: expecting DBSCAN to have a predict method
db.predict(coords.iloc[:3])
A new point joins a cluster only if it lies within eps of a core point.
seen = housing.sample(frac=0.95, random_state=42).index
new = housing.index.difference(seen)
db2 = DBSCAN(eps=eps, min_samples=50, metric="haversine")
db2.fit(coords.loc[seen])
print(len(seen), len(new), db2.labels_.max() + 1)
core_idx = db2.core_sample_indices_
core_xy = coords.loc[seen].iloc[core_idx]
core_lab = db2.labels_[core_idx]
nn1 = NearestNeighbors(n_neighbors=1, metric="haversine")
d, i = nn1.fit(core_xy).kneighbors(coords.loc[new])
new_lab = np.where(d[:, 0] <= eps, core_lab[i[:, 0]], -1)
print(round((new_lab == -1).mean(), 3))
full_noise = labels[new.to_numpy()] == -1
print(round(((new_lab == -1) == full_noise).mean(), 3))
Your turn#
Use the core-point rule on the most easterly district. Cluster or noise?
east = housing["longitude"].idxmax()
d_e, _ = nn1.kneighbors(coords.loc[[east]])
km_away = d_e[0, 0] * EARTH_KM
print(round(km_away, 1), "noise" if km_away > 8 else "cluster")
Mini-project: are crowded districts pricier?#
Do districts inside dense clusters cost more than the scattered noise?
housing["setting"] = np.where(housing["cluster"] == -1,
"scattered", "clustered")
housing.groupby("setting")[["median_income",
"median_house_value"]].median()
pd.crosstab(housing["setting"], housing["ocean_proximity"],
normalize="index").round(2)
housing.boxplot(column="median_house_value", by="setting",
showfliers=False)
plt.title("House value: clustered vs scattered districts")
plt.suptitle("")
plt.xlabel("")
plt.ylabel("Median house value ($)")
plt.show()
inland = housing[housing["ocean_proximity"] == "INLAND"]
v = inland.groupby("setting")["median_house_value"].median()
gap = v["clustered"] / v["scattered"] - 1
print(v.round(), f"premium: {gap:.0%}", sep="\n")
Recap#
- DBSCAN grows clusters from core points in crowded areas
- Points near no crowd are noise, labelled -1; no k needed
- Map data: haversine distance, radians, latitude first
- Pick eps from a k-distance plot; min_samples sets the crowd size
- Profile noise separately; never treat -1 as a cluster
Challenges#
- Run DBSCAN with eps = 3 km. Does Los Angeles split into several clusters?
- Scale
median_incomeandmedian_house_valueand cluster them with DBSCAN. - Profile the noise districts by
housing_median_age. Are they older or newer?
What's next#
Lesson 31: Cluster Evaluation Metrics and Interpretation Score and compare clusterings, with and without known labels.
Data Mining 31: Cluster Evaluation Metrics and Interpretation#
In this lesson you'll be able to:
- Read inertia and the elbow, and know their limits
- Compute and plot the silhouette score, by hand and with sklearn
- Compare k with Davies-Bouldin and Calinski-Harabasz
- Check clusters against known labels with the adjusted Rand index
- Turn cluster profiles into plain-English descriptions Dataset: 178 Italian wines, 13 chemical measurements (data/mining/wine.csv)
Warm-up: DBSCAN from lesson 30#
Lesson 30 grouped points by density. Here it is on our wines.
import pandas as pd
from sklearn.preprocessing import StandardScaler
w = pd.read_csv("wine.csv")
features = w.columns.drop("cultivar")
X = StandardScaler().fit_transform(w[features])
print(X.shape)
from sklearn.cluster import DBSCAN
db = DBSCAN(eps=2.2, min_samples=5).fit_predict(X)
pd.Series(db).value_counts()
Inertia and the elbow: how tight are the clusters?#
Problem: we need a number that says how compact each cluster is.
- Inertia: total squared distance from each row to its centre
- Lower means tighter clusters, but it always falls as k grows
from sklearn.cluster import KMeans
km3 = KMeans(n_clusters=3, n_init=10, random_state=42).fit(X)
print(round(km3.inertia_, 1))
centres = km3.cluster_centers_[km3.labels_]
print(round(((X - centres) ** 2).sum(), 1))
import numpy as np
order = km3.cluster_centers_[:, features.get_loc("proline")]
lab = order.argsort().argsort()[km3.labels_]
np.bincount(lab)
from sklearn.decomposition import PCA
xy = PCA(n_components=2).fit_transform(X)
plt.scatter(xy[:, 0], xy[:, 1], c=lab, cmap="viridis", alpha=0.7)
plt.title("Three k-means clusters, seen through PCA")
plt.xlabel("First principal component")
plt.ylabel("Second principal component")
plt.show()
inertia = {}
for k in range(1, 9):
km = KMeans(n_clusters=k, n_init=10, random_state=42).fit(X)
inertia[k] = km.inertia_
inertia = pd.Series(inertia).round(1)
inertia
import matplotlib.pyplot as plt
inertia.plot(marker="o")
plt.title("Elbow chart for the wines")
plt.xlabel("Number of clusters k")
plt.ylabel("Inertia")
plt.show()
Quick check: if inertia always falls, why not pick k = 178?
km_all = KMeans(n_clusters=178, n_init=1, random_state=42).fit(X)
print(km_all.inertia_)
Silhouette: tight inside, far from the neighbours#
Problem: inertia ignores how far apart the clusters are.
- a: average distance to the other members of my cluster
- b: average distance to the nearest other cluster
- silhouette = (b - a) / max(a, b): near 1 good, 0 on the border, below 0 misplaced
from sklearn.metrics import pairwise_distances
d = pairwise_distances(X[:1], X)[0]
own = lab == lab[0]
a = d[own].sum() / (own.sum() - 1)
others = [d[lab == c].mean() for c in range(3) if c != lab[0]]
b = min(others)
print(round(a, 3), round(b, 3), round((b - a) / max(a, b), 3))
from sklearn.metrics import silhouette_samples, silhouette_score
sil = silhouette_samples(X, lab)
print(round(sil[0], 3))
print(round(silhouette_score(X, lab), 3))
per = pd.Series(sil).groupby(lab).agg(["mean", "min", "size"])
per.round(3)
fig, ax = plt.subplots(figsize=(7, 4))
y = 0
for c in range(3):
vals = np.sort(sil[lab == c])
ax.barh(range(y, y + len(vals)), vals, height=1,
label=f"Cluster {c}")
y += len(vals) + 5
ax.axvline(sil.mean(), color="black", linestyle="--")
ax.set_title("Silhouette plot, k-means with k = 3")
ax.set_xlabel("Silhouette value")
ax.set_ylabel("Wines, sorted within cluster")
ax.legend()
Your turn: find the wine with the lowest silhouette. Which cluster is it nearer to?
worst = sil.argmin()
dw = pairwise_distances(X[[worst]], X)[0]
means = pd.Series(dw).groupby(lab).mean().round(3)
print(worst, lab[worst], round(sil[worst], 3))
means
Common mistake: asking for a silhouette with only one cluster
one = KMeans(n_clusters=1, n_init=10, random_state=42).fit_predict(X)
silhouette_score(X, one)
sil_k = {}
for k in range(2, 9):
labels_k = KMeans(n_clusters=k, n_init=10,
random_state=42).fit_predict(X)
sil_k[k] = silhouette_score(X, labels_k)
sil_k = pd.Series(sil_k).round(3)
sil_k
sil_k.plot(marker="o")
plt.title("Average silhouette by k")
plt.xlabel("Number of clusters k")
plt.ylabel("Average silhouette")
plt.show()
Two more referees: Davies-Bouldin and Calinski-Harabasz#
Problem: one score can be fooled, so we ask for a second opinion.
- Davies-Bouldin: how much each cluster resembles its closest rival. Lower is better
- Calinski-Harabasz: spread between clusters over spread within. Higher is better
from sklearn.metrics import davies_bouldin_score
from sklearn.metrics import calinski_harabasz_score
print(round(davies_bouldin_score(X, lab), 3))
print(round(calinski_harabasz_score(X, lab), 1))
Your turn: build one table of all three scores for k = 2 to 6. Which k does each score prefer?
rows = []
for k in range(2, 7):
lk = KMeans(n_clusters=k, n_init=10,
random_state=42).fit_predict(X)
rows.append({"k": k, "silhouette": silhouette_score(X, lk),
"davies_bouldin": davies_bouldin_score(X, lk),
"calinski": calinski_harabasz_score(X, lk)})
scores = pd.DataFrame(rows).set_index("k").round(3)
scores
print("silhouette:", scores["silhouette"].idxmax())
print("davies_bouldin:", scores["davies_bouldin"].idxmin())
print("calinski:", scores["calinski"].idxmax())
Common mistake: scoring labels from a sample against the full table
idx = w.sample(100, random_state=42).index
km_part = KMeans(n_clusters=3, n_init=10, random_state=42)
part = km_part.fit_predict(X[idx])
davies_bouldin_score(X, part)
full = km_part.predict(X)
print(len(full), round(davies_bouldin_score(X, full), 3))
When labels exist: the adjusted Rand index#
Problem: the wines do have a cultivar column. Do clusters find it?
truth = w["cultivar"].str[-1].astype(int)
pd.crosstab(truth, lab, rownames=["cultivar"],
colnames=["cluster"])
Quick check: cluster numbers are arbitrary. Will plain accuracy be fair here?
from sklearn.metrics import accuracy_score, adjusted_rand_score
print(round(accuracy_score(truth, lab), 3))
print(round(adjusted_rand_score(truth, lab), 3))
renamed = (lab + 1) % 3
print(round(accuracy_score(truth, renamed), 3))
print(round(adjusted_rand_score(truth, renamed), 3))
mixed = pd.Series(lab).sample(frac=1, random_state=42)
shuffled = mixed.to_numpy()
print(round(adjusted_rand_score(truth, shuffled), 3))
Adjusted Rand: 1 is a perfect match, 0 is no better than random.
Interpreting clusters: from numbers to words#
Problem: a score says the groups are distinct, not what they are.
cols = ["alcohol", "flavanoids", "color_intensity", "hue",
"proline"]
w.groupby(lab)[cols].mean().round(2)
z = pd.DataFrame(X, columns=features).groupby(lab).mean()
import seaborn as sns
sns.heatmap(z.T.round(1), annot=True, cmap="RdBu_r", center=0)
plt.title("Cluster profiles in standard deviations")
plt.xlabel("Cluster")
plt.show()
w.assign(cluster=lab).boxplot(column="proline", by="cluster")
plt.suptitle("")
plt.title("Proline by cluster")
plt.xlabel("Cluster")
plt.ylabel("Proline")
plt.show()
for c in z.index:
top = z.loc[c].abs().nlargest(3).index
print(c, list(top))
Your turn: give each cluster a short plain-English name.
names = {0: "light, low alcohol", 1: "deep colour, low flavanoids",
2: "rich, high proline"}
pd.Series(lab).map(names).value_counts()
Quick check: which cultivar does each named cluster match?
named = pd.Series(lab, name="cluster").map(names)
pd.crosstab(named, w["cultivar"])
Mini-project: which clustering should we trust?#
DBSCAN, hierarchical or k-means: grade all three on the same wines.
from sklearn.cluster import AgglomerativeClustering
ward = AgglomerativeClustering(n_clusters=3).fit_predict(X)
methods = {"DBSCAN": db, "Hierarchical": ward, "k-means": lab}
DBSCAN labels noise as -1. Score only the clustered wines, and report the noise share.
rows = []
for name, l in methods.items():
keep = l != -1
rows.append({"method": name, "clusters": len(set(l[keep])),
"noise_%": round(100 * (~keep).mean(), 1),
"silhouette": silhouette_score(X[keep], l[keep]),
"ARI": adjusted_rand_score(truth[keep], l[keep])})
report = pd.DataFrame(rows).set_index("method").round(3)
report
report[["silhouette", "ARI"]].plot(kind="bar", rot=0)
plt.title("Grading three clusterings of the wines")
plt.xlabel("Method")
plt.ylabel("Score")
plt.show()
Finding: k-means with k = 3 is the one to trust: every wine clustered, ARI 0.897. DBSCAN's higher silhouette is bought by discarding 31% of the wines as noise.
Recap#
- Inertia measures tightness but always falls as k grows
- Silhouette compares distance inside (a) with the nearest neighbour (b)
- Davies-Bouldin: lower is better. Calinski-Harabasz: higher is better
- Adjusted Rand checks clusters against true labels, ignoring names
- A profile turns clusters into words; the scores only grade them
Challenges#
- Repeat the score table using only alcohol, flavanoids and proline. Does k = 3 still win?
- Score k-means on the unscaled wines. What happens to the adjusted Rand index?
- Try DBSCAN with eps from 2.0 to 3.0. Which setting gives the best silhouette?
What's next#
Lesson 32: Hands-on Clustering Project: Socio-Economic Segmentation Cluster the world's countries by wealth, health and progress.
Data Mining 32: Hands-on Clustering Project: Socio-Economic Segmentation#
In this lesson you'll be able to:
- Turn a long yearly table into one row of features per country
- Engineer wealth, health and progress features, and log skewed money
- Choose k with scores, then make the business call
- Profile, name and map the segments
- Follow segments over time to answer a real question Dataset: Gapminder, 142 countries every 5 years from 1952 to 2007 (data/pandas/gapminder.csv)
Warm-up: grading clusters, from lesson 31#
A first, naive clustering of the 2007 countries, and its silhouette.
import pandas as pd
import numpy as np
g = pd.read_csv("gapminder.csv")
g7 = g[g["year"] == 2007]
print(g.shape, g7.shape)
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
Xq = StandardScaler().fit_transform(g7[["lifeExp", "gdpPercap"]])
km = KMeans(n_clusters=3, n_init=10, random_state=42)
print(round(silhouette_score(Xq, km.fit_predict(Xq)), 3))
Frame the question and meet the data#
Problem: which countries share a development path, not just a snapshot?
g.head(3)
print(g["country"].nunique(), g["year"].nunique())
print(g["year"].min(), g["year"].max())
g7[["lifeExp", "gdpPercap", "pop"]].describe().round(1)
import matplotlib.pyplot as plt
g7["gdpPercap"].plot(kind="hist", bins=30)
plt.title("Income per person, 2007")
plt.xlabel("GDP per person (US dollars)")
plt.ylabel("Number of countries")
plt.show()
Common mistake: passing the whole table, text columns and all, to the scaler
StandardScaler().fit_transform(g7)
num = g7.select_dtypes("number").drop(columns="year")
print(num.columns.tolist())
Engineer features: wealth, health and progress#
Problem: the data is long, one row per country per year. We need one row per country.
- Wealth: log of GDP per person in 2007
- Health: life expectancy in 2007
- Progress: life years gained and GDP growth since 1952
Common mistake: pivoting on a column that is not unique
g.pivot(index="continent", columns="year", values="lifeExp")
life = g.pivot(index="country", columns="year", values="lifeExp")
life.iloc[:3, [0, -1]]
gdp = g.pivot(index="country", columns="year", values="gdpPercap")
print(life.shape, gdp.shape)
four = ["China", "India", "Nigeria", "United States"]
life.loc[four].T.plot(marker=".")
plt.title("Life expectancy, 1952 to 2007")
plt.xlabel("Year")
plt.ylabel("Life expectancy (years)")
plt.show()
feats = pd.DataFrame({
"life_2007": life[2007],
"log_gdp_2007": np.log10(gdp[2007]),
"life_gain": life[2007] - life[1952],
"gdp_growth": np.log10(gdp[2007] / gdp[1952]),
})
feats.describe().round(2)
Quick check: why use the log of GDP rather than GDP itself?
print(round(gdp[2007].skew(), 2))
print(round(feats["log_gdp_2007"].skew(), 2))
feats.nsmallest(4, "life_gain").round(2)
Your turn: which three countries grew their income the most since 1952?
fast = feats.nlargest(3, "gdp_growth")
fast["times_richer"] = 10 ** fast["gdp_growth"]
fast.round(2)
Choose k: scores advise, the business decides#
Problem: how many segments should we report?
X = StandardScaler().fit_transform(feats)
print(X.shape, X.mean().round(2), X.std().round(2))
from sklearn.metrics import davies_bouldin_score
rows = []
for k in range(2, 8):
lk = KMeans(n_clusters=k, n_init=10,
random_state=42).fit_predict(X)
rows.append({"k": k, "silhouette": silhouette_score(X, lk),
"davies_bouldin": davies_bouldin_score(X, lk),
"smallest": np.bincount(lk).min()})
scores = pd.DataFrame(rows).set_index("k").round(3)
scores
scores["silhouette"].plot(marker="o")
plt.title("Silhouette by number of segments")
plt.xlabel("Number of clusters k")
plt.ylabel("Average silhouette")
plt.show()
inertia = {}
for k in range(1, 8):
km = KMeans(n_clusters=k, n_init=10, random_state=42).fit(X)
inertia[k] = km.inertia_
pd.Series(inertia).plot(marker="o")
plt.title("Elbow chart for the countries")
plt.xlabel("Number of clusters k")
plt.ylabel("Inertia")
plt.show()
Quick check: the scores prefer k = 2. Should we report two segments?
l2 = KMeans(n_clusters=2, n_init=10, random_state=42).fit_predict(X)
feats.groupby(l2).median().round(2)
l3 = KMeans(n_clusters=3, n_init=10, random_state=42).fit_predict(X)
pd.crosstab(l2, l3, rownames=["k=2"], colnames=["k=3"])
Two clusters just says rich and poor. k = 3 costs a little silhouette but adds a story.
Fit, profile and name the segments#
Problem: turn cluster numbers into segments people can use.
km3 = KMeans(n_clusters=3, n_init=10, random_state=42).fit(X)
wealth = km3.cluster_centers_[:, feats.columns.get_loc("log_gdp_2007")]
seg = wealth.argsort().argsort()[km3.labels_]
np.bincount(seg)
prof = feats.groupby(seg).median()
prof["gdp_2007"] = 10 ** prof["log_gdp_2007"]
prof.round(2)
Quick check: why profile with the median rather than the mean?
raw = gdp[2007].groupby(seg)
pd.DataFrame({"mean": raw.mean(), "median": raw.median()}).round(0)
import seaborn as sns
centres = pd.DataFrame(km3.cluster_centers_, columns=feats.columns)
centres.index = wealth.argsort().argsort()
sns.heatmap(centres.sort_index().round(2), annot=True, cmap="RdBu_r",
center=0)
plt.title("Segment centres in standard deviations")
plt.ylabel("Segment")
plt.show()
Your turn: give each segment a short, honest name.
names = {0: "Struggling", 1: "Health risers", 2: "Prosperous"}
feats["segment"] = pd.Series(seg, index=feats.index).map(names)
feats["segment"].value_counts()
pop07 = g7.set_index("country")["pop"]
for name, grp in feats.groupby("segment"):
big = pop07[grp.index].nlargest(4).index
print(name, list(big))
cont = g7.set_index("country")["continent"]
pd.crosstab(cont, feats["segment"])
See the segments and check the borderline countries#
Problem: are the segments visible, and which countries sit on a fence?
for name, grp in feats.groupby("segment"):
plt.scatter(grp["log_gdp_2007"], grp["life_2007"], label=name,
alpha=0.7)
plt.title("Segments by wealth and health, 2007")
plt.xlabel("Log10 GDP per person")
plt.ylabel("Life expectancy (years)")
plt.legend()
plt.show()
for name, grp in feats.groupby("segment"):
plt.scatter(grp["gdp_growth"], grp["life_gain"], label=name,
alpha=0.7)
plt.title("Segments by progress since 1952")
plt.xlabel("Log10 GDP growth, 1952 to 2007")
plt.ylabel("Life years gained")
plt.legend()
plt.show()
from sklearn.metrics import silhouette_samples
feats["sil"] = silhouette_samples(X, seg)
feats.groupby("segment")["sil"].mean().round(3)
feats.nsmallest(5, "sil")[["segment", "life_2007", "sil"]].round(2)
Your turn: look up where five large countries fall.
picks = ["China", "India", "United States", "Nigeria", "Brazil"]
cols = ["segment", "life_2007", "life_gain", "sil"]
feats.loc[picks, cols].round(2)
Stability check: would a slightly different sample give the same segments?
from sklearn.metrics import adjusted_rand_score
part = feats.sample(frac=0.8, random_state=42).index
rows = feats.index.get_indexer(part)
km_s = KMeans(n_clusters=3, n_init=10, random_state=42).fit(X[rows])
again = km_s.predict(X)
print(len(rows), round(adjusted_rand_score(seg, again), 3))
Mini-project: did the health risers close the gap?#
Follow each 2007 segment back to 1952. How did the life expectancy gap change?
g["segment"] = g["country"].map(feats["segment"])
trend = g.groupby(["year", "segment"])["lifeExp"].median()
trend = trend.unstack()
trend.loc[[1952, 1982, 2007]].round(1)
trend.plot(marker="o")
plt.title("Median life expectancy by 2007 segment")
plt.xlabel("Year")
plt.ylabel("Life expectancy (years)")
plt.show()
gap = trend["Prosperous"] - trend["Health risers"]
print(round(gap[1952], 1), round(gap[2007], 1))
gap2 = trend["Prosperous"] - trend["Struggling"]
print(round(gap2[1952], 1), round(gap2[2007], 1))
top = feats.sort_values("life_gain", ascending=False)
top.groupby("segment").head(1)[["segment", "life_gain"]].round(2)
import os
out = feats[["segment", "life_2007", "life_gain", "sil"]].round(3)
out.to_csv("m32_segments.csv")
back = pd.read_csv("m32_segments.csv", index_col=0)
print(back.shape)
back["segment"].value_counts()
Finding: the health risers cut their gap to the prosperous from 21.1 to 6.1 years. The struggling segment fell further behind: 25.8 years then, 26.9 now.
Recap#
- Pivot long yearly data into one row per country
- Log skewed money values before scaling and clustering
- Scores advise on k; usefulness to people decides
- Relabel clusters by a feature, then profile and name them
- Low-silhouette rows are the borderline cases to discuss
Challenges#
- Add log population as a fifth feature. Do the segments change?
- Build the features for 1982 instead of 2007. Who were the health risers then?
- Try agglomerative clustering with 3 clusters. Use the adjusted Rand index to compare.
What's next#
Lesson 33: Association Rule Mining with Apriori Algorithm From grouping rows to finding items that are bought together.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.



