Mathew K Analytics

Lesson 4 · Data Mining Full Course

Data Mining Full Course Part 4 (4 Hours): Random Forest, XGBoost & Clustering in Python

All 10 lesson notebooks from Part 4 of the Data Mining Full Course in Python, in teaching order. Each lesson starts with a heading; the timestamps below…

📓 Full notebook

Download .ipynb

Data Mining Full Course, Part 4: Ensembles, Random Forest, XGBoost and Clustering#

All 10 lesson notebooks from Part 4 of the Data Mining Full Course in Python, in teaching order. Each lesson starts with a heading; the timestamps below jump to it in the video.

  • 0:00 Lesson 23: Introduction to Ensemble Learning
  • 0:23:30 Lesson 24: Bagging and Random Forests
  • 0:47:10 Lesson 25: Boosting Techniques- AdaBoost and XGBoost
  • 1:10:51 Lesson 26: Hands-on Project- Predicting Diabetes with Ensemble Models
  • 1:35:13 Lesson 27: Introduction to Clustering
  • 1:58:45 Lesson 28: k-Means Clustering Algorithm
  • 2:23:20 Lesson 29: Hierarchical Clustering Techniques
  • 2:46:39 Lesson 30: Density-Based Clustering with DBSCAN
  • 3:11:20 Lesson 31: Cluster Evaluation Metrics and Interpretation
  • 3:35:59 Lesson 32: Hands-on Clustering Project- Socio-Economic Segmentation

Data files: save the datasets from the lesson page in the same folder as this notebook. Files the lessons create are written to that folder too.

Data Mining 23: Introduction to Ensemble Learning#

In this lesson you'll be able to:

  • Explain why a vote of several models can beat any one of them
  • Build hard and soft voting ensembles in scikit-learn
  • Recognise bagging, boosting and stacking, and try each one
  • Judge ensembles fairly with cross-validation Dataset: the Titanic passenger list, 891 people (data/pandas/titanic.csv)

Warm-up: set the baseline, as in lesson 22#

Lesson 22 compared classifiers against a do-nothing baseline. Same habit, new data.

import pandas as pd
df = pd.read_csv("titanic.csv")
print(df.shape)
print(round(df["Survived"].mean(), 3))
(891, 12)
0.384
cols = ["Pclass", "Sex", "Age", "SibSp", "Parch", "Fare", "Embarked"]
X = df[cols].copy()
X["Sex"] = X["Sex"].eq("male").astype(int)
X = pd.get_dummies(X, columns=["Embarked"], drop_first=True,
                   dtype=int)
y = df["Survived"]
X.head(3)
Pclass Sex Age SibSp Parch Fare Embarked_Q Embarked_S
0 3 1 22.0 1 0 7.2500 0 1
1 1 0 38.0 1 0 71.2833 0 0
2 3 0 26.0 0 0 7.9250 0 1
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42)
age_fill = X_train["Age"].median()
X_train = X_train.fillna({"Age": age_fill})
X_test = X_test.fillna({"Age": age_fill})
print(age_fill, len(X_train), len(X_test))
29.0 668 223
from sklearn.dummy import DummyClassifier
base = DummyClassifier(strategy="most_frequent")
base.fit(X_train, y_train)
print(round(base.score(X_test, y_test), 3))
0.614

Why a crowd of models can beat one#

Problem: one model's mistakes are hard to fix; several can outvote them.

  • Three models, each right 70% of the time, mistakes independent
  • The vote is right whenever at least 2 of the 3 are right
from scipy.stats import binom
print(round(binom.sf(1, 3, 0.7), 3))
print(round(binom.sf(5, 11, 0.7), 3))
0.784
0.922

Real models are not independent, so the gain is smaller. Let's measure it.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
models = {
    "Logistic": make_pipeline(StandardScaler(),
                              LogisticRegression()),
    "kNN": make_pipeline(StandardScaler(),
                         KNeighborsClassifier(n_neighbors=7)),
    "Tree": DecisionTreeClassifier(max_depth=4, random_state=42),
}
preds = {}
for name, model in models.items():
    model.fit(X_train, y_train)
    preds[name] = model.predict(X_test)
    print(f"{name:9} {model.score(X_test, y_test):.3f}")
Logistic  0.776
kNN       0.789
Tree      0.776

Quick check: do the three models get the same passengers wrong?

votes = pd.DataFrame(preds, index=X_test.index)
wrong = votes.ne(y_test, axis=0)
print(wrong.sum().to_dict())
print(wrong.any(axis=1).sum(), wrong.all(axis=1).sum())
{'Logistic': 50, 'kNN': 47, 'Tree': 50}
72 27
majority = (votes.sum(axis=1) >= 2).astype(int)
vote_acc = (majority == y_test).mean()
print(round(vote_acc, 3))
0.785
import matplotlib.pyplot as plt
acc = votes.eq(y_test, axis=0).mean()
acc["Majority vote"] = vote_acc
ax = acc.plot(kind="bar", rot=0, figsize=(7, 4), color="tab:blue")
ax.set_ylim(0.7, 0.82)
ax.set_title("Test accuracy: three models and their vote")
ax.set_xlabel("Model")
ax.set_ylabel("Accuracy on 223 test passengers")
plt.show()
No description has been provided for this image

Let scikit-learn hold the vote#

Problem: a hand-made vote can't be saved, tuned or cross-validated.

  • Hard voting: each model casts one vote; the majority wins
  • Soft voting: average the predicted probabilities, then decide

Common mistake: handing VotingClassifier bare models without names

from sklearn.ensemble import VotingClassifier
VotingClassifier(list(models.values())).fit(X_train, y_train)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[13], line 2
      1 from sklearn.ensemble import VotingClassifier
----> 2 VotingClassifier(list(models.values())).fit(X_train, y_train)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py:1403, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1393     estimator._validate_params()
   1395 with (
   1396     config_context(
   1397         skip_parameter_validation=(
   (...)
   1401     callback_management_context(estimator),
   1402 ):
-> 1403     return fit_method(estimator, *args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\ensemble\_voting.py:409, in VotingClassifier.fit(self, X, y, **fit_params)
    406 self.classes_ = self.le_.classes_
    407 transformed_y = self.le_.transform(y)
--> 409 return super().fit(X, transformed_y, **fit_params)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\ensemble\_voting.py:81, in _BaseVoting.fit(self, X, y, **fit_params)
     78 @abstractmethod
     79 def fit(self, X, y, **fit_params):
     80     """Get common fit operations."""
---> 81     names, clfs = self._validate_estimators()
     83     if self.weights is not None and len(self.weights) != len(self.estimators):
     84         raise ValueError(
     85             "Number of `estimators` and weights must be equal; got"
     86             f" {len(self.weights)} weights, {len(self.estimators)} estimators"
     87         )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\ensemble\_base.py:224, in _BaseHeterogeneousEnsemble._validate_estimators(self)
    219 def _validate_estimators(self):
    220     if len(self.estimators) == 0 or not all(
    221         isinstance(item, (tuple, list)) and isinstance(item[0], str)
    222         for item in self.estimators
    223     ):
--> 224         raise ValueError(
    225             "Invalid 'estimators' attribute, 'estimators' should be a "
    226             "non-empty list of (string, estimator) tuples."
    227         )
    228     names, estimators = zip(*self.estimators)
    229     # defined by MetaEstimatorMixin

ValueError: Invalid 'estimators' attribute, 'estimators' should be a non-empty list of (string, estimator) tuples.
members = list(models.items())
hard = VotingClassifier(members, voting="hard")
hard.fit(X_train, y_train)
print(round(hard.score(X_test, y_test), 3))
0.785
soft = VotingClassifier(members, voting="soft")
soft.fit(X_train, y_train)
print(round(soft.score(X_test, y_test), 3))
0.776
one = X_test.head(1)
for name, member in soft.named_estimators_.items():
    p = member.predict_proba(one)[0, 1]
    print(f"{name:9} {p:.2f}")
print(f"{'Average':9} {soft.predict_proba(one)[0, 1]:.2f}")
Logistic  0.08
kNN       0.14
Tree      0.10
Average   0.11

Your turn#

Build a hard vote of three identical depth-4 trees. Does it beat one tree?

clones = [(f"tree{i}", DecisionTreeClassifier(max_depth=4,
                                              random_state=42))
          for i in range(3)]
same = VotingClassifier(clones, voting="hard").fit(X_train, y_train)
print(round(same.score(X_test, y_test), 3))
0.776

Bagging: many trees on resampled rows#

Problem: a deep tree changes a lot when the data changes a little.

  • Draw a bootstrap sample: rows picked at random, with replacement
  • Train one tree per sample, then let all the trees vote

Quick check: a bootstrap sample of 668 rows. How many different passengers?

boot = X_train.sample(frac=1, replace=True, random_state=42)
print(len(boot), boot.index.nunique())
print(round(boot.index.nunique() / len(boot), 3))
668 409
0.612
deep = DecisionTreeClassifier(random_state=42)
deep.fit(X_train, y_train)
print(round(deep.score(X_train, y_train), 3))
print(round(deep.score(X_test, y_test), 3))
0.978
0.74
for seed in range(5):
    rows = X_train.sample(frac=1, replace=True, random_state=seed)
    t = DecisionTreeClassifier(random_state=42)
    t.fit(rows, y_train.loc[rows.index])
    print(seed, round(t.score(X_test, y_test), 3))
0 0.753
1 0.78
2 0.731
3 0.7
4 0.758
from sklearn.ensemble import BaggingClassifier
bag = BaggingClassifier(DecisionTreeClassifier(random_state=42),
                        n_estimators=100, random_state=42)
bag.fit(X_train, y_train)
print(round(bag.score(X_test, y_test), 3))
0.744

Boosting: each model fixes the last one's mistakes#

Problem: simple models miss the hard cases; boosting focuses on them.

  • AdaBoost: give extra weight to the rows the last model missed
  • Gradient boosting: each new tree predicts the errors left over
from sklearn.ensemble import AdaBoostClassifier
from sklearn.ensemble import GradientBoostingClassifier
ada = AdaBoostClassifier(n_estimators=100, random_state=42)
ada.fit(X_train, y_train)
print(round(ada.score(X_test, y_test), 3))
0.789
errors = pd.Series(ada.estimator_errors_)
print(errors.head(4).round(3).to_list())
print(round(errors.mean(), 3))
[0.207, 0.327, 0.449, 0.428]
0.475
gb = GradientBoostingClassifier(random_state=42)
gb.fit(X_train, y_train)
print(round(gb.score(X_test, y_test), 3))
0.803
steps = [(p == y_test).mean() for p in gb.staged_predict(X_test)]
ax = pd.Series(steps, index=range(1, 101)).plot(figsize=(7, 4))
ax.set_title("Gradient boosting: accuracy as trees are added")
ax.set_xlabel("Number of trees")
ax.set_ylabel("Test accuracy")
plt.show()
print(round(steps[0], 3), round(max(steps), 3))
No description has been provided for this image
0.614 0.816

Your turn#

Train gradient boosting with only 10 trees (n_estimators=10). Better or worse?

gb10 = GradientBoostingClassifier(n_estimators=10, random_state=42)
gb10.fit(X_train, y_train)
print(round(gb10.score(X_test, y_test), 3))
0.785

Stacking: a model that learns whom to trust#

Problem: equal votes ignore that some members are more reliable.

from sklearn.ensemble import StackingClassifier
stack = StackingClassifier(members,
                           final_estimator=LogisticRegression(),
                           cv=5)
stack.fit(X_train, y_train)
print(round(stack.score(X_test, y_test), 3))
0.771

Quick check: which member will the top model trust most?

trust = stack.final_estimator_.coef_[0]
pd.Series(trust, index=list(models)).round(2)
Logistic    1.64
kNN         1.80
Tree        1.79
dtype: float64

Judge ensembles fairly with cross-validation#

Problem: on 223 test passengers, one person moves accuracy by almost half a point.

one_person = 1 / len(y_test)
print(round(one_person * 100, 2))
0.45
  • Cross-validation splits the training rows into 5 folds
  • Train on 4, test on the 5th, rotate, then average the 5 scores

Common mistake: cross-validating the raw table that still has gaps

from sklearn.model_selection import cross_val_score
cross_val_score(soft, X, y, cv=5)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[30], line 2
      1 from sklearn.model_selection import cross_val_score
----> 2 cross_val_score(soft, X, y, cv=5)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_param_validation.py:218, in validate_params.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    212 try:
    213     with config_context(
    214         skip_parameter_validation=(
    215             prefer_skip_nested_validation or global_skip_validation
    216         )
    217     ):
--> 218         return func(*args, **kwargs)
    219 except InvalidParameterError as e:
    220     # When the function is just a wrapper around an estimator, we allow
    221     # the function to delegate validation to the estimator, but we replace
    222     # the name of the estimator by the name of the function in the error
    223     # message to avoid confusion.
    224     msg = re.sub(
    225         r"parameter of \w+ must be",
    226         f"parameter of {func.__qualname__} must be",
    227         str(e),
    228     )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\model_selection\_validation.py:654, in cross_val_score(estimator, X, y, groups, scoring, cv, n_jobs, verbose, params, pre_dispatch, error_score)
    651 # To ensure multimetric format is not supported
    652 scorer = check_scoring(estimator, scoring=scoring)
--> 654 cv_results = cross_validate(
    655     estimator=estimator,
    656     X=X,
    657     y=y,
    658     groups=groups,
    659     scoring={"score": scorer},
    660     cv=cv,
    661     n_jobs=n_jobs,
    662     verbose=verbose,
    663     params=params,
    664     pre_dispatch=pre_dispatch,
    665     error_score=error_score,
    666 )
    667 return cv_results["test_score"]

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_param_validation.py:218, in validate_params.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    212 try:
    213     with config_context(
    214         skip_parameter_validation=(
    215             prefer_skip_nested_validation or global_skip_validation
    216         )
    217     ):
--> 218         return func(*args, **kwargs)
    219 except InvalidParameterError as e:
    220     # When the function is just a wrapper around an estimator, we allow
    221     # the function to delegate validation to the estimator, but we replace
    222     # the name of the estimator by the name of the function in the error
    223     # message to avoid confusion.
    224     msg = re.sub(
    225         r"parameter of \w+ must be",
    226         f"parameter of {func.__qualname__} must be",
    227         str(e),
    228     )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\model_selection\_validation.py:396, in cross_validate(estimator, X, y, groups, scoring, cv, n_jobs, verbose, params, pre_dispatch, return_train_score, return_estimator, return_indices, error_score)
    375 parallel = Parallel(n_jobs=n_jobs, verbose=verbose, pre_dispatch=pre_dispatch)
    376 results = parallel(
    377     delayed(_fit_and_score)(
    378         clone(estimator),
   (...)
    393     for train, test in indices
    394 )
--> 396 _warn_or_raise_about_fit_failures(results, error_score)
    398 # For callable scoring, the return type is only know after calling. If the
    399 # return type is a dictionary, the error scores can now be inserted with
    400 # the correct key.
    401 if callable(scoring):

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\model_selection\_validation.py:482, in _warn_or_raise_about_fit_failures(results, error_score)
    475 if num_failed_fits == num_fits:
    476     all_fits_failed_message = (
    477         f"\nAll the {num_fits} fits failed.\n"
    478         "It is very likely that your model is misconfigured.\n"
    479         "You can try to debug the error by setting error_score='raise'.\n\n"
    480         f"Below are more details about the failures:\n{fit_errors_summary}"
    481     )
--> 482     raise ValueError(all_fits_failed_message)
    484 else:
    485     some_fits_failed_message = (
    486         f"\n{num_failed_fits} fits failed out of a total of {num_fits}.\n"
    487         "The score on these train-test partitions for these parameters"
   (...)
    491         f"Below are more details about the failures:\n{fit_errors_summary}"
    492     )

ValueError: 
All the 5 fits failed.
It is very likely that your model is misconfigured.
You can try to debug the error by setting error_score='raise'.

Below are more details about the failures:
--------------------------------------------------------------------------------
5 fits failed with the following error:
Traceback (most recent call last):
  File "C:\Users\makmw\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\model_selection\_validation.py", line 860, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "C:\Users\makmw\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py", line 1403, in wrapper
    return fit_method(estimator, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\makmw\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\ensemble\_voting.py", line 409, in fit
    return super().fit(X, transformed_y, **fit_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\makmw\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\ensemble\_voting.py", line 99, in fit
    self.estimators_ = Parallel(n_jobs=self.n_jobs)(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\makmw\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\parallel.py", line 91, in __call__
    return super().__call__(iterable_with_config_and_warning_filters)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\makmw\AppData\Local\Programs\Python\Python312\Lib\site-packages\joblib\parallel.py", line 2012, in __call__
    return output if self.return_generator else list(output)
                                                ^^^^^^^^^^^^
  File "C:\Users\makmw\AppData\Local\Programs\Python\Python312\Lib\site-packages\joblib\parallel.py", line 1940, in _get_sequential_output
    res = func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\makmw\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\parallel.py", line 184, in __call__
    return self.function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\makmw\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\ensemble\_base.py", line 45, in _fit_single_estimator
    estimator.fit(X, y, **fit_params)
  File "C:\Users\makmw\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py", line 1403, in wrapper
    return fit_method(estimator, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\makmw\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\pipeline.py", line 645, in fit
    self._final_estimator.fit(Xt, y, **last_step_params["fit"])
  File "C:\Users\makmw\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py", line 1403, in wrapper
    return fit_method(estimator, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\makmw\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\linear_model\_logistic.py", line 1437, in fit
    X, y = validate_data(
           ^^^^^^^^^^^^^^
  File "C:\Users\makmw\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py", line 3058, in validate_data
    X, y = check_X_y(X, y, **check_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\makmw\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py", line 1330, in check_X_y
    X = check_array(
        ^^^^^^^^^^^^
  File "C:\Users\makmw\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py", line 1090, in check_array
    _assert_all_finite(
  File "C:\Users\makmw\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py", line 137, in _assert_all_finite
    _assert_all_finite_element_wise(
  File "C:\Users\makmw\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py", line 186, in _assert_all_finite_element_wise
    raise ValueError(msg_err)
ValueError: Input X contains NaN.
LogisticRegression does not accept missing values encoded as NaN natively. For supervised learning, you might want to consider sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept missing values encoded as NaNs natively. Alternatively, it is possible to preprocess the data, for instance by using an imputer transformer in a pipeline or drop samples with missing values. See https://scikit-learn.org/stable/modules/impute.html You can find a list of all estimators that handle NaN values at the following page: https://scikit-learn.org/stable/modules/impute.html#estimators-that-handle-nan-values
cv = cross_val_score(soft, X_train, y_train, cv=5)
print(cv.round(3))
print(round(cv.mean(), 3), round(cv.std(), 3))
[0.813 0.776 0.866 0.82  0.842]
0.823 0.03

Quick check: soft voting lost to hard voting on the test set. Does it lose across 5 folds too?

cv_hard = cross_val_score(hard, X_train, y_train, cv=5)
print(round(cv_hard.mean(), 3), round(cv_hard.std(), 3))
0.817 0.024

Mini-project: which approach should we trust?#

Rank three single models and five ensembles by 5-fold cross-validation.

contenders = dict(models)
contenders["Voting (soft)"] = soft
contenders["Bagging"] = bag
contenders["AdaBoost"] = ada
contenders["Gradient boosting"] = gb
contenders["Stacking"] = stack
print(len(contenders))
8
rows = {}
for name, model in contenders.items():
    s = cross_val_score(model, X_train, y_train, cv=5)
    rows[name] = {"mean": s.mean(), "std": s.std()}
results = pd.DataFrame(rows).T.sort_values("mean")
results.round(3)
mean std
Tree 0.801 0.029
Logistic 0.804 0.032
kNN 0.808 0.021
Bagging 0.810 0.026
Stacking 0.820 0.032
AdaBoost 0.820 0.031
Voting (soft) 0.823 0.030
Gradient boosting 0.829 0.022
ax = results["mean"].plot(kind="barh", xerr=results["std"],
                          figsize=(7, 5), color="tab:green")
ax.set_xlim(0.7, 0.87)
ax.set_title("5-fold cross-validated accuracy on Titanic")
ax.set_xlabel("Mean accuracy (bars show one standard deviation)")
ax.set_ylabel("Model")
plt.show()
No description has been provided for this image
singles = results.loc[list(models), "mean"]
ensembles = results.drop(index=list(models))["mean"]
print(f"Best single: {singles.idxmax()} {singles.max():.3f}")
print(f"Best ensemble: {ensembles.idxmax()} {ensembles.max():.3f}")
print((ensembles > singles.max()).sum(), "of", len(ensembles))
Best single: kNN 0.808
Best ensemble: Gradient boosting 0.829
5 of 5
print(round(gb.score(X_test, y_test), 3))
0.803

Recap#

  • Ensembles combine several models so their mistakes can cancel out
  • Voting only helps when members make different mistakes
  • Bagging averages many trees on bootstrap samples
  • Boosting adds models one by one to fix earlier errors
  • Stacking learns how much to trust each member
  • Compare with cross-validation; a small test set is noisy

Challenges#

  1. Add GaussianNB as a fourth voting member. Does the CV score rise?
  2. Give soft voting weights=[1, 2, 1]. What changes?
  3. Use 10-fold cross-validation for the top three. Does the ranking hold?

What's next#

Lesson 24: Bagging and Random Forests Go deeper into bagging and meet the random forest.

Data Mining 24: Bagging and Random Forests#

In this lesson you'll be able to:

  • Explain why averaging many bootstrap trees steadies a jumpy tree
  • Build bagged trees and random forests in scikit-learn
  • Use the out-of-bag score to tune a forest without the test set
  • Read feature importances, and check them with permutation importance Dataset: California housing, 1990 census districts (data/pandas/housing.csv)

Warm-up: from lesson 23 to house prices#

Lesson 23 met bagging as one of three ensemble families. Today it gets a whole lesson.

import pandas as pd
housing = pd.read_csv("housing.csv")
print(housing.shape)
housing["median_house_value"].describe().round(0)
(20640, 10)
count     20640.0
mean     206856.0
std      115396.0
min       14999.0
25%      119600.0
50%      179700.0
75%      264725.0
max      500001.0
Name: median_house_value, dtype: float64
df = housing.sample(6000, random_state=42)
capped = (df["median_house_value"] >= 500001).sum()
print(len(df), capped)
6000 268
y = df["median_house_value"]
X = pd.get_dummies(df.drop(columns="median_house_value"),
                   columns=["ocean_proximity"], dtype=int)
print(X.shape)
X.columns[-5:].tolist()
(6000, 13)
['ocean_proximity_<1H OCEAN',
 'ocean_proximity_INLAND',
 'ocean_proximity_ISLAND',
 'ocean_proximity_NEAR BAY',
 'ocean_proximity_NEAR OCEAN']
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42)
fill = X_train["total_bedrooms"].median()
X_train = X_train.fillna({"total_bedrooms": fill})
X_test = X_test.fillna({"total_bedrooms": fill})
print(len(X_train), len(X_test))
4500 1500

Why one deep tree is not enough#

Problem: a deep tree memorises its rows and changes with every sample.

  • R squared: share of the price variation explained (1 is perfect)
  • MAE: the average miss, in dollars
from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import mean_absolute_error
tree = DecisionTreeRegressor(random_state=42)
tree.fit(X_train, y_train)
print(round(tree.score(X_train, y_train), 3))
print(round(tree.score(X_test, y_test), 3))
print(round(mean_absolute_error(y_test, tree.predict(X_test))))
1.0
0.597
48761
print(tree.get_depth(), tree.get_n_leaves())
28 4343

Quick check: three trees, three bootstrap samples. Will they agree on one district?

district = X_test.head(1)
for seed in range(3):
    rows = X_train.sample(frac=1, replace=True, random_state=seed)
    t = DecisionTreeRegressor(random_state=42)
    t.fit(rows, y_train.loc[rows.index])
    print(seed, round(t.predict(district)[0]))
print("actual", y_test.iloc[0])
0 225300
1 227700
2 210300
actual 213700.0

Bagging: average many bootstrap trees#

Problem: each tree is noisy, but their noise points in different directions.

  • Grow each tree on its own bootstrap sample of the rows
  • Average their predictions (or take a vote, for classes)

Common mistake: copying old code that says base_estimator

from sklearn.ensemble import BaggingRegressor
BaggingRegressor(base_estimator=DecisionTreeRegressor())
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[9], line 2
      1 from sklearn.ensemble import BaggingRegressor
----> 2 BaggingRegressor(base_estimator=DecisionTreeRegressor())

TypeError: BaggingRegressor.__init__() got an unexpected keyword argument 'base_estimator'
bag = BaggingRegressor(estimator=DecisionTreeRegressor(),
                       n_estimators=50, random_state=42)
bag.fit(X_train, y_train)
print(round(bag.score(X_test, y_test), 3))
print(round(mean_absolute_error(y_test, bag.predict(X_test))))
0.787
35472
import numpy as np
X_test_np = X_test.to_numpy()
per_tree = np.array([t.predict(X_test_np) for t in bag.estimators_])
counts = np.arange(1, len(per_tree) + 1)[:, None]
running = per_tree.cumsum(axis=0) / counts
maes = [mean_absolute_error(y_test, p) for p in running]
print(round(maes[0]), round(maes[9]), round(maes[-1]))
51037 37728 35472
import matplotlib.pyplot as plt
ax = pd.Series(maes, index=counts.ravel()).plot(figsize=(7, 4))
ax.set_title("Bagging: the average miss falls as trees are added")
ax.set_xlabel("Number of trees averaged")
ax.set_ylabel("Mean absolute error (dollars)")
plt.show()
No description has been provided for this image

Your turn#

Bag 50 trees of max_depth=3. Does bagging rescue a shallow tree?

short = DecisionTreeRegressor(max_depth=3, random_state=42)
short.fit(X_train, y_train)
print(round(mean_absolute_error(y_test, short.predict(X_test))))
bag3 = BaggingRegressor(estimator=short, n_estimators=50,
                        random_state=42).fit(X_train, y_train)
print(round(mean_absolute_error(y_test, bag3.predict(X_test))))
55064
53512

Random forests: bagging plus random feature choices#

Problem: bagged trees still look alike, because they all split on the same strong column.

  • At each split, a tree may only look at a random subset of columns
  • max_features sets the subset size, e.g. 0.5 = half the columns
  • For regression, scikit-learn's default (1.0) is plain bagging
from sklearn.ensemble import RandomForestRegressor
forest = RandomForestRegressor(n_estimators=100, max_features=0.5,
                               random_state=42, n_jobs=-1)
forest.fit(X_train, y_train)
print(round(forest.score(X_test, y_test), 3))
print(round(mean_absolute_error(y_test, forest.predict(X_test))))
0.791
35551

Quick check: do forest trees agree with each other more or less than bagged trees?

def agreement(model):
    first20 = model.estimators_[:20]
    preds = np.array([t.predict(X_test_np) for t in first20])
    return np.corrcoef(preds).mean()
print(round(agreement(bag), 3), round(agreement(forest), 3))
0.76 0.73
models = {"One tree": tree, "Bagging": bag, "Forest": forest}
mae = pd.Series({name: mean_absolute_error(y_test, m.predict(X_test))
                 for name, m in models.items()})
ax = mae.plot(kind="bar", rot=0, figsize=(7, 4), color="tab:green")
ax.set_title("Average miss on 1,500 test districts")
ax.set_xlabel("Model")
ax.set_ylabel("Mean absolute error (dollars)")
plt.show()
No description has been provided for this image

The out-of-bag score: a free validation check#

Problem: we want to tune the forest without spending the test set.

  • Each bootstrap sample leaves out about a third of the rows
  • Those left-out rows grade the tree: that's the out-of-bag score
used = np.unique(bag.estimators_samples_[0]).size
print(used, round(1 - used / len(X_train), 3))
2834 0.37

Common mistake: asking for oob_score_ without switching it on

forest.oob_score_
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[18], line 1
----> 1 forest.oob_score_

AttributeError: 'RandomForestRegressor' object has no attribute 'oob_score_'
forest = RandomForestRegressor(n_estimators=100, max_features=0.5,
                               oob_score=True, random_state=42,
                               n_jobs=-1)
forest.fit(X_train, y_train)
print(round(forest.oob_score_, 3))
print(round(forest.score(X_test, y_test), 3))
0.771
0.791

Tune the forest without touching the test set#

Problem: pick max_features using out-of-bag scores only.

oob = {}
for share in [0.2, 0.33, 0.5, 1.0]:
    rf = RandomForestRegressor(n_estimators=100, max_features=share,
                               oob_score=True, random_state=42,
                               n_jobs=-1).fit(X_train, y_train)
    oob[share] = rf.oob_score_
pd.Series(oob).round(3)
0.20    0.754
0.33    0.769
0.50    0.771
1.00    0.767
dtype: float64
ax = pd.Series(oob).plot(kind="bar", rot=0, figsize=(7, 4))
ax.set_ylim(0.74, 0.78)
ax.set_title("Out-of-bag R squared by max_features")
ax.set_xlabel("Share of columns tried at each split")
ax.set_ylabel("Out-of-bag R squared")
plt.show()
No description has been provided for this image

How many trees? Grow one forest step by step with warm_start.

grow = RandomForestRegressor(max_features=0.5, oob_score=True,
                             warm_start=True, random_state=42,
                             n_jobs=-1)
by_size = {}
for n in [25, 50, 100, 200]:
    grow.set_params(n_estimators=n).fit(X_train, y_train)
    by_size[n] = grow.oob_score_
pd.Series(by_size).round(3)
25     0.748
50     0.765
100    0.771
200    0.774
dtype: float64

Your turn#

Keep max_features=0.5 and set min_samples_leaf=5. Does the OOB score rise or fall?

leafy = RandomForestRegressor(n_estimators=100, max_features=0.5,
                              min_samples_leaf=5, oob_score=True,
                              random_state=42, n_jobs=-1)
leafy.fit(X_train, y_train)
print(round(leafy.oob_score_, 3))
0.767

Read what the forest learned#

Problem: a valuation tool needs reasons, not just numbers.

Quick check: which column do you think matters most to district values?

imp = pd.Series(forest.feature_importances_, index=X.columns)
imp.nlargest(6).round(3)
median_income             0.432
ocean_proximity_INLAND    0.138
latitude                  0.104
longitude                 0.103
housing_median_age        0.049
population                0.045
dtype: float64

Built-in importances favour columns with many distinct values. Check them.

from sklearn.inspection import permutation_importance
perm = permutation_importance(forest, X_test, y_test, n_repeats=5,
                              random_state=42, n_jobs=-1)
drops = pd.Series(perm.importances_mean, index=X.columns)
drops.nlargest(6).round(3)
median_income             0.619
longitude                 0.236
latitude                  0.197
ocean_proximity_INLAND    0.163
population                0.054
housing_median_age        0.035
dtype: float64

Quick check: which three columns could we drop with almost no loss?

drops.nsmallest(3).round(4)
ocean_proximity_ISLAND        0.0000
ocean_proximity_NEAR BAY      0.0024
ocean_proximity_NEAR OCEAN    0.0089
dtype: float64
ax = drops.nlargest(8).sort_values().plot(kind="barh", figsize=(7, 4))
ax.set_title("What the forest relies on (permutation importance)")
ax.set_xlabel("Drop in test R squared when shuffled")
ax.set_ylabel("Feature")
plt.show()
No description has been provided for this image

Mini-project: could this forest value a district?#

How often is the forest within 20% of the true value, and where does it struggle?

check = df.loc[X_test.index, ["ocean_proximity"]].copy()
check["actual"] = y_test
check["predicted"] = forest.predict(X_test)
gap = (check["predicted"] - check["actual"]).abs()
check["pct_error"] = gap / check["actual"]
check.head().round(2)
ocean_proximity actual predicted pct_error
7120 <1H OCEAN 213700.0 203100.00 0.05
4625 <1H OCEAN 404500.0 294902.08 0.27
13017 INLAND 131700.0 147807.00 0.12
16493 INLAND 152200.0 157282.00 0.03
14695 NEAR OCEAN 133200.0 186054.01 0.40
within = (check["pct_error"] <= 0.2).mean()
median_err = check["pct_error"].median()
print(round(within, 3), round(median_err, 3))
0.657 0.134
by_area = check.groupby("ocean_proximity")["pct_error"]
by_area.agg(["count", "median"]).round(3)
count median
ocean_proximity
<1H OCEAN 641 0.108
INLAND 464 0.150
NEAR BAY 183 0.173
NEAR OCEAN 212 0.142
ax = check.plot.scatter(x="actual", y="predicted", s=8, alpha=0.4,
                        figsize=(6, 6))
ax.plot([0, 500001], [0, 500001], "k--")
ax.set_title("Forest estimates against real district values")
ax.set_xlabel("Real median value (dollars)")
ax.set_ylabel("Predicted value (dollars)")
plt.show()
No description has been provided for this image
top = check["actual"] >= 500001
print(top.sum(), round(check.loc[top, "predicted"].mean()))
68 408237
new = X_test.iloc[[4]]
trees = [t.predict(new.to_numpy())[0] for t in forest.estimators_]
low, high = np.percentile(trees, [10, 90])
print(round(low), round(high), y_test.iloc[4])
73500 305960 133200.0
print(f"{within:.0%} of districts valued within 20%; "
      f"typical miss {median_err:.0%}")
66% of districts valued within 20%; typical miss 13%

Recap#

  • One deep tree memorises its data and changes with every sample
  • Bagging averages trees grown on bootstrap samples, cutting the error
  • Random forests also limit the columns per split (max_features), so trees differ more
  • The out-of-bag score tunes the forest without touching the test set
  • Check built-in importances with permutation importance

Challenges#

  1. Try max_features='sqrt' and compare its OOB score with the table.
  2. Drop latitude and longitude. How much does the OOB score fall?
  3. Fit a RandomForestClassifier on the Titanic data from lesson 23.

What's next#

Lesson 25: Boosting Techniques- AdaBoost and XGBoost Trees that learn one after another, each fixing earlier mistakes.

Data Mining 25: Boosting Techniques- AdaBoost and XGBoost#

In this lesson you'll be able to:

  • Explain how boosting learns from the mistakes of earlier models
  • Train and inspect an AdaBoost model built from tree stumps
  • Use gradient boosting and see what the learning rate does
  • Train XGBoost with early stopping on a validation set
  • Compare boosted models fairly and read them in plain terms Dataset: Wisconsin breast cancer biopsies (data/mining/breast_cancer.csv)

Warm-up: a random forest as our baseline#

Lesson 24 grew many trees side by side and let them vote.

import pandas as pd
df = pd.read_csv("breast_cancer.csv")
print(df.shape)
df["diagnosis"].value_counts()
(569, 31)
diagnosis
benign       357
malignant    212
Name: count, dtype: int64
from sklearn.model_selection import train_test_split
X = df.drop(columns="diagnosis")
y = df["diagnosis"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42)
print(X_train.shape, X_test.shape)
(426, 30) (143, 30)
from sklearn.ensemble import RandomForestClassifier
forest = RandomForestClassifier(n_estimators=200, random_state=42)
forest.fit(X_train, y_train)
print(round(forest.score(X_test, y_test), 3))
0.958

Boosting in plain words: learn from mistakes#

Problem: a forest's trees never learn from each other.

  • Bagging: many strong trees, trained side by side, then a vote
  • Boosting: many weak models, trained one after another
  • Each new model focuses on the cases the chain still gets wrong
from sklearn.tree import DecisionTreeClassifier
stump = DecisionTreeClassifier(max_depth=1, random_state=42)
stump.fit(X_train, y_train)
print(round(stump.score(X_test, y_test), 3))
0.888

Quick check: which single measurement would you split on?

feature = X.columns[stump.tree_.feature[0]]
cut = stump.tree_.threshold[0]
print(feature, round(cut, 1))
worst perimeter 112.8
wrong = stump.predict(X_train) != y_train
print(wrong.sum())
y_train[wrong].value_counts()
30
diagnosis
malignant    20
benign       10
Name: count, dtype: int64

AdaBoost: give the hard cases more weight#

Problem: turn many weak stumps into one strong classifier.

  • Start with every training row weighted equally
  • Fit a stump, then raise the weight of the rows it got wrong
  • The next stump must try harder on those heavy rows
  • Final answer: a weighted vote, better stumps get a bigger say
from sklearn.ensemble import AdaBoostClassifier
ada = AdaBoostClassifier(n_estimators=50, random_state=42)
ada.fit(X_train, y_train)
print(round(ada.score(X_test, y_test), 3))
0.972
firsts = ada.estimators_[:5]
names = [X.columns[s.tree_.feature[0]] for s in firsts]
print(names)
print(ada.estimator_weights_[:5].round(2))
['worst perimeter', 'mean concave points', 'worst texture', 'worst concave points', 'area error']
[2.58 1.72 1.59 1.06 0.88]

Common mistake: copying algorithm="SAMME.R" from an old tutorial

AdaBoostClassifier(algorithm="SAMME.R", random_state=42)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[10], line 1
----> 1 AdaBoostClassifier(algorithm="SAMME.R", random_state=42)

TypeError: AdaBoostClassifier.__init__() got an unexpected keyword argument 'algorithm'
ada = AdaBoostClassifier(n_estimators=50, random_state=42)
print("algorithm" in ada.get_params())
False

How many rounds? Ask a validation set#

Never tune on the test set: carve a validation set from training.

X_fit, X_val, y_fit, y_val = train_test_split(
    X_train, y_train, test_size=0.25, stratify=y_train,
    random_state=42)
print(X_fit.shape, X_val.shape)
(319, 30) (107, 30)
ada_big = AdaBoostClassifier(n_estimators=300, random_state=42)
A_fit, A_val = X_fit.to_numpy(), X_val.to_numpy()
ada_big.fit(A_fit, y_fit)
fit_acc = list(ada_big.staged_score(A_fit, y_fit))
val_acc = list(ada_big.staged_score(A_val, y_val))
best = max(val_acc)
print(val_acc.index(best) + 1, round(best, 3))
70 0.972
import matplotlib.pyplot as plt
rounds = range(1, len(val_acc) + 1)
plt.plot(rounds, fit_acc, label="fitting rows")
plt.plot(rounds, val_acc, label="validation rows")
plt.title("AdaBoost accuracy as stumps are added")
plt.xlabel("Number of boosting rounds")
plt.ylabel("Accuracy")
plt.legend()
plt.show()
No description has been provided for this image

Your turn#

Train AdaBoost with learning_rate=0.5 on the fitting rows and score it on validation.

slow = AdaBoostClassifier(n_estimators=100, learning_rate=0.5,
                          random_state=42)
slow.fit(X_fit, y_fit)
print(round(slow.score(X_val, y_val), 3))
0.972

Gradient boosting: fit what is still wrong#

Problem: correct the chain's errors directly, not via weights.

  • Start from a simple guess: the overall malignant rate
  • Each small tree predicts the gap between guess and truth
  • Add a small step of that correction: the learning rate
from sklearn.ensemble import GradientBoostingClassifier
gb = GradientBoostingClassifier(n_estimators=100,
                                learning_rate=0.1, max_depth=3,
                                random_state=42)
gb.fit(X_fit, y_fit)
print(round(gb.score(X_val, y_val), 3))
0.953
patient = X_val.iloc[[0]]
print(y_val.iloc[0])
steps = list(gb.staged_predict_proba(patient))
for r in [1, 5, 20, 100]:
    print(r, round(steps[r - 1][0, 1], 3))
benign
1 0.338
5 0.228
20 0.056
100 0.0

Quick check: will a learning rate of 1.0 beat 0.1 on validation?

for lr in [0.01, 0.1, 1.0]:
    m = GradientBoostingClassifier(learning_rate=lr,
                                   random_state=42)
    m.fit(X_fit, y_fit)
    tr = round(m.score(X_fit, y_fit), 3)
    va = round(m.score(X_val, y_val), 3)
    print(lr, tr, va)
0.01 0.991 0.907
0.1 1.0 0.953
1.0 1.0 0.925

Watch out: deep trees do not make a better booster

deep = GradientBoostingClassifier(max_depth=None, random_state=42)
deep.fit(X_fit, y_fit)
print(len(deep.estimators_))
print(round(deep.score(X_val, y_val), 3))
100
0.897

XGBoost: gradient boosting with extra brakes#

Problem: boost fast, avoid overfitting, and stop at the right time.

  • Same chain of small trees correcting each other
  • Penalties on complex trees: regularisation
  • Early stopping: halt when the validation score stops improving

Common mistake: giving XGBoost text labels

from xgboost import XGBClassifier
xgb = XGBClassifier(random_state=42)
xgb.fit(X_train, y_train)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[20], line 3
      1 from xgboost import XGBClassifier
      2 xgb = XGBClassifier(random_state=42)
----> 3 xgb.fit(X_train, y_train)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\xgboost\core.py:569, in require_keyword_args.<locals>.throw_if.<locals>.inner_f(*args, **kwargs)
    567 for k, arg in zip(sig.parameters, args):
    568     kwargs[k] = arg
--> 569 return func(**kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\xgboost\sklearn.py:1812, in XGBClassifier.fit(self, X, y, sample_weight, base_margin, eval_set, verbose, xgb_model, sample_weight_eval_set, base_margin_eval_set, feature_weights)
   1807     expected_classes = self.classes_
   1808 if (
   1809     classes.shape != expected_classes.shape
   1810     or not (classes == expected_classes).all()
   1811 ):
-> 1812     raise ValueError(
   1813         f"Invalid classes inferred from unique values of `y`.  "
   1814         f"Expected: {expected_classes}, got {classes}"
   1815     )
   1817 params = self.get_xgb_params()
   1819 if callable(self.objective):

ValueError: Invalid classes inferred from unique values of `y`.  Expected: [0 1], got ['benign' 'malignant']
yn_train = (y_train == "malignant").astype(int)
yn_test = (y_test == "malignant").astype(int)
yn_fit = (y_fit == "malignant").astype(int)
yn_val = (y_val == "malignant").astype(int)
print(yn_train.value_counts())
diagnosis
0    267
1    159
Name: count, dtype: int64
xgb = XGBClassifier(n_estimators=200, learning_rate=0.1,
                    max_depth=3, random_state=42)
xgb.fit(X_fit, yn_fit)
print(round(xgb.score(X_val, yn_val), 3))
0.944
xgb_es = XGBClassifier(n_estimators=500, learning_rate=0.1,
                       max_depth=3, early_stopping_rounds=20,
                       eval_metric="logloss", random_state=42)
xgb_es.fit(X_fit, yn_fit, eval_set=[(X_val, yn_val)],
           verbose=False)
print(xgb_es.best_iteration)
print(round(xgb_es.score(X_val, yn_val), 3))
47
0.925

Your turn#

Repeat early stopping with max_depth=1 (stumps). Does it need more rounds?

stumps = XGBClassifier(n_estimators=500, learning_rate=0.1,
                       max_depth=1, early_stopping_rounds=20,
                       eval_metric="logloss", random_state=42)
stumps.fit(X_fit, yn_fit, eval_set=[(X_val, yn_val)],
           verbose=False)
print(stumps.best_iteration)
347

Compare the models fairly, then test once#

Problem: one split is noisy. Cross-validate, then test once.

from sklearn.model_selection import cross_val_score
models = {
    "forest": RandomForestClassifier(random_state=42),
    "adaboost": AdaBoostClassifier(random_state=42),
    "grad boost": GradientBoostingClassifier(random_state=42),
    "xgboost": XGBClassifier(n_estimators=xgb_es.best_iteration + 1,
                             max_depth=3, random_state=42),
}
for name, model in models.items():
    s = cross_val_score(model, X_train, yn_train, cv=5)
    print(f"{name:11} {s.mean():.3f} +/- {s.std():.3f}")
forest      0.965 +/- 0.015
adaboost    0.955 +/- 0.014
grad boost  0.960 +/- 0.019
xgboost     0.960 +/- 0.018
from sklearn.metrics import recall_score
rows = []
for name, model in models.items():
    pred = model.fit(X_train, yn_train).predict(X_test)
    missed = int(((pred == 0) & (yn_test == 1)).sum())
    rows.append({"model": name,
                 "accuracy": (pred == yn_test).mean(),
                 "recall": recall_score(yn_test, pred),
                 "missed": missed})
results = pd.DataFrame(rows).round(3)
results
model accuracy recall missed
0 forest 0.965 0.906 5
1 adaboost 0.972 0.925 4
2 grad boost 0.965 0.906 5
3 xgboost 0.979 0.943 3

Quick check: accuracies are close. Which column matters most to a doctor?

from sklearn.metrics import ConfusionMatrixDisplay
xgb_final = models["xgboost"]
ConfusionMatrixDisplay.from_predictions(
    yn_test, xgb_final.predict(X_test),
    display_labels=["benign", "malignant"])
plt.title("XGBoost on the test tumours")
plt.show()
No description has been provided for this image
imp = pd.Series(xgb_final.feature_importances_, index=X.columns)
top = imp.nlargest(10).sort_values()
ax = top.plot.barh(figsize=(7, 4))
ax.set_title("XGBoost: top 10 measurements")
ax.set_xlabel("Importance (share of total)")
ax.set_ylabel("Measurement")
plt.show()
No description has been provided for this image

Mini-project: can 5 measurements do the work of 30?#

Fewer measurements means a cheaper, simpler test for a clinic.

top5 = imp.nlargest(5).index.to_list()
print(top5)
['worst perimeter', 'mean concave points', 'worst concave points', 'worst concavity', 'worst radius']
small = XGBClassifier(n_estimators=xgb_es.best_iteration + 1,
                      max_depth=3, random_state=42)
s5 = cross_val_score(small, X_train[top5], yn_train, cv=5)
s30 = cross_val_score(models["xgboost"], X_train, yn_train, cv=5)
print(round(s5.mean(), 3), round(s30.mean(), 3))
0.932 0.96
small.fit(X_train[top5], yn_train)
pred5 = small.predict(X_test[top5])
acc5 = (pred5 == yn_test).mean()
missed5 = int(((pred5 == 0) & (yn_test == 1)).sum())
print(round(acc5, 3), missed5)
0.951 6
full = results.set_index("model").loc["xgboost"]
print(f"30 measurements: {full.accuracy:.1%}, missed {full.missed:.0f}")
print(f" 5 measurements: {acc5:.1%}, missed {missed5}")
30 measurements: 97.9%, missed 3
 5 measurements: 95.1%, missed 6

Recap#

  • Boosting trains weak models in a chain, each fixing earlier mistakes
  • AdaBoost reweights hard rows; better stumps get a bigger vote
  • Gradient boosting fits the remaining error, one small step at a time
  • XGBoost needs 0/1 labels and can stop early on a validation set
  • Tune on validation or cross-validation; test once at the end

Challenges#

  1. Plot XGBoost's validation log loss per round (evals_result()).
  2. Cross-validate AdaBoost with 50, 200 and 500 rounds. Which is best?
  3. Find the 5-measurement model's missed tumours and inspect their values.

What's next#

Lesson 26: Hands-on Project- Predicting Diabetes with Ensemble Models Clean real medical data, then compare forests, boosting and voting.

Data Mining 26: Hands-on Project- Predicting Diabetes with Ensemble Models#

In this lesson you'll be able to:

  • Find missing values hiding as zeros in medical data
  • Fill gaps inside a pipeline without leaking test information
  • Compare simple models and ensembles with cross-validation
  • Combine models with soft voting and stacking
  • Test a final model once and tune it for screening Dataset: Pima Indians diabetes study, 768 women (data/mining/pima_diabetes.csv)

Warm-up: boosting on the raw data#

Lesson 25 boosted tree stumps with AdaBoost. Same tool, new data.

import numpy as np
import pandas as pd
df = pd.read_csv("pima_diabetes.csv")
print(df.shape)
df.head(3)
(768, 9)
pregnancies glucose blood_pressure skin_thickness insulin bmi diabetes_pedigree age outcome
0 6 148 72 35 0 33.6 0.627 50 1
1 1 85 66 29 0 26.6 0.351 31 0
2 8 183 64 0 0 23.3 0.672 32 1
from sklearn.ensemble import AdaBoostClassifier
from sklearn.model_selection import cross_val_score
X_raw = df.drop(columns="outcome")
ada = AdaBoostClassifier(random_state=42)
s = cross_val_score(ada, X_raw, df["outcome"], cv=5)
print(round(s.mean(), 3))
0.767

Step 1: find the missing values hiding as zeros#

Problem: some zeros are not measurements, they are blanks.

print(df.isna().sum().sum())
(df == 0).sum()
0
pregnancies          111
glucose                5
blood_pressure        35
skin_thickness       227
insulin              374
bmi                   11
diabetes_pedigree      0
age                    0
outcome              500
dtype: int64

Quick check: which of these columns can honestly be zero?

zero_cols = ["glucose", "blood_pressure", "skin_thickness",
             "insulin", "bmi"]
(df[zero_cols] == 0).mean().round(3)
glucose           0.007
blood_pressure    0.046
skin_thickness    0.296
insulin           0.487
bmi               0.014
dtype: float64

Common mistake: forcing gaps back into whole numbers

df["glucose"].replace(0, np.nan).astype(int)
---------------------------------------------------------------------------
IntCastingNaNError                        Traceback (most recent call last)
Cell In[6], line 1
----> 1 df["glucose"].replace(0, np.nan).astype(int)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\generic.py:6529, in NDFrame.astype(self, dtype, copy, errors)
   6525     results = [ser.astype(dtype, errors=errors) for _, ser in self.items()]
   6527 else:
   6528     # else, only a single dtype is given
-> 6529     new_data = self._mgr.astype(dtype=dtype, errors=errors)
   6530     res = self._constructor_from_mgr(new_data, axes=new_data.axes)
   6531     return res.__finalize__(self, method="astype")

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\managers.py:627, in BaseBlockManager.astype(self, dtype, errors)
    626 def astype(self, dtype, errors: str = "raise") -> Self:
--> 627     return self.apply("astype", dtype=dtype, errors=errors)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\managers.py:445, in BaseBlockManager.apply(self, f, align_keys, **kwargs)
    443         applied = b.apply(f, **kwargs)
    444     else:
--> 445         applied = getattr(b, f)(**kwargs)
    446     result_blocks = extend_blocks(applied, result_blocks)
    448 out = type(self).from_blocks(result_blocks, [ax.view() for ax in self.axes])

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\blocks.py:607, in Block.astype(self, dtype, errors, squeeze)
    604         raise ValueError("Can not squeeze with more than one column.")
    605     values = values[0, :]  # type: ignore[call-overload]
--> 607 new_values = astype_array_safe(values, dtype, errors=errors)
    609 new_values = maybe_coerce_values(new_values)
    611 refs = None

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\dtypes\astype.py:240, in astype_array_safe(values, dtype, copy, errors)
    237     dtype = dtype.numpy_dtype
    239 try:
--> 240     new_values = astype_array(values, dtype, copy=copy)
    241 except (ValueError, TypeError):
    242     # e.g. _astype_nansafe can fail on object-dtype of strings
    243     #  trying to convert to float
    244     if errors == "ignore":

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\dtypes\astype.py:185, in astype_array(values, dtype, copy)
    182     values = values.astype(dtype, copy=copy)
    184 else:
--> 185     values = _astype_nansafe(values, dtype, copy=copy)
    187 # in pandas we don't store numpy str dtypes, so convert to object
    188 if isinstance(dtype, np.dtype) and issubclass(values.dtype.type, str):

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\dtypes\astype.py:102, in _astype_nansafe(arr, dtype, copy, skipna)
     97     return lib.ensure_string_array(
     98         arr, skipna=skipna, convert_na_value=False
     99     ).reshape(shape)
    101 elif np.issubdtype(arr.dtype, np.floating) and dtype.kind in "iu":
--> 102     return _astype_float_to_int_nansafe(arr, dtype, copy)
    104 elif arr.dtype == object:
    105     # if we have a datetime/timedelta array of objects
    106     # then coerce to datetime64[ns] and use DatetimeArray.astype
    108     if lib.is_np_dtype(dtype, "M"):

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\dtypes\astype.py:146, in _astype_float_to_int_nansafe(values, dtype, copy)
    142 """
    143 astype with a check preventing converting NaN to a meaningless integer value.
    144 """
    145 if not np.isfinite(values).all():
--> 146     raise IntCastingNaNError(
    147         "Cannot convert non-finite values (NA or inf) to integer."
    148         "Replace or remove non-finite values or cast to an integer type"
    149         "that supports these values (e.g. 'Int64')"
    150     )
    151 if dtype.kind == "u":
    152     # GH#45151
    153     if not (values >= 0).all():

IntCastingNaNError: Cannot convert non-finite values (NA or inf) to integer.Replace or remove non-finite values or cast to an integer typethat supports these values (e.g. 'Int64')
clean = df.copy()
clean[zero_cols] = clean[zero_cols].replace(0, np.nan)
clean[zero_cols].isna().sum()
glucose             5
blood_pressure     35
skin_thickness    227
insulin           374
bmi                11
dtype: int64
import matplotlib.pyplot as plt
import seaborn as sns
ax = sns.histplot(data=clean, x="glucose", hue="outcome",
                  bins=30, element="step")
ax.set_title("Glucose by diabetes outcome")
ax.set_xlabel("Plasma glucose (mg/dL)")
ax.set_ylabel("Number of women")
plt.show()
No description has been provided for this image
clean.groupby("outcome")[["glucose", "bmi", "age"]].median()
glucose bmi age
outcome
0 107.0 30.1 27.0
1 140.0 34.3 36.0

Step 2: split first, then fill the gaps#

Problem: filling with medians from all rows leaks the test set.

y = clean["outcome"]
y.value_counts(normalize=True).round(3)
outcome
0    0.651
1    0.349
Name: proportion, dtype: float64
from sklearn.model_selection import train_test_split
X = clean.drop(columns="outcome")
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42)
print(X_train.shape, X_test.shape)
(576, 8) (192, 8)
from sklearn.impute import SimpleImputer
imputer = SimpleImputer(strategy="median")
imputer.fit(X_train)
learned = pd.Series(imputer.statistics_, index=X.columns)
learned.round(1)
pregnancies            3.0
glucose              117.0
blood_pressure        72.0
skin_thickness        29.0
insulin              125.0
bmi                   32.4
diabetes_pedigree      0.4
age                   30.0
dtype: float64

Quick check: do the test rows have the same medians as the training rows?

gap = pd.DataFrame({"train": X_train[zero_cols].median(),
                    "test": X_test[zero_cols].median()})
gap
train test
glucose 117.0 120.0
blood_pressure 72.0 72.0
skin_thickness 29.0 29.0
insulin 125.0 130.0
bmi 32.4 31.6
from sklearn.pipeline import make_pipeline
from sklearn.tree import DecisionTreeClassifier
tree = make_pipeline(SimpleImputer(strategy="median"),
                     DecisionTreeClassifier(random_state=42))
tree
Pipeline(steps=[('simpleimputer', SimpleImputer(strategy='median')),
                ('decisiontreeclassifier',
                 DecisionTreeClassifier(random_state=42))])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Step 3: set the bar with simple models#

Problem: an ensemble is only worth it if it beats simpler models.

  • Accuracy: share of women labelled correctly
  • Recall: share of diabetic women the model finds
  • ROC AUC: how well it ranks patients by risk (0.5 = coin flip)
from sklearn.model_selection import StratifiedKFold, cross_validate
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
metrics = ["accuracy", "recall", "roc_auc"]
scores, models = {}, {}
def evaluate(name, model):
    r = cross_validate(model, X_train, y_train, cv=cv,
                       scoring=metrics)
    scores[name] = {m: r["test_" + m].mean() for m in metrics}
    models[name] = model
    return pd.Series(scores[name]).round(3)
from sklearn.dummy import DummyClassifier
dummy = make_pipeline(SimpleImputer(strategy="median"),
                      DummyClassifier())
evaluate("dummy", dummy)
accuracy    0.651
recall      0.000
roc_auc     0.500
dtype: float64
evaluate("tree", tree)
accuracy    0.710
recall      0.582
roc_auc     0.680
dtype: float64
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
logreg = make_pipeline(SimpleImputer(strategy="median"),
                       StandardScaler(), LogisticRegression())
evaluate("logreg", logreg)
accuracy    0.778
recall      0.583
roc_auc     0.831
dtype: float64

Your turn#

Evaluate a tree limited to max_depth=4. Does it beat the full tree?

tree4 = make_pipeline(SimpleImputer(strategy="median"),
                      DecisionTreeClassifier(max_depth=4,
                                             random_state=42))
evaluate("tree depth 4", tree4)
accuracy    0.717
recall      0.652
roc_auc     0.758
dtype: float64

Step 4: ensembles head to head#

Problem: which ensemble, if any, beats logistic regression?

from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(n_estimators=100, min_samples_leaf=3,
                            random_state=42)
forest = make_pipeline(SimpleImputer(strategy="median"), rf)
evaluate("forest", forest)
accuracy    0.760
recall      0.552
roc_auc     0.827
dtype: float64
adaboost = make_pipeline(SimpleImputer(strategy="median"),
                         AdaBoostClassifier(random_state=42))
evaluate("adaboost", adaboost)
accuracy    0.759
recall      0.557
roc_auc     0.817
dtype: float64
from xgboost import XGBClassifier
xgb = XGBClassifier(n_estimators=200, learning_rate=0.05,
                    max_depth=3, n_jobs=1, random_state=42)
boost = make_pipeline(SimpleImputer(strategy="median"), xgb)
evaluate("xgboost", boost)
accuracy    0.762
recall      0.587
roc_auc     0.824
dtype: float64

Common mistake: naming a metric after its function

cross_validate(tree, X_train, y_train, cv=cv,
               scoring="recall_score")
---------------------------------------------------------------------------
InvalidParameterError                     Traceback (most recent call last)
Cell In[23], line 1
----> 1 cross_validate(tree, X_train, y_train, cv=cv,
      2                scoring="recall_score")

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_param_validation.py:208, in validate_params.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    205 to_ignore += ["self", "cls"]
    206 params = {k: v for k, v in params.arguments.items() if k not in to_ignore}
--> 208 validate_parameter_constraints(
    209     parameter_constraints, params, caller_name=func.__qualname__
    210 )
    212 try:
    213     with config_context(
    214         skip_parameter_validation=(
    215             prefer_skip_nested_validation or global_skip_validation
    216         )
    217     ):

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_param_validation.py:98, in validate_parameter_constraints(parameter_constraints, params, caller_name)
     92 else:
     93     constraints_str = (
     94         f"{', '.join([str(c) for c in constraints[:-1]])} or"
     95         f" {constraints[-1]}"
     96     )
---> 98 raise InvalidParameterError(
     99     f"The {param_name!r} parameter of {caller_name} must be"
    100     f" {constraints_str}. Got {param_val!r} instead."
    101 )

InvalidParameterError: The 'scoring' parameter of cross_validate must be a str among {'precision_micro', 'precision', 'precision_samples', 'f1', 'neg_mean_absolute_percentage_error', 'recall_samples', 'd2_log_loss_score', 'neg_negative_likelihood_ratio', 'neg_log_loss', 'neg_root_mean_squared_log_error', 'neg_brier_score', 'neg_mean_absolute_error', 'normalized_mutual_info_score', 'recall_weighted', 'neg_root_mean_squared_error', 'precision_macro', 'recall_macro', 'matthews_corrcoef', 'adjusted_rand_score', 'r2', 'f1_weighted', 'top_k_accuracy', 'completeness_score', 'roc_auc_ovo_weighted', 'rand_score', 'neg_mean_poisson_deviance', 'recall_micro', 'jaccard_macro', 'neg_max_error', 'roc_auc_ovr', 'f1_macro', 'jaccard_weighted', 'adjusted_mutual_info_score', 'jaccard', 'explained_variance', 'neg_mean_squared_error', 'v_measure_score', 'neg_mean_gamma_deviance', 'f1_samples', 'homogeneity_score', 'd2_brier_score', 'jaccard_micro', 'mutual_info_score', 'roc_auc_ovo', 'accuracy', 'roc_auc_ovr_weighted', 'jaccard_samples', 'precision_weighted', 'balanced_accuracy', 'fowlkes_mallows_score', 'roc_auc', 'd2_absolute_error_score', 'average_precision', 'recall', 'f1_micro', 'positive_likelihood_ratio', 'neg_median_absolute_error', 'neg_mean_squared_log_error'}, a callable, an instance of 'list', an instance of 'tuple', an instance of 'dict' or None. Got 'recall_score' instead.
r = cross_validate(tree, X_train, y_train, cv=cv,
                   scoring="recall")
print(r["test_score"].round(3))
[0.561 0.525 0.65  0.55  0.625]
board = pd.DataFrame(scores).T
board.sort_values("roc_auc", ascending=False).round(3)
accuracy recall roc_auc
logreg 0.778 0.583 0.831
forest 0.760 0.552 0.827
xgboost 0.762 0.587 0.824
adaboost 0.759 0.557 0.817
tree depth 4 0.717 0.652 0.758
tree 0.710 0.582 0.680
dummy 0.651 0.000 0.500

Quick check: which models find the most diabetic women?

board["recall"].sort_values(ascending=False).head(3).round(3)
tree depth 4    0.652
xgboost         0.587
logreg          0.583
Name: recall, dtype: float64

Step 5: combine models with voting and stacking#

Problem: different models make different mistakes. Pool them.

  • Soft voting: average each model's probability of diabetes
  • Stacking: a small referee model learns how to weigh them
from sklearn.model_selection import cross_val_predict
members = {"logreg": logreg, "forest": forest, "xgboost": boost}
probs = pd.DataFrame({
    name: cross_val_predict(m, X_train, y_train, cv=cv,
                            method="predict_proba")[:, 1]
    for name, m in members.items()})
probs.corr().round(2)
logreg forest xgboost
logreg 1.00 0.91 0.89
forest 0.91 1.00 0.95
xgboost 0.89 0.95 1.00
from sklearn.ensemble import VotingClassifier
team = [("logreg", logreg), ("forest", forest), ("xgboost", boost)]
vote = VotingClassifier(team, voting="soft")
evaluate("soft vote", vote)
accuracy    0.762
recall      0.562
roc_auc     0.835
dtype: float64
from sklearn.ensemble import StackingClassifier
stack = StackingClassifier(team, cv=3,
                           final_estimator=LogisticRegression())
evaluate("stacking", stack)
accuracy    0.760
recall      0.548
roc_auc     0.835
dtype: float64

Common mistake: asking a hard-voting model for probabilities

hard = VotingClassifier(team, voting="hard")
hard.fit(X_train, y_train)
hard.predict_proba(X_train.head(3))
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_available_if.py:32, in _AvailableIfDescriptor._check(self, obj, owner)
     31 try:
---> 32     check_result = self.check(obj)
     33 except Exception as e:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\ensemble\_voting.py:446, in VotingClassifier._check_voting(self)
    445 if self.voting == "hard":
--> 446     raise AttributeError(
    447         f"predict_proba is not available when voting={self.voting!r}"
    448     )
    449 return True

AttributeError: predict_proba is not available when voting='hard'

The above exception was the direct cause of the following exception:

AttributeError                            Traceback (most recent call last)
Cell In[30], line 3
      1 hard = VotingClassifier(team, voting="hard")
      2 hard.fit(X_train, y_train)
----> 3 hard.predict_proba(X_train.head(3))

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_available_if.py:43, in _AvailableIfDescriptor.__get__(self, obj, owner)
     39 def __get__(self, obj, owner=None):
     40     if obj is not None:
     41         # delegate only on instances, not the classes.
     42         # this is to allow access to the docstrings.
---> 43         self._check(obj, owner=owner)
     44         out = MethodType(self.fn, obj)
     46     else:
     47         # This makes it possible to use the decorated method as an unbound method,
     48         # for instance when monkeypatching.

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_available_if.py:34, in _AvailableIfDescriptor._check(self, obj, owner)
     32     check_result = self.check(obj)
     33 except Exception as e:
---> 34     raise AttributeError(attr_err_msg) from e
     36 if not check_result:
     37     raise AttributeError(attr_err_msg)

AttributeError: This 'VotingClassifier' has no attribute 'predict_proba'
vote.fit(X_train, y_train)
vote.predict_proba(X_train.head(3)).round(3)
array([[0.853, 0.147],
       [0.812, 0.188],
       [0.826, 0.174]])
board = pd.DataFrame(scores).T
board = board.sort_values("roc_auc")
ax = board["roc_auc"].plot.barh(figsize=(7, 4), xlim=(0.5, 0.9))
ax.set_title("Cross-validated ROC AUC on the training set")
ax.set_xlabel("ROC AUC (0.5 = coin flip)")
ax.set_ylabel("Model")
plt.show()
No description has been provided for this image

Final check: screen the test patients#

Pick the winner by cross-validation, then test it exactly once.

top = board.sort_values("roc_auc", ascending=False)
print(top["roc_auc"].head(3).round(3))
stacking     0.835
soft vote    0.835
logreg       0.831
Name: roc_auc, dtype: float64
best = "soft vote"
final = models[best].fit(X_train, y_train)
print(final.named_estimators_.keys())
dict_keys(['logreg', 'forest', 'xgboost'])
from sklearn.metrics import recall_score, roc_auc_score
proba = final.predict_proba(X_test)[:, 1]
pred = (proba >= 0.5).astype(int)
print(round((pred == y_test).mean(), 3))
print(round(recall_score(y_test, pred), 3))
print(round(roc_auc_score(y_test, proba), 3))
0.734
0.537
0.826
oof = cross_val_predict(models[best], X_train, y_train, cv=cv,
                        method="predict_proba")[:, 1]
for cut in [0.5, 0.4, 0.3, 0.2]:
    flag = oof >= cut
    found = flag[y_train == 1].mean()
    sent = flag.mean()
    print(cut, round(found, 3), round(sent, 3))
0.5 0.562 0.281
0.4 0.701 0.377
0.3 0.801 0.46
0.2 0.871 0.58
screen = (proba >= 0.3).astype(int)
print(round(recall_score(y_test, screen), 3))
print(round(screen.mean(), 3))
0.791
0.464
from sklearn.metrics import ConfusionMatrixDisplay
ConfusionMatrixDisplay.from_predictions(
    y_test, screen, display_labels=["no diabetes", "diabetes"])
plt.title("Screening the test patients at a 0.3 cut-off")
plt.show()
No description has been provided for this image
from sklearn.inspection import permutation_importance
pi = permutation_importance(final, X_test, y_test, n_repeats=5,
                            scoring="roc_auc", random_state=42)
drops = pd.Series(pi.importances_mean, index=X.columns)
drops.sort_values(ascending=False).round(3)
glucose              0.132
bmi                  0.052
age                  0.011
pregnancies          0.009
diabetes_pedigree    0.004
insulin             -0.000
skin_thickness      -0.001
blood_pressure      -0.002
dtype: float64
new = X_test.head(3)
risk = final.predict_proba(new)[:, 1].round(3)
print(pd.DataFrame({"risk": risk, "flag": risk >= 0.3,
                    "actual": y_test.head(3).to_numpy()}))
    risk   flag  actual
0  0.391   True       1
1  0.345   True       0
2  0.065  False       0

Recap#

  • Zeros in medical columns were missing values in disguise
  • Imputing inside a pipeline keeps the test set out of training
  • Judge every model with the same cross-validation folds
  • Ensembles help a little here; a simple model came close
  • Pick the cut-off on training data to match the clinic's goal

Challenges#

  1. Add class_weight="balanced" to the forest. What happens to recall?
  2. Add the depth-4 tree to the voting team. Does ROC AUC improve?
  3. Find the cut-off that flags 90% of diabetic women in training.

What's next#

Lesson 27: Introduction to Clustering No labels at all: let the data reveal its own groups.

Data Mining 27: Introduction to Clustering#

In this lesson you'll be able to:

  • Explain what clustering is and when you'd use it
  • Measure similarity as distance, and scale features fairly
  • Group data with k-means and describe each cluster
  • Compare choices of k and three clustering families
  • Check clusters against known groups Dataset: Palmer penguins, 344 birds from Antarctica (data/pandas/penguins.csv)

Warm-up: classification needs labels#

Lesson 26 predicted a known label with ensembles. Here's that idea once more.

import pandas as pd
df = pd.read_csv("penguins.csv")
features = ["bill_length_mm", "bill_depth_mm",
            "flipper_length_mm", "body_mass_g"]
peng = df.dropna(subset=features).reset_index(drop=True)
print(len(df), len(peng))
344 342
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
forest = RandomForestClassifier(random_state=42)
s = cross_val_score(forest, peng[features], peng["species"], cv=5)
print(round(s.mean(), 3))
0.982

Clustering: finding groups without labels#

Problem: no answer column, but we still want meaningful groups.

  • Classification: learn from labelled examples, predict the label
  • Clustering: no labels; put similar rows in the same group
  • Uses: customer segments, patient types, grouping documents
X = peng[features]
X.describe().round(1)
bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
count 342.0 342.0 342.0 342.0
mean 43.9 17.2 200.9 4201.8
std 5.5 2.0 14.1 802.0
min 32.1 13.1 172.0 2700.0
25% 39.2 15.6 190.0 3550.0
50% 44.4 17.3 197.0 4050.0
75% 48.5 18.7 213.0 4750.0
max 59.6 21.5 231.0 6300.0
import matplotlib.pyplot as plt
plt.scatter(X["bill_length_mm"], X["flipper_length_mm"],
            color="grey", alpha=0.6)
plt.title("Penguins with no labels")
plt.xlabel("Bill length (mm)")
plt.ylabel("Flipper length (mm)")
plt.show()
No description has been provided for this image

Quick check: how many groups can you see in that cloud?

for name, grp in peng.groupby("species"):
    plt.scatter(grp["bill_length_mm"], grp["flipper_length_mm"],
                label=name, alpha=0.6)
plt.title("The same penguins, coloured by species")
plt.xlabel("Bill length (mm)")
plt.ylabel("Flipper length (mm)")
plt.legend()
plt.show()
No description has been provided for this image

Similarity is distance#

Problem: a computer needs a number for 'these two are alike'.

trio = peng.loc[[0, 1, 200], ["species"] + features]
trio
species bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
0 Adelie 39.1 18.7 181.0 3750.0
1 Adelie 39.5 17.4 186.0 3800.0
200 Chinstrap 49.8 17.3 198.0 3675.0

Common mistake: measuring distance with a text column included

from sklearn.metrics.pairwise import euclidean_distances
euclidean_distances(trio)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[8], line 2
      1 from sklearn.metrics.pairwise import euclidean_distances
----> 2 euclidean_distances(trio)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_param_validation.py:218, in validate_params.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    212 try:
    213     with config_context(
    214         skip_parameter_validation=(
    215             prefer_skip_nested_validation or global_skip_validation
    216         )
    217     ):
--> 218         return func(*args, **kwargs)
    219 except InvalidParameterError as e:
    220     # When the function is just a wrapper around an estimator, we allow
    221     # the function to delegate validation to the estimator, but we replace
    222     # the name of the estimator by the name of the function in the error
    223     # message to avoid confusion.
    224     msg = re.sub(
    225         r"parameter of \w+ must be",
    226         f"parameter of {func.__qualname__} must be",
    227         str(e),
    228     )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\metrics\pairwise.py:345, in euclidean_distances(X, Y, Y_norm_squared, squared, X_norm_squared)
    271 """
    272 Compute the distance matrix between each pair from a feature array X and Y.
    273 
   (...)
    342        [1.41421356]])
    343 """
    344 xp, _ = get_namespace(X, Y)
--> 345 X, Y = check_pairwise_arrays(X, Y)
    347 if X_norm_squared is not None:
    348     X_norm_squared = check_array(X_norm_squared, ensure_2d=False)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\metrics\pairwise.py:175, in check_pairwise_arrays(X, Y, precomputed, dtype, accept_sparse, ensure_all_finite, ensure_2d, copy)
    172     dtype = dtype_float
    174 if Y is X or Y is None:
--> 175     X = Y = check_array(
    176         X,
    177         accept_sparse=accept_sparse,
    178         dtype=dtype,
    179         copy=copy,
    180         ensure_all_finite=ensure_all_finite,
    181         estimator=estimator,
    182         ensure_2d=ensure_2d,
    183     )
    184 else:
    185     X = check_array(
    186         X,
    187         accept_sparse=accept_sparse,
   (...)
    192         ensure_2d=ensure_2d,
    193     )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:1038, in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, ensure_all_finite, ensure_non_negative, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)
   1036         array = xp.astype(array, dtype, copy=False)
   1037     else:
-> 1038         array = _asarray_with_order(array, order=order, dtype=dtype, xp=xp)
   1039 except ComplexWarning as complex_warning:
   1040     raise ValueError(
   1041         "Complex data not supported\n{}\n".format(array)
   1042     ) from complex_warning

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_array_api.py:983, in _asarray_with_order(array, dtype, order, copy, xp, device)
    981     array = numpy.array(array, order=order, dtype=dtype)
    982 else:
--> 983     array = numpy.asarray(array, order=order, dtype=dtype)
    985 # At this point array is a NumPy ndarray. We convert it to an array
    986 # container that is consistent with the input's namespace.
    987 return xp.asarray(array)

ValueError: could not convert string to float: 'Adelie'
d = euclidean_distances(trio[features])
pd.DataFrame(d, index=trio.index, columns=trio.index).round(1)
0 1 200
0 0.0 50.3 77.7
1 50.3 0.0 126.0
200 77.7 126.0 0.0

Quick check: which column is dominating those distances?

X.std().round(1)
bill_length_mm         5.5
bill_depth_mm          2.0
flipper_length_mm     14.1
body_mass_g          802.0
dtype: float64
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
ds = euclidean_distances(X_scaled[[0, 1, 200]])
pd.DataFrame(ds, index=trio.index, columns=trio.index).round(2)
0 1 200
0 0.00 0.76 2.41
1 0.76 0.00 2.08
200 2.41 2.08 0.00
import numpy as np
gap = X_scaled[0] - X_scaled[1]
print(gap.round(2))
print(round(np.sqrt((gap ** 2).sum()), 2))
[-0.07  0.66 -0.36 -0.06]
0.76

A first clustering with k-means#

Problem: assign every penguin to a group automatically.

  • Choose k, the number of clusters, and drop k centres
  • Give each bird to its nearest centre
  • Move each centre to the middle of its birds; repeat until stable

Common mistake: feeding k-means rows with missing values

from sklearn.cluster import KMeans
km = KMeans(n_clusters=3, n_init=10, random_state=42)
km.fit(df[features])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[13], line 3
      1 from sklearn.cluster import KMeans
      2 km = KMeans(n_clusters=3, n_init=10, random_state=42)
----> 3 km.fit(df[features])

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py:1403, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1393     estimator._validate_params()
   1395 with (
   1396     config_context(
   1397         skip_parameter_validation=(
   (...)
   1401     callback_management_context(estimator),
   1402 ):
-> 1403     return fit_method(estimator, *args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\cluster\_kmeans.py:1459, in KMeans.fit(self, X, y, sample_weight)
   1431 @_fit_context(prefer_skip_nested_validation=True)
   1432 def fit(self, X, y=None, sample_weight=None):
   1433     """Compute k-means clustering.
   1434 
   1435     Parameters
   (...)
   1457         Fitted estimator.
   1458     """
-> 1459     X = validate_data(
   1460         self,
   1461         X,
   1462         accept_sparse="csr",
   1463         dtype=[np.float64, np.float32],
   1464         order="C",
   1465         copy=self.copy_x,
   1466         accept_large_sparse=False,
   1467     )
   1469     self._check_params_vs_input(X)
   1471     random_state = check_random_state(self.random_state)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:3041, in validate_data(_estimator, X, y, reset, validate_separately, skip_check_array, **check_params)
   3039         out = X, y
   3040 elif not no_val_X and no_val_y:
-> 3041     out = check_array(X, input_name="X", **check_params)
   3042 elif no_val_X and not no_val_y:
   3043     out = _check_y(y, **check_params)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:1090, in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, ensure_all_finite, ensure_non_negative, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)
   1084     raise ValueError(
   1085         f"Found array with dim {array.ndim},"
   1086         f" while dim <= 2 is required{context}."
   1087     )
   1089 if ensure_all_finite:
-> 1090     _assert_all_finite(
   1091         array,
   1092         input_name=input_name,
   1093         estimator_name=estimator_name,
   1094         allow_nan=ensure_all_finite == "allow-nan",
   1095     )
   1097 if copy:
   1098     if _is_numpy_namespace(xp):
   1099         # only make a copy if `array` and `array_orig` may share memory`

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:137, in _assert_all_finite(X, allow_nan, msg_dtype, estimator_name, input_name)
    134 if first_pass_isfinite:
    135     return
--> 137 _assert_all_finite_element_wise(
    138     X,
    139     xp=xp,
    140     allow_nan=allow_nan,
    141     msg_dtype=msg_dtype,
    142     estimator_name=estimator_name,
    143     input_name=input_name,
    144 )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:186, in _assert_all_finite_element_wise(X, xp, allow_nan, msg_dtype, estimator_name, input_name)
    169 if estimator_name and input_name == "X" and has_nan_error:
    170     # Improve the error message on how to handle missing values in
    171     # scikit-learn.
    172     msg_err += (
    173         f"\n{estimator_name} does not accept missing values"
    174         " encoded as NaN natively. For supervised learning, you might want"
   (...)
    184         "#estimators-that-handle-nan-values"
    185     )
--> 186 raise ValueError(msg_err)

ValueError: Input X contains NaN.
KMeans does not accept missing values encoded as NaN natively. For supervised learning, you might want to consider sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept missing values encoded as NaNs natively. Alternatively, it is possible to preprocess the data, for instance by using an imputer transformer in a pipeline or drop samples with missing values. See https://scikit-learn.org/stable/modules/impute.html You can find a list of all estimators that handle NaN values at the following page: https://scikit-learn.org/stable/modules/impute.html#estimators-that-handle-nan-values
km.fit(X_scaled)
peng["raw_label"] = km.labels_
peng["raw_label"].value_counts().sort_index()
raw_label
0     87
1    123
2    132
Name: count, dtype: int64
order = peng.groupby("raw_label")["flipper_length_mm"].mean()
rank = order.rank().astype(int)
peng["cluster"] = peng["raw_label"].map(rank)
peng["cluster"].value_counts().sort_index()
cluster
1    132
2     87
3    123
Name: count, dtype: int64
peng.groupby("cluster")[features].mean().round(1)
bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
cluster
1 38.2 18.1 188.4 3584.7
2 47.5 18.8 196.9 3902.0
3 47.5 15.0 217.2 5076.0
centres = scaler.inverse_transform(km.cluster_centers_)
centres = pd.DataFrame(centres, columns=features)
centres.index = rank.to_list()
centres.sort_index().round(1)
bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
1 38.2 18.1 188.4 3584.7
2 47.5 18.8 196.9 3902.0
3 47.5 15.0 217.2 5076.0
for c, grp in peng.groupby("cluster"):
    plt.scatter(grp["bill_length_mm"], grp["flipper_length_mm"],
                label=f"cluster {c}", alpha=0.6)
plt.title("k-means clusters (k = 3, scaled data)")
plt.xlabel("Bill length (mm)")
plt.ylabel("Flipper length (mm)")
plt.legend()
plt.show()
No description has been provided for this image
near = km.transform(X_scaled[[0, 200]])
near = pd.DataFrame(near, columns=rank.to_list(), index=[0, 200])
near.sort_index(axis=1).round(2)
1 2 3
0 0.66 1.93 3.91
200 2.27 0.90 2.55

Quick check: which species do you expect in cluster 2?

pd.crosstab(peng["cluster"], peng["species"])
species Adelie Chinstrap Gentoo
cluster
1 127 5 0
2 24 63 0
3 0 0 123

Your turn#

Run the same k-means on the unscaled measurements. How many birds per cluster?

km_raw = KMeans(n_clusters=3, n_init=10, random_state=42)
raw = km_raw.fit_predict(X)
peng.assign(raw=raw).groupby("raw")["body_mass_g"].agg(
    ["size", "min", "max"])
size min max
raw
0 165 2700.0 3975.0
1 68 4975.0 6300.0
2 109 4000.0 4950.0

How many clusters? Let the data advise#

Problem: k-means needs k, but nobody tells us the right number.

inertia = []
for k in range(1, 9):
    m = KMeans(n_clusters=k, n_init=10, random_state=42)
    inertia.append(m.fit(X_scaled).inertia_)
print([round(v) for v in inertia])
[1368, 566, 379, 300, 233, 204, 187, 171]
plt.plot(range(1, 9), inertia, marker="o")
plt.title("Elbow chart: inertia by number of clusters")
plt.xlabel("Number of clusters (k)")
plt.ylabel("Inertia (lower = tighter)")
plt.show()
No description has been provided for this image

Quick check: silhouette scores run from -1 to 1. Which k will score best?

from sklearn.metrics import silhouette_score
for k in range(2, 7):
    m = KMeans(n_clusters=k, n_init=10, random_state=42)
    lab = m.fit_predict(X_scaled)
    print(k, round(silhouette_score(X_scaled, lab), 3))
2 0.532
3 0.447
4 0.4
5 0.379
6 0.368
km2 = KMeans(n_clusters=2, n_init=10, random_state=42)
peng["k2"] = km2.fit_predict(X_scaled)
pd.crosstab(peng["k2"], peng["species"])
species Adelie Chinstrap Gentoo
k2
0 151 68 0
1 0 0 123

Your turn#

Fit k-means with k = 4. Which species gets split, and by what?

km4 = KMeans(n_clusters=4, n_init=10, random_state=42)
peng["k4"] = km4.fit_predict(X_scaled)
pd.crosstab(peng["k4"], [peng["species"], peng["sex"]])
species Adelie Chinstrap Gentoo
sex FEMALE MALE FEMALE MALE FEMALE MALE
k4
0 0 0 0 0 56 6
1 0 23 29 34 0 0
2 0 0 0 0 2 55
3 73 50 5 0 0 0

Three families of clustering#

Problem: k-means likes round blobs. Other shapes need other methods.

  • Centroid-based (k-means): groups around centre points
  • Hierarchical: merge the closest groups step by step
  • Density-based (DBSCAN): dense regions; sparse points are noise
from sklearn.cluster import AgglomerativeClustering
ward = AgglomerativeClustering(n_clusters=3, linkage="ward")
peng["ward"] = ward.fit_predict(X_scaled)
pd.crosstab(peng["ward"], peng["species"])
species Adelie Chinstrap Gentoo
ward
0 151 11 0
1 0 0 123
2 0 57 0
from sklearn.cluster import DBSCAN
db = DBSCAN(eps=0.6, min_samples=5)
peng["dbscan"] = db.fit_predict(X_scaled)
peng["dbscan"].value_counts().sort_index()
dbscan
-1     33
 0    193
 1    116
Name: count, dtype: int64
noise = peng["dbscan"] == -1
plt.scatter(peng.loc[~noise, "bill_length_mm"],
            peng.loc[~noise, "flipper_length_mm"],
            c=peng.loc[~noise, "dbscan"], alpha=0.6)
plt.scatter(peng.loc[noise, "bill_length_mm"],
            peng.loc[noise, "flipper_length_mm"],
            color="black", marker="x", label="noise")
plt.title("DBSCAN: dense groups and noise")
plt.xlabel("Bill length (mm)")
plt.ylabel("Flipper length (mm)")
plt.legend()
No description has been provided for this image

Your turn#

Repeat the hierarchical clustering with linkage="single". What goes wrong?

single = AgglomerativeClustering(n_clusters=3, linkage="single")
lab = single.fit_predict(X_scaled)
pd.Series(lab).value_counts()
0    218
2    123
1      1
Name: count, dtype: int64

Quick check: by silhouette, is Ward's 3-cluster answer better than k-means'?

print(round(silhouette_score(X_scaled, peng["cluster"]), 3))
print(round(silhouette_score(X_scaled, peng["ward"]), 3))
0.447
0.454

Mini-project: which measurements reveal the species?#

Cluster with different feature sets; score agreement with the hidden species.

from sklearn.metrics import adjusted_rand_score
truth = peng["species"]
print(round(adjusted_rand_score(truth, peng["cluster"]), 3))
print(round(adjusted_rand_score(truth, raw), 3))
print(round(adjusted_rand_score(truth, peng["ward"]), 3))
0.793
0.314
0.916
def cluster_score(cols):
    Z = StandardScaler().fit_transform(peng[cols])
    m = KMeans(n_clusters=3, n_init=10, random_state=42)
    return round(adjusted_rand_score(truth, m.fit_predict(Z)), 3)
print(cluster_score(["bill_length_mm", "bill_depth_mm"]))
print(cluster_score(["flipper_length_mm", "body_mass_g"]))
print(cluster_score(["bill_length_mm", "flipper_length_mm"]))
print(cluster_score(features))
0.826
0.423
0.887
0.793
best_cols = ["bill_length_mm", "flipper_length_mm"]
Z = StandardScaler().fit_transform(peng[best_cols])
m = KMeans(n_clusters=3, n_init=10, random_state=42)
peng["pair"] = m.fit_predict(Z)
pd.crosstab(peng["pair"], peng["species"])
species Adelie Chinstrap Gentoo
pair
0 1 4 122
1 146 5 0
2 4 59 1
match = pd.crosstab(peng["pair"], peng["species"]).max(axis=1)
share = match.sum() / len(peng)
print(f"{share:.1%} of penguins share a cluster with their species")
95.6% of penguins share a cluster with their species

Recap#

  • Clustering groups similar rows without any labels
  • Similarity is distance, so scale features first
  • k-means: choose k, assign to nearest centre, move centres, repeat
  • Elbow and silhouette advise on k; domain sense decides
  • Hierarchical and DBSCAN suit other shapes of data

Challenges#

  1. Cluster with k = 3 using bill depth and body mass. Which species mix up?
  2. Try DBSCAN with eps 0.5 and 0.8. How do noise and cluster counts change?
  3. Cross-tab the k = 3 clusters against sex. Do any clusters split by sex?

What's next#

Lesson 28: k-Means Clustering Algorithm Watch k-means step by step, pick k with care, and avoid its traps.

Data Mining 28: k-Means Clustering Algorithm#

In this lesson you'll be able to:

  • Run the two steps of k-means by hand and watch them converge
  • Explain why starting points matter, and how n_init helps
  • Choose k with inertia, silhouette and business sense
  • Name segments and place brand-new rows into them Dataset: 398 real cars from 1970-1982 (data/pandas/mpg.csv)

Warm-up: k-means on penguins#

Lesson 27 grouped penguins with k-means. One quick replay.

import pandas as pd
from sklearn.preprocessing import StandardScaler
peng = pd.read_csv("penguins.csv")
pair = ["bill_length_mm", "flipper_length_mm"]
peng = peng.dropna(subset=pair)
Zp = StandardScaler().fit_transform(peng[pair])
from sklearn.cluster import KMeans
km = KMeans(n_clusters=3, n_init=10, random_state=42)
peng["cluster"] = km.fit_predict(Zp)
pd.crosstab(peng["cluster"], peng["species"])
species Adelie Chinstrap Gentoo
cluster
0 1 4 122
1 146 5 0
2 4 59 1

Getting cars ready for k-means#

Problem: 398 cars and no segment label. Can the data suggest some?

df = pd.read_csv("mpg.csv")
print(df.shape)
df[["name", "mpg", "horsepower", "weight", "model_year"]].head(3)
(398, 9)
name mpg horsepower weight model_year
0 chevrolet chevelle malibu 18.0 130.0 3504 70
1 buick skylark 320 15.0 165.0 3693 70
2 plymouth satellite 18.0 150.0 3436 70
df.isna().sum()
mpg             0
cylinders       0
displacement    0
horsepower      6
weight          0
acceleration    0
model_year      0
origin          0
name            0
dtype: int64
df.loc[df["horsepower"].isna(), ["name", "model_year"]]
name model_year
32 ford pinto 71
126 ford maverick 74
330 renault lecar deluxe 80
336 ford mustang cobra 80
354 renault 18i 81
374 amc concord dl 82
features = ["mpg", "horsepower", "weight", "acceleration"]
cars = df.dropna(subset=features).reset_index(drop=True)
cars[features].describe().round(1)
mpg horsepower weight acceleration
count 392.0 392.0 392.0 392.0
mean 23.4 104.5 2977.6 15.5
std 7.8 38.5 849.4 2.8
min 9.0 46.0 1613.0 8.0
25% 17.0 75.0 2225.2 13.8
50% 22.8 93.5 2803.5 15.5
75% 29.0 126.0 3614.8 17.0
max 46.6 230.0 5140.0 24.8

Quick check: which feature would swamp the distances if we left the units alone?

cars[features].std().round(1)
mpg               7.8
horsepower       38.5
weight          849.4
acceleration      2.8
dtype: float64
scaler = StandardScaler()
X = scaler.fit_transform(cars[features])
print(X.mean(axis=0).round(2), X.std(axis=0).round(2))
[ 0. -0. -0.  0.] [1. 1. 1. 1.]

Watching k-means work, step by step#

Problem: k-means feels like magic until you run its steps yourself.

  1. Start: pick k rows as the first centres
  2. Assign: each car joins its nearest centre
  3. Update: move each centre to the mean of its cars; repeat 2-3
import matplotlib.pyplot as plt
plt.scatter(cars["weight"], cars["mpg"], color="grey", alpha=0.5)
plt.title("392 cars before clustering")
plt.xlabel("Weight (lb)")
plt.ylabel("Miles per gallon")
plt.show()
No description has been provided for this image
import numpy as np
two = ["weight", "mpg"]
Z = StandardScaler().fit_transform(cars[two])
start = cars.sample(3, random_state=42).index
centres = Z[start]
cars.loc[start, ["name"] + two]
name weight mpg
78 renault 12 (sw) 2189 26.0
274 saab 99gle 2795 21.6
246 honda civic cvcc 1800 36.1
from sklearn.metrics.pairwise import euclidean_distances
d = euclidean_distances(Z, centres)
labels = d.argmin(axis=1)
np.bincount(labels)
array([ 93, 229,  70])
centres = np.array([Z[labels == j].mean(axis=0) for j in range(3)])
labels = euclidean_distances(Z, centres).argmin(axis=1)
np.bincount(labels)
array([142, 179,  71])

Quick check: if we keep repeating, when should we stop?

def my_kmeans(Z, start, rounds=100):
    centres, labels = Z[start], None
    for step in range(1, rounds + 1):
        new = euclidean_distances(Z, centres).argmin(axis=1)
        if labels is not None and (new == labels).all():
            break
        labels = new
        k = len(centres)
        centres = np.array([Z[labels == j].mean(axis=0)
                            for j in range(k)])
    return labels, centres, step
labels, centres, step = my_kmeans(Z, start)
inertia = ((Z - centres[labels]) ** 2).sum()
print("stopped at round", step, "sizes", np.bincount(labels))
print(round(inertia, 2))
stopped at round 17 sizes [148 125 119]
148.46
plt.scatter(cars["weight"], cars["mpg"], c=labels, alpha=0.6)
sd = cars[two].std(ddof=0).to_numpy()
cent = centres * sd + cars[two].mean().to_numpy()
plt.scatter(cent[:, 0], cent[:, 1], c="red", marker="X", s=200)
plt.title("Our hand-made k-means: weight vs mpg")
plt.xlabel("Weight (lb)")
plt.ylabel("Miles per gallon")
plt.show()
No description has been provided for this image

Common mistake: giving KMeans starting centres that don't match k

check = KMeans(n_clusters=4, init=Z[start], n_init=1)
check.fit(Z)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[17], line 2
      1 check = KMeans(n_clusters=4, init=Z[start], n_init=1)
----> 2 check.fit(Z)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py:1403, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1393     estimator._validate_params()
   1395 with (
   1396     config_context(
   1397         skip_parameter_validation=(
   (...)
   1401     callback_management_context(estimator),
   1402 ):
-> 1403     return fit_method(estimator, *args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\cluster\_kmeans.py:1480, in KMeans.fit(self, X, y, sample_weight)
   1478 if init_is_array_like:
   1479     init = check_array(init, dtype=X.dtype, copy=True, order="C")
-> 1480     self._validate_center_shape(X, init)
   1482 # subtract of mean of x for more accurate distance computations
   1483 if not sp.issparse(X):

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\cluster\_kmeans.py:938, in _BaseKMeans._validate_center_shape(self, X, centers, n_centroids)
    935     n_centroids = self.n_clusters
    937 if centers.shape[0] != n_centroids:
--> 938     raise ValueError(
    939         f"The shape of the initial centers {centers.shape} does not "
    940         f"match the number of clusters {n_centroids}."
    941     )
    942 if centers.shape[1] != X.shape[1]:
    943     raise ValueError(
    944         f"The shape of the initial centers {centers.shape} does not "
    945         f"match the number of features of the data {X.shape[1]}."
    946     )

ValueError: The shape of the initial centers (3, 2) does not match the number of clusters 4.
check = KMeans(n_clusters=3, init=Z[start], n_init=1)
check.fit(Z)
print(np.bincount(check.labels_), round(check.inertia_, 2))
[150 125 117] 148.51

Starting points matter: k-means++ and n_init#

Problem: an unlucky start can trap k-means in a poorer answer.

for seed in range(5):
    m = KMeans(n_clusters=6, init="random", n_init=1,
               random_state=seed)
    print(seed, round(m.fit(X).inertia_, 1))
0 312.7
1 312.6
2 296.9
3 312.1
4 312.6
  • k-means++ (the default) spreads the first centres far apart
  • n_init runs several starts and keeps the lowest inertia
for n in [1, 10]:
    m = KMeans(n_clusters=6, n_init=n, random_state=42).fit(X)
    print(n, round(m.inertia_, 1))
1 298.5
10 296.9

Your turn#

Run my_kmeans from 3 cars picked with random_state=7. Same clusters as before?

start7 = cars.sample(3, random_state=7).index
lab7, cen7, step7 = my_kmeans(Z, start7)
in7 = ((Z - cen7[lab7]) ** 2).sum()
print(step7, np.bincount(lab7), round(in7, 2))
5 [119 148 125] 148.46

Choosing k for the car market#

Problem: k-means needs k up front. The data can advise; we decide.

Common mistake: asking for a silhouette score with one cluster

from sklearn.metrics import silhouette_score
for k in range(1, 8):
    m = KMeans(n_clusters=k, n_init=10, random_state=42)
    print(k, silhouette_score(X, m.fit_predict(X)))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[22], line 4
      2 for k in range(1, 8):
      3     m = KMeans(n_clusters=k, n_init=10, random_state=42)
----> 4     print(k, silhouette_score(X, m.fit_predict(X)))

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_param_validation.py:218, in validate_params.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    212 try:
    213     with config_context(
    214         skip_parameter_validation=(
    215             prefer_skip_nested_validation or global_skip_validation
    216         )
    217     ):
--> 218         return func(*args, **kwargs)
    219 except InvalidParameterError as e:
    220     # When the function is just a wrapper around an estimator, we allow
    221     # the function to delegate validation to the estimator, but we replace
    222     # the name of the estimator by the name of the function in the error
    223     # message to avoid confusion.
    224     msg = re.sub(
    225         r"parameter of \w+ must be",
    226         f"parameter of {func.__qualname__} must be",
    227         str(e),
    228     )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\metrics\cluster\_unsupervised.py:146, in silhouette_score(X, labels, metric, sample_size, random_state, **kwds)
    144     else:
    145         X, labels = X[indices], labels[indices]
--> 146 return float(np.mean(silhouette_samples(X, labels, metric=metric, **kwds)))

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_param_validation.py:191, in validate_params.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    189 global_skip_validation = get_config()["skip_parameter_validation"]
    190 if global_skip_validation:
--> 191     return func(*args, **kwargs)
    193 func_sig = signature(func)
    195 # Map *args/**kwargs to the function signature

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\metrics\cluster\_unsupervised.py:304, in silhouette_samples(X, labels, metric, **kwds)
    302 n_samples = len(labels)
    303 label_freqs = np.bincount(labels)
--> 304 check_number_of_labels(len(le.classes_), n_samples)
    306 kwds["metric"] = metric
    307 reduce_func = functools.partial(
    308     _silhouette_reduce, labels=labels, label_freqs=label_freqs
    309 )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\metrics\cluster\_unsupervised.py:43, in check_number_of_labels(n_labels, n_samples)
     32 """Check that number of labels are valid.
     33 
     34 Parameters
   (...)
     40     Number of samples.
     41 """
     42 if not 1 < n_labels < n_samples:
---> 43     raise ValueError(
     44         "Number of labels is %d. Valid values are 2 to n_samples - 1 (inclusive)"
     45         % n_labels
     46     )

ValueError: Number of labels is 1. Valid values are 2 to n_samples - 1 (inclusive)
rows = []
for k in range(2, 8):
    m = KMeans(n_clusters=k, n_init=10, random_state=42).fit(X)
    sil = silhouette_score(X, m.labels_)
    rows.append({"k": k, "inertia": m.inertia_, "silhouette": sil})
scores = pd.DataFrame(rows).set_index("k")
scores.round(3)
inertia silhouette
k
2 719.871 0.499
3 491.478 0.375
4 396.776 0.355
5 341.994 0.334
6 296.947 0.296
7 266.086 0.294
scores["inertia"].plot(marker="o")
plt.title("Inertia by number of car segments")
plt.xlabel("Number of clusters (k)")
plt.ylabel("Inertia (lower = tighter)")
plt.show()
No description has been provided for this image
scores["silhouette"].plot(kind="bar", color="steelblue")
plt.title("Silhouette score by number of car segments")
plt.xlabel("Number of clusters (k)")
plt.ylabel("Silhouette (higher = better separated)")
plt.show()
No description has been provided for this image

Quick check: k = 2 scores best. Is two segments useful to a car maker?

m2 = KMeans(n_clusters=2, n_init=10, random_state=42).fit(X)
cars.groupby(m2.labels_)[features].mean().round(1)
mpg horsepower weight acceleration
0 26.5 84.9 2578.8 16.5
1 14.7 160.0 4111.3 12.8

Naming the segments#

Problem: cluster numbers mean nothing to a sales team.

train = cars[cars["model_year"] < 82]
new = cars[cars["model_year"] == 82]
scaler = StandardScaler().fit(train[features])
Xtr = scaler.transform(train[features])
km = KMeans(n_clusters=3, n_init=10, random_state=42).fit(Xtr)
print(len(train), len(new))
362 30
heavy = train.groupby(km.labels_)["weight"].mean().rank()
names = {1: "economy", 2: "mid-size", 3: "heavy"}
seg_of = {c: names[int(r)] for c, r in heavy.items()}
train = train.assign(segment=[seg_of[c] for c in km.labels_])
train["segment"].value_counts()
segment
mid-size    143
economy     122
heavy        97
Name: count, dtype: int64
train.groupby("segment")[features].mean().round(1)
mpg horsepower weight acceleration
segment
economy 31.3 71.5 2173.4 16.8
heavy 14.5 161.8 4144.0 12.6
mid-size 21.0 98.5 2986.5 16.2
centres = scaler.inverse_transform(km.cluster_centers_)
centres = pd.DataFrame(centres, columns=features)
centres.index = [seg_of[c] for c in range(3)]
centres.sort_index().round(1)
mpg horsepower weight acceleration
economy 31.3 71.5 2173.4 16.8
heavy 14.5 161.8 4144.0 12.6
mid-size 21.0 98.5 2986.5 16.2
pd.crosstab(train["segment"], train["cylinders"])
cylinders 3 4 5 6 8
segment
economy 0 120 1 1 0
heavy 0 0 0 1 96
mid-size 4 52 2 78 7

Your turn#

Which share of each segment comes from the USA, Europe and Japan?

pd.crosstab(train["segment"], train["origin"],
            normalize="index").round(2)
origin europe japan usa
segment
economy 0.35 0.39 0.26
heavy 0.00 0.00 1.00
mid-size 0.16 0.16 0.68

Placing new cars into segments#

Problem: the 1982 models arrive after the model was built.

Common mistake: new rows with the columns in a different order

shuffled = ["weight", "mpg", "horsepower", "acceleration"]
scaler.transform(new[shuffled])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[33], line 2
      1 shuffled = ["weight", "mpg", "horsepower", "acceleration"]
----> 2 scaler.transform(new[shuffled])

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_set_output.py:327, in _wrap_method_output.<locals>.wrapped(self, X, *args, **kwargs)
    325 @wraps(f)
    326 def wrapped(self, X, *args, **kwargs):
--> 327     data_to_wrap = f(self, X, *args, **kwargs)
    328     if isinstance(data_to_wrap, tuple):
    329         # only wrap the first output for cross decomposition
    330         return_tuple = (
    331             _wrap_data_with_container(method, data_to_wrap[0], X, self),
    332             *data_to_wrap[1:],
    333         )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\preprocessing\_data.py:1111, in StandardScaler.transform(self, X, copy)
   1108 check_is_fitted(self)
   1110 copy = copy if copy is not None else self.copy
-> 1111 X = validate_data(
   1112     self,
   1113     X,
   1114     reset=False,
   1115     accept_sparse="csr",
   1116     copy=copy,
   1117     dtype=supported_float_dtypes(xp, X_device),
   1118     force_writeable=True,
   1119     ensure_all_finite="allow-nan",
   1120 )
   1122 if sparse.issparse(X):
   1123     if self.with_mean:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:3016, in validate_data(_estimator, X, y, reset, validate_separately, skip_check_array, **check_params)
   2932 def validate_data(
   2933     _estimator,
   2934     /,
   (...)
   2940     **check_params,
   2941 ):
   2942     """Validate input data and set or check feature names and counts of the input.
   2943 
   2944     This helper function should be used in an estimator that requires input
   (...)
   3014         validated.
   3015     """
-> 3016     _check_feature_names(_estimator, X, reset=reset)
   3017     tags = get_tags(_estimator)
   3018     if y is None and tags.target_tags.required:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:2868, in _check_feature_names(estimator, X, reset)
   2865 if not missing_names and not unexpected_names:
   2866     message += "Feature names must be in the same order as they were in fit.\n"
-> 2868 raise ValueError(message)

ValueError: The feature names should match those that were passed during fit.
Feature names must be in the same order as they were in fit.
Xnew = scaler.transform(new[features])
pred = [seg_of[c] for c in km.predict(Xnew)]
new = new.assign(segment=pred)
new[["name", "mpg", "weight", "segment"]].head(6)
name mpg weight segment
362 chevrolet cavalier 28.0 2605 economy
363 chevrolet cavalier wagon 27.0 2640 economy
364 chevrolet cavalier 2-door 34.0 2395 economy
365 pontiac j2000 se hatchback 31.0 2575 economy
366 dodge aries se 29.0 2525 economy
367 pontiac phoenix 27.0 2735 economy

Quick check: will any 1982 car land in the heavy segment?

new["segment"].value_counts()
segment
economy     24
mid-size     6
Name: count, dtype: int64
car = (new["name"] == "ford mustang gl").to_numpy()
dist = km.transform(Xnew[car])[0]
centre_names = [seg_of[c] for c in range(3)]
pd.Series(dist, index=centre_names).round(2).sort_values()
mid-size    0.91
economy     1.08
heavy       3.17
dtype: float64
d_all = np.sort(km.transform(Xnew), axis=1)
margin = d_all[:, 1] - d_all[:, 0]
new.assign(margin=margin.round(2)).nsmallest(3, "margin")[
    ["name", "mpg", "weight", "segment", "margin"]]
name mpg weight segment margin
367 pontiac phoenix 27.0 2735 economy 0.02
387 ford mustang gl 27.0 2790 mid-size 0.17
363 chevrolet cavalier wagon 27.0 2640 economy 0.18

Mini-project: did buyers shift to economy cars?#

The 1970s brought two oil crises. Did the economy segment grow?

all_X = scaler.transform(cars[features])
cars = cars.assign(segment=[seg_of[c] for c in km.predict(all_X)])
share = pd.crosstab(cars["model_year"], cars["segment"],
                    normalize="index")
share.round(2).iloc[[0, 3, 6, 9, 12]]
segment economy heavy mid-size
model_year
70 0.07 0.62 0.31
73 0.10 0.50 0.40
76 0.35 0.26 0.38
79 0.38 0.24 0.38
82 0.80 0.00 0.20
share.plot(marker="o")
plt.title("Share of new car models by segment, 1970-1982")
plt.xlabel("Model year (19xx)")
plt.ylabel("Share of models")
plt.show()
No description has been provided for this image
eco = cars[cars["segment"] == "economy"]
eco.groupby("model_year")["mpg"].mean().round(1).iloc[[0, 6, 12]]
model_year
70    26.5
76    28.1
82    33.7
Name: mpg, dtype: float64
early = share.loc[70:72, "economy"].mean()
late = share.loc[80:82, "economy"].mean()
print(f"Economy share: {early:.0%} in 1970-72, {late:.0%} in 1980-82")
Economy share: 21% in 1970-72, 76% in 1980-82

Recap#

  • k-means repeats two steps: assign to the nearest centre, move centres
  • It stops when no row changes cluster; inertia measures tightness
  • Starts matter: k-means++ and n_init guard against bad answers
  • Silhouette advises on k; business sense makes the final call
  • Rank clusters to name them; predict new rows with the same scaler

Challenges#

  1. Add displacement to the features. Do the three segments change?
  2. Try k = 4. Which segment splits, and does the split make sense?
  3. Run my_kmeans on all four features and compare with KMeans.

What's next#

Lesson 29: Hierarchical Clustering Techniques Build a family tree of clusters and choose k by cutting it.

Data Mining 29: Hierarchical Clustering Techniques#

In this lesson you'll be able to:

  • Build a cluster tree from the bottom up with linkage
  • Read a dendrogram and cut it into clusters
  • Compare single, complete, average and Ward linkage
  • Use AgglomerativeClustering, and place new rows afterwards Dataset: 398 real cars and their makers (data/pandas/mpg.csv)

Warm-up: three segments with k-means#

Lesson 28 found three car segments with k-means. One quick replay.

import pandas as pd
from sklearn.preprocessing import StandardScaler
df = pd.read_csv("mpg.csv")
features = ["mpg", "horsepower", "weight", "acceleration"]
cars = df.dropna(subset=features).reset_index(drop=True)
X = StandardScaler().fit_transform(cars[features])
from sklearn.cluster import KMeans
km = KMeans(n_clusters=3, n_init=10, random_state=42).fit(X)
rank = cars.groupby(km.labels_)["weight"].mean().rank().astype(int)
cars["km"] = [rank[c] for c in km.labels_]
cars["km"].value_counts().sort_index()
km
1    142
2    153
3     97
Name: count, dtype: int64

From cars to brands: tidying the names#

Problem: we want to compare car makers, but the brand is hidden in the name.

cars["brand"] = cars["name"].str.split().str[0]
counts = cars["brand"].value_counts()
print(len(counts))
counts.tail(12)
37
brand
bmw              2
maxda            2
mercedes-benz    2
cadillac         2
hi               1
toyouta          1
chevroelt        1
capri            1
mercedes         1
vokswagen        1
triumph          1
nissan           1
Name: count, dtype: int64
odd = cars["brand"].isin(["hi", "capri"])
cars.loc[odd, ["name", "model_year"]]
name model_year
28 hi 1200d 70
182 capri ii 76

Quick check: which of those rare brands are really typos or nicknames?

fix = {"chevy": "chevrolet", "chevroelt": "chevrolet",
       "vw": "volkswagen", "vokswagen": "volkswagen",
       "maxda": "mazda", "toyouta": "toyota",
       "mercedes": "mercedes-benz"}
cars["brand"] = cars["brand"].replace(fix)
print(cars["brand"].nunique())
30
counts = cars["brand"].value_counts()
keep = counts[counts >= 4].index
mask = cars["brand"].isin(keep)
brands = cars[mask].groupby("brand")[features].mean()
print(len(brands))
brands.round(1).head()
22
mpg horsepower weight acceleration
brand
amc 18.1 114.7 3289.7 15.1
audi 26.7 86.7 2556.1 15.9
buick 19.2 136.4 3553.4 14.7
chevrolet 20.2 114.6 3317.4 15.4
chrysler 17.3 153.7 3912.0 13.3

Quick check: which three brands have the best fuel economy?

brands["mpg"].nlargest(3).round(1)
brand
honda         33.8
volkswagen    31.8
datsun        31.1
Name: mpg, dtype: float64
Xb = StandardScaler().fit_transform(brands)
names = brands.index.to_list()
print(Xb.shape, names[:4])
(22, 4) ['amc', 'audi', 'buick', 'chevrolet']

Building the tree from the bottom up#

Problem: k-means needs k first. A tree lets us decide later.

  • Start with every brand in its own cluster
  • Merge the two closest clusters; repeat until one is left
  • Ward picks the merge that adds the least spread
from scipy.cluster.hierarchy import linkage
Z = linkage(Xb, method="ward")
print(Z.shape)
Z[:3].round(3)
(21, 4)
array([[ 3.   ,  8.   ,  0.204,  2.   ],
       [ 0.   , 11.   ,  0.387,  2.   ],
       [ 6.   , 15.   ,  0.403,  2.   ]])

Each row of Z: cluster a, cluster b, merge distance, size Numbers 0-21 are brands; 22 and up are clusters made earlier

def label(i):
    i = int(i)
    return names[i] if i < len(names) else f"group {i}"

for a, b, height, size in Z[:6]:
    print(label(a), "+", label(b), round(height, 2))
chevrolet + ford 0.2
amc + mercury 0.39
dodge + plymouth 0.4
datsun + mazda 0.42
audi + toyota 0.52
buick + pontiac 0.56

Quick check: why might Chevrolet and Ford merge first?

brands.loc[["chevrolet", "ford", "dodge", "plymouth"]].round(1)
mpg horsepower weight acceleration
brand
chevrolet 20.2 114.6 3317.4 15.4
ford 19.5 112.2 3344.5 15.3
dodge 22.1 117.2 3206.2 14.5
plymouth 21.7 111.4 3219.0 14.9

Reading a dendrogram#

Problem: a merge table is hard to read. A dendrogram draws it.

Common mistake: labelling the leaves with the wrong list

from scipy.cluster.hierarchy import dendrogram
dendrogram(Z, labels=cars["brand"].to_list())
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[13], line 2
      1 from scipy.cluster.hierarchy import dendrogram
----> 2 dendrogram(Z, labels=cars["brand"].to_list())

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\scipy\cluster\hierarchy\_hierarchy_impl.py:3312, in dendrogram(Z, p, truncate_mode, color_threshold, get_leaves, orientation, labels, count_sort, distance_sort, show_leaf_counts, no_plot, no_labels, leaf_font_size, leaf_rotation, leaf_label_func, show_contracted, link_color_func, ax, above_threshold_color)
   3310         len_labels = labels.shape[0]
   3311     if Z.shape[0] + 1 != len_labels:
-> 3312         raise ValueError("Dimensions of Z and labels must be consistent.")
   3314 _is_valid_linkage(Z, throw=True, name='Z', materialize=True, xp=xp)
   3315 Zs = Z.shape

ValueError: Dimensions of Z and labels must be consistent.
import matplotlib.pyplot as plt
plt.figure(figsize=(10, 4))
dendrogram(Z, labels=names, leaf_rotation=90)
plt.title("Ward dendrogram of 22 car brands")
plt.ylabel("Merge distance")
plt.show()
No description has been provided for this image
  • Leaves at the bottom are brands; each join is a merge
  • The height of a join is how different the two groups were
print(Z[-4:, 2].round(2))
[ 2.3   3.71  4.38 10.99]

Cutting the tree into clusters#

Problem: we still need flat labels. Cut by count or by height.

from scipy.cluster.hierarchy import fcluster
brands["tree3"] = fcluster(Z, t=3, criterion="maxclust")
for g, grp in brands.groupby("tree3"):
    print(g, grp.index.to_list())
1 ['amc', 'buick', 'chevrolet', 'chrysler', 'dodge', 'ford', 'mercury', 'oldsmobile', 'plymouth', 'pontiac', 'saab', 'volvo']
2 ['audi', 'datsun', 'fiat', 'honda', 'mazda', 'opel', 'subaru', 'toyota', 'volkswagen']
3 ['peugeot']
cut = fcluster(Z, t=6, criterion="distance")
print(cut.max(), "clusters")
2 clusters
plt.figure(figsize=(10, 4))
dendrogram(Z, labels=names, leaf_rotation=90, color_threshold=6)
plt.axhline(6, color="grey", linestyle="--")
plt.title("Cutting the brand tree at height 6")
plt.ylabel("Merge distance")
plt.show()
No description has been provided for this image
brands.groupby("tree3")[features].mean().round(1)
mpg horsepower weight acceleration
tree3
1 20.3 121.0 3346.7 14.9
2 29.6 76.9 2211.3 16.2
3 23.7 88.0 2972.8 19.6
from sklearn.metrics import silhouette_score
for k in range(2, 7):
    lab = fcluster(Z, t=k, criterion="maxclust")
    print(k, round(silhouette_score(Xb, lab), 3))
2 0.591
3 0.592
4 0.499
5 0.344
6 0.273

Your turn#

Cut the tree into 4 clusters. Which group splits, and who leaves?

brands["tree4"] = fcluster(Z, t=4, criterion="maxclust")
pd.crosstab(brands["tree3"], brands["tree4"])
tree4 1 2 3 4
tree3
1 3 9 0 0
2 0 0 9 0
3 0 0 0 1
brands[brands["tree4"] == 1].index.to_list()
['buick', 'chrysler', 'pontiac']

Linkage methods change the tree#

Problem: 'closest clusters' can be measured in several ways.

  • single: the closest pair of members
  • complete: the furthest pair of members
  • average: the average over all pairs
  • ward: the merge that adds the least spread
from scipy.cluster.hierarchy import cophenet
from scipy.spatial.distance import pdist
d = pdist(Xb)
for m in ["single", "complete", "average", "ward"]:
    Zm = linkage(Xb, method=m)
    r = cophenet(Zm, d)[0]
    sizes = pd.Series(fcluster(Zm, 5, "maxclust")).value_counts()
    print(m, round(r, 3), sorted(sizes, reverse=True))
single 0.595 [10, 9, 1, 1, 1]
complete 0.81 [9, 5, 4, 3, 1]
average 0.828 [9, 8, 3, 1, 1]
ward 0.794 [9, 6, 3, 3, 1]
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
for ax, m in zip(axes, ["single", "ward"]):
    dendrogram(linkage(Xb, method=m), labels=names, ax=ax,
               leaf_rotation=90)
    ax.set_title(f"{m} linkage")
    ax.set_ylabel("Merge distance")
plt.show()
No description has been provided for this image

Common mistake: setting both n_clusters and distance_threshold

from sklearn.cluster import AgglomerativeClustering
AgglomerativeClustering(n_clusters=3, distance_threshold=6).fit(Xb)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[25], line 2
      1 from sklearn.cluster import AgglomerativeClustering
----> 2 AgglomerativeClustering(n_clusters=3, distance_threshold=6).fit(Xb)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py:1403, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1393     estimator._validate_params()
   1395 with (
   1396     config_context(
   1397         skip_parameter_validation=(
   (...)
   1401     callback_management_context(estimator),
   1402 ):
-> 1403     return fit_method(estimator, *args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\cluster\_agglomerative.py:992, in AgglomerativeClustering.fit(self, X, y)
    974 """Fit the hierarchical clustering from features, or distance matrix.
    975 
    976 Parameters
   (...)
    989     Returns the fitted instance.
    990 """
    991 X = validate_data(self, X, ensure_min_samples=2)
--> 992 return self._fit(X)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\cluster\_agglomerative.py:1011, in AgglomerativeClustering._fit(self, X)
   1008 memory = check_memory(self.memory)
   1010 if not ((self.n_clusters is None) ^ (self.distance_threshold is None)):
-> 1011     raise ValueError(
   1012         "Exactly one of n_clusters and "
   1013         "distance_threshold has to be set, and the other "
   1014         "needs to be None."
   1015     )
   1017 if self.distance_threshold is not None and not self.compute_full_tree:
   1018     raise ValueError(
   1019         "compute_full_tree must be True if distance_threshold is set."
   1020     )

ValueError: Exactly one of n_clusters and distance_threshold has to be set, and the other needs to be None.
cut6 = AgglomerativeClustering(n_clusters=None, distance_threshold=6)
cut6.fit(Xb)
print(cut6.n_clusters_)
2

Clustering every car with scikit-learn#

Problem: 392 cars make an unreadable tree, but the clusters still help.

train = cars["model_year"] < 82
scaler = StandardScaler().fit(cars.loc[train, features])
Xtr = scaler.transform(cars.loc[train, features])
Xnew = scaler.transform(cars.loc[~train, features])
print(len(Xtr), len(Xnew))
362 30
agg = AgglomerativeClustering(n_clusters=3, linkage="ward")
cars["agg"] = 0
raw = agg.fit_predict(Xtr)
w = cars.loc[train, "weight"].groupby(raw).mean().rank().astype(int)
cars.loc[train, "agg"] = [w[c] for c in raw]
cars.loc[train, "agg"].value_counts().sort_index()
agg
1     94
2    168
3    100
Name: count, dtype: int64
from sklearn.metrics import adjusted_rand_score
tr = cars[train]
print(pd.crosstab(tr["agg"], tr["km"]))
print(round(adjusted_rand_score(tr["agg"], tr["km"]), 3))
km    1    2   3
agg             
1    92    2   0
2    27  141   0
3     0    3  97
0.748
plt.scatter(tr["weight"], tr["mpg"], c=tr["agg"], alpha=0.6)
plt.title("Ward clusters of 362 cars")
plt.xlabel("Weight (lb)")
plt.ylabel("Miles per gallon")
plt.show()
No description has been provided for this image
Zc = linkage(Xtr, method="ward")
plt.figure(figsize=(10, 4))
dendrogram(Zc, truncate_mode="lastp", p=12)
plt.title("Top of the tree for 362 cars")
plt.ylabel("Merge distance")
plt.show()
No description has been provided for this image

Your turn#

Re-fit Ward on the training cars with 4 clusters. Which group splits?

agg4 = AgglomerativeClustering(n_clusters=4, linkage="ward")
lab4 = agg4.fit_predict(Xtr)
pd.crosstab(tr["agg"], lab4)
col_0 0 1 2 3
agg
1 94 0 0 0
2 0 0 95 73
3 0 100 0 0

Common mistake: Ward linkage with a non-Euclidean distance

AgglomerativeClustering(n_clusters=3, linkage="ward",
                        metric="manhattan").fit(Xtr)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[33], line 2
      1 AgglomerativeClustering(n_clusters=3, linkage="ward",
----> 2                         metric="manhattan").fit(Xtr)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py:1403, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1393     estimator._validate_params()
   1395 with (
   1396     config_context(
   1397         skip_parameter_validation=(
   (...)
   1401     callback_management_context(estimator),
   1402 ):
-> 1403     return fit_method(estimator, *args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\cluster\_agglomerative.py:992, in AgglomerativeClustering.fit(self, X, y)
    974 """Fit the hierarchical clustering from features, or distance matrix.
    975 
    976 Parameters
   (...)
    989     Returns the fitted instance.
    990 """
    991 X = validate_data(self, X, ensure_min_samples=2)
--> 992 return self._fit(X)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\cluster\_agglomerative.py:1023, in AgglomerativeClustering._fit(self, X)
   1018     raise ValueError(
   1019         "compute_full_tree must be True if distance_threshold is set."
   1020     )
   1022 if self.linkage == "ward" and self.metric not in ("euclidean", "l2"):
-> 1023     raise ValueError(
   1024         f"{self.metric} was provided as metric. Ward can only "
   1025         "work with euclidean distances (i.e. 'euclidean' and 'l2')."
   1026     )
   1028 tree_builder = _TREE_BUILDERS[self.linkage]
   1030 connectivity = self.connectivity

ValueError: manhattan was provided as metric. Ward can only work with euclidean distances (i.e. 'euclidean' and 'l2').
man = AgglomerativeClustering(n_clusters=3, linkage="average",
                              metric="manhattan").fit(Xtr)
pd.Series(man.labels_).value_counts().to_list()
[237, 100, 25]

Common mistake: asking agglomerative clustering to predict

agg.predict(Xnew)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[35], line 1
----> 1 agg.predict(Xnew)

AttributeError: 'AgglomerativeClustering' object has no attribute 'predict'
from sklearn.neighbors import NearestCentroid
nc = NearestCentroid().fit(Xtr, cars.loc[train, "agg"])
cars.loc[~train, "agg"] = nc.predict(Xnew)
cars.loc[~train, "agg"].value_counts().sort_index()
agg
1    22
2     8
Name: count, dtype: int64

Mini-project: do brands cluster by country?#

Our brand tree never saw origin. Does it split by country anyway?

origin = cars.groupby("brand")["origin"].first()
brands["origin"] = origin
pd.crosstab(brands["tree3"], brands["origin"])
origin europe japan usa
tree3
1 2 0 10
2 4 5 0
3 1 0 0
euro = brands[brands["origin"] == "europe"]
euro[["tree3", "mpg", "weight", "acceleration"]].round(1)
tree3 mpg weight acceleration
brand
audi 2 26.7 2556.1 15.9
fiat 2 28.9 2143.1 15.6
opel 2 25.8 2200.2 15.2
peugeot 3 23.7 2972.8 19.6
saab 1 23.9 2625.2 15.2
volkswagen 2 31.8 2045.8 17.4
volvo 1 21.1 3032.7 15.6
from scipy.spatial.distance import squareform
grid = pd.DataFrame(squareform(d), index=names, columns=names)
grid["toyota"].drop("toyota").nsmallest(3).round(2)
mazda    0.48
audi     0.52
fiat     0.62
Name: toyota, dtype: float64
home = brands.groupby("origin")["tree3"].agg(lambda s: s.mode()[0])
fits = (brands["tree3"] == brands["origin"].map(home)).mean()
print(f"{fits:.0%} of brands sit in their country's usual cluster")
86% of brands sit in their country's usual cluster

Recap#

  • Agglomerative clustering merges the closest groups, bottom up
  • A dendrogram shows every merge; big jumps suggest where to cut
  • fcluster cuts by number of clusters or by height
  • Linkage matters: single chains, Ward makes compact groups
  • No predict: use NearestCentroid to place new rows

Challenges#

  1. Keep brands with at least 2 cars. Where do BMW and Cadillac land?
  2. Build the brand tree with complete linkage. Is Peugeot still alone?
  3. Add displacement to the features. Do the American brands still group?

What's next#

Lesson 30: Density-Based Clustering with DBSCAN Find clusters of any shape, and flag the points that fit nowhere.

Data Mining 30: Density-Based Clustering with DBSCAN#

In this lesson you'll be able to:

  • Explain core points, border points and noise
  • Cluster map coordinates with DBSCAN and haversine distance
  • Choose eps with a k-distance plot, then tune min_samples
  • Profile clusters and noise to answer a business question Dataset: 20,640 California districts, 1990 census (data/pandas/housing.csv)

Warm-up: Ward clusters on cars#

Lesson 29 built cluster trees. One quick replay on the cars.

import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import AgglomerativeClustering
cars = pd.read_csv("mpg.csv")
features = ["mpg", "horsepower", "weight", "acceleration"]
cars = cars.dropna(subset=features)
X = StandardScaler().fit_transform(cars[features])
ward = AgglomerativeClustering(n_clusters=3, linkage="ward")
print(sorted(ward.fit_predict(X).tolist().count(c) for c in range(3)))
[95, 122, 175]

Every car got a cluster, even the oddballs. Today: a method that can say 'none'.

Where k-means falls short#

Problem: real groups have odd shapes, and some points belong nowhere.

housing = pd.read_csv("housing.csv")
print(housing.shape)
cols = ["longitude", "latitude", "median_income",
        "median_house_value"]
housing[cols].head(3)
(20640, 10)
longitude latitude median_income median_house_value
0 -122.23 37.88 8.3252 452600.0
1 -122.22 37.86 8.3014 358500.0
2 -122.24 37.85 7.2574 352100.0
top = housing["median_house_value"].max()
print(top, (housing["median_house_value"] == top).sum())
500001.0 965
import matplotlib.pyplot as plt
plt.scatter(housing["longitude"], housing["latitude"], s=1,
            color="grey")
plt.title("20,640 California districts")
plt.xlabel("Longitude")
plt.ylabel("Latitude")
plt.show()
No description has been provided for this image
import numpy as np
from sklearn.cluster import KMeans
where = housing[["latitude", "longitude"]]
km = KMeans(n_clusters=4, n_init=10, random_state=42).fit(where)
plt.scatter(where["longitude"], where["latitude"], s=1,
            c=km.labels_)
plt.title("k-means: four clusters of districts")
plt.xlabel("Longitude")
plt.ylabel("Latitude")
plt.show()
No description has been provided for this image

Quick check: what's wrong with putting a lonely desert district in the LA cluster?

print(sorted(km.labels_.tolist().count(c) for c in range(4)))
[1736, 2188, 7727, 8989]

Your turn#

Try k-means with k = 8. Are any lonely districts left out now?

km8 = KMeans(n_clusters=8, n_init=10, random_state=42).fit(where)
sizes8 = np.bincount(km8.labels_)
print(sorted(sizes8.tolist()), sizes8.sum())
[685, 865, 1114, 1937, 2202, 3160, 4846, 5831] 20640

How DBSCAN thinks: core, border and noise#

Problem: we want clusters to grow only where points are crowded.

  • eps: the radius of each point's neighbourhood
  • min_samples: points needed inside that radius to be a core point
  • Border: near a core point but not core itself; noise: neither
coords = np.radians(housing[["latitude", "longitude"]])
EARTH_KM = 6371
eps = 8 / EARTH_KM
print(round(eps, 5))
0.00126
from sklearn.neighbors import NearestNeighbors
nn = NearestNeighbors(radius=eps, metric="haversine").fit(coords)
rows = [0, housing["longitude"].idxmax()]
found = nn.radius_neighbors(coords.iloc[rows],
                            return_distance=False)
for r, f in zip(rows, found):
    print(housing.loc[r, ["longitude", "latitude"]].to_list(), len(f))
[-122.23, 37.88] 289
[-114.31, 34.19] 1

Common mistake: giving haversine more than two columns

from sklearn.cluster import DBSCAN
extra = np.radians(housing[["latitude", "longitude", "median_income"]])
DBSCAN(eps=eps, min_samples=50, metric="haversine").fit(extra)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[11], line 3
      1 from sklearn.cluster import DBSCAN
      2 extra = np.radians(housing[["latitude", "longitude", "median_income"]])
----> 3 DBSCAN(eps=eps, min_samples=50, metric="haversine").fit(extra)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py:1403, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1393     estimator._validate_params()
   1395 with (
   1396     config_context(
   1397         skip_parameter_validation=(
   (...)
   1401     callback_management_context(estimator),
   1402 ):
-> 1403     return fit_method(estimator, *args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\cluster\_dbscan.py:449, in DBSCAN.fit(self, X, y, sample_weight)
    438         X = sort_graph_by_row_values(X, warn_when_not_sorted=False)
    440 neighbors_model = NearestNeighbors(
    441     radius=self.eps,
    442     algorithm=self.algorithm,
   (...)
    447     n_jobs=self.n_jobs,
    448 )
--> 449 neighbors_model.fit(X)
    450 # This has worst case O(n^2) memory complexity
    451 neighborhoods = neighbors_model.radius_neighbors(X, return_distance=False)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py:1403, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1393     estimator._validate_params()
   1395 with (
   1396     config_context(
   1397         skip_parameter_validation=(
   (...)
   1401     callback_management_context(estimator),
   1402 ):
-> 1403     return fit_method(estimator, *args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\neighbors\_unsupervised.py:179, in NearestNeighbors.fit(self, X, y)
    158 @_fit_context(
    159     # NearestNeighbors.metric is not validated yet
    160     prefer_skip_nested_validation=False
    161 )
    162 def fit(self, X, y=None):
    163     """Fit the nearest neighbors estimator from the training dataset.
    164 
    165     Parameters
   (...)
    177         The fitted nearest neighbors estimator.
    178     """
--> 179     return self._fit(X)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\neighbors\_base.py:673, in NeighborsBase._fit(self, X, y)
    666         raise ValueError(
    667             f'algorithm="{self._fit_method}" does not support 0 < p < 1 for '
    668             "the Minkowski metric. To resolve this problem either "
    669             'set p >= 1 or algorithm="brute".'
    670         )
    672 if self._fit_method == "ball_tree":
--> 673     self._tree = BallTree(
    674         X,
    675         self.leaf_size,
    676         metric=self.effective_metric_,
    677         **self.effective_metric_params_,
    678     )
    679 elif self._fit_method == "kd_tree":
    680     if (
    681         self.effective_metric_ == "minkowski"
    682         and self.effective_metric_params_.get("w") is not None
    683     ):

File sklearn/neighbors/_binary_tree.pxi:912, in sklearn.neighbors._ball_tree.BinaryTree64.__init__()

File sklearn/metrics/_dist_metrics.pyx:2623, in sklearn.metrics._dist_metrics.HaversineDistance64._validate_data()

ValueError: Haversine distance only valid in 2 dimensions
db = DBSCAN(eps=eps, min_samples=50, metric="haversine")
labels = db.fit_predict(coords)
print(labels.max() + 1, "clusters")
print((labels == -1).sum(), "noise districts")
21 clusters
3665 noise districts
core = np.zeros(len(labels), dtype=bool)
core[db.core_sample_indices_] = True
kind = np.where(core, "core", np.where(labels == -1, "noise",
                                       "border"))
pd.Series(kind).value_counts()
core      15775
noise      3665
border     1200
Name: count, dtype: int64
b = np.flatnonzero(kind == "border")[0]
near = nn.radius_neighbors(coords.iloc[[b]], return_distance=False)
print(len(near[0]), labels[b] >= 0)
47 True
la = housing["latitude"].between(33.5, 34.5) & housing[
    "longitude"].between(-119, -117)
for name, colour in [("core", "tab:blue"), ("border", "orange"),
                     ("noise", "black")]:
    pick = la.to_numpy() & (kind == name)
    plt.scatter(housing.loc[pick, "longitude"],
                housing.loc[pick, "latitude"], s=2, c=colour,
                label=name)
plt.title("Core, border and noise around Los Angeles")
plt.xlabel("Longitude")
plt.ylabel("Latitude")
plt.legend()
No description has been provided for this image

Choosing eps with a k-distance plot#

Problem: 8 km was a guess. The data can suggest a radius.

  • For each point, find the distance to its 50th nearest neighbour
  • Sort them and plot: the bend marks where crowds end
nn50 = NearestNeighbors(n_neighbors=50, metric="haversine")
dist, _ = nn50.fit(coords).kneighbors(coords)
kdist = np.sort(dist[:, -1]) * EARTH_KM
print(np.percentile(kdist, [25, 50, 75, 90]).round(1))
[ 2.8  3.6  7.5 17.9]
plt.plot(kdist)
plt.axhline(8, color="red", linestyle="--")
plt.ylim(0, 60)
plt.title("k-distance plot: 50th nearest neighbour")
plt.xlabel("Districts, sorted")
plt.ylabel("Distance to 50th neighbour (km)")
plt.show()
No description has been provided for this image

Quick check: what share of districts would be core with eps = 8 km?

print(round((kdist <= 8).mean(), 3))
0.764

Tuning eps and min_samples#

Problem: small changes can merge cities or shatter them.

def summary(eps_km, min_samples):
    m = DBSCAN(eps=eps_km / EARTH_KM, min_samples=min_samples,
               metric="haversine")
    lab = m.fit_predict(coords)
    return {"eps_km": eps_km, "min_samples": min_samples,
            "clusters": lab.max() + 1,
            "noise": round((lab == -1).mean(), 3)}

Quick check: as eps grows, does noise go up or down?

tests = [summary(e, 50) for e in [3, 5, 8, 12]]
pd.DataFrame(tests)
eps_km min_samples clusters noise
0 3 50 24 0.452
1 5 50 30 0.259
2 8 50 21 0.178
3 12 50 23 0.104
tests = [summary(8, m) for m in [10, 50, 200]]
pd.DataFrame(tests)
eps_km min_samples clusters noise
0 8 10 55 0.053
1 8 50 21 0.178
2 8 200 9 0.395
sweep = pd.DataFrame([summary(e, 50) for e in range(2, 21, 2)])
fig, ax = plt.subplots(1, 2, figsize=(10, 3.5))
sweep.plot(x="eps_km", y="clusters", ax=ax[0], legend=False)
sweep.plot(x="eps_km", y="noise", ax=ax[1], legend=False)
ax[0].set_title("Clusters by eps")
ax[1].set_title("Noise share by eps")
ax[0].set_ylabel("Clusters")
ax[1].set_ylabel("Share of districts")
plt.show()
No description has been provided for this image

Your turn#

With min_samples=20, find an eps that leaves under 5% noise. How many clusters?

pd.DataFrame([summary(e, 20) for e in [8, 12, 15]])
eps_km min_samples clusters noise
0 8 20 42 0.085
1 12 20 25 0.049
2 15 20 17 0.035

Reading the clusters#

Problem: cluster numbers mean nothing until we profile them.

sizes = pd.Series(labels[labels >= 0]).value_counts()
rank = {c: i for i, c in enumerate(sizes.index, start=1)}
housing["cluster"] = [rank.get(c, -1) for c in labels]
housing["cluster"].value_counts().head(6)
cluster
 1    8316
 2    3804
-1    3665
 3    1541
 4     720
 5     362
Name: count, dtype: int64
profile = housing.groupby("cluster").agg(
    districts=("cluster", "size"),
    lat=("latitude", "mean"), lon=("longitude", "mean"),
    income=("median_income", "median"),
    value=("median_house_value", "median"))
profile.loc[[-1, 1, 2, 3, 4, 5]].round(2)
districts lat lon income value
cluster
-1 3665 36.92 -120.04 2.86 109700.0
1 8316 33.98 -118.10 3.71 202900.0
2 3804 37.66 -122.17 4.26 258300.0
3 1541 32.84 -117.13 3.54 168700.0
4 720 38.61 -121.38 3.17 122050.0
5 362 36.78 -119.77 2.68 74300.0

Quick check: using the map centres, which cities are clusters 1 to 5?

big = housing[housing["cluster"].between(1, 5)]
noise = housing[housing["cluster"] == -1]
plt.scatter(noise["longitude"], noise["latitude"], s=1, c="grey")
plt.scatter(big["longitude"], big["latitude"], s=1, c=big["cluster"],
            cmap="tab10")
plt.title("DBSCAN: five biggest clusters, noise in grey")
plt.xlabel("Longitude")
plt.ylabel("Latitude")
plt.show()
No description has been provided for this image

Your turn#

Which three real clusters have the highest typical income?

profile.drop(-1).nlargest(3, "income")[["districts", "lat", "lon",
                                         "income"]].round(2)
districts lat lon income
cluster
2 3804 37.66 -122.17 4.26
11 148 34.43 -119.74 4.08
14 101 38.31 -122.00 4.04

Watch out: -1 is a label for noise, not a cluster

print(housing["cluster"].nunique())
print(housing.loc[housing["cluster"] >= 0, "cluster"].nunique())
22
21
real = profile.drop(-1)
cheap = real.nsmallest(3, "value")
cheap[["districts", "lat", "lon", "value"]].round(2)
districts lat lon value
cluster
9 223 35.37 -119.01 70700.0
17 76 39.13 -121.60 71750.0
5 362 36.78 -119.77 74300.0

Placing new districts#

Problem: a new district arrives. Which cluster, or is it noise?

Common mistake: expecting DBSCAN to have a predict method

db.predict(coords.iloc[:3])
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[30], line 1
----> 1 db.predict(coords.iloc[:3])

AttributeError: 'DBSCAN' object has no attribute 'predict'

A new point joins a cluster only if it lies within eps of a core point.

seen = housing.sample(frac=0.95, random_state=42).index
new = housing.index.difference(seen)
db2 = DBSCAN(eps=eps, min_samples=50, metric="haversine")
db2.fit(coords.loc[seen])
print(len(seen), len(new), db2.labels_.max() + 1)
19608 1032 20
core_idx = db2.core_sample_indices_
core_xy = coords.loc[seen].iloc[core_idx]
core_lab = db2.labels_[core_idx]
nn1 = NearestNeighbors(n_neighbors=1, metric="haversine")
d, i = nn1.fit(core_xy).kneighbors(coords.loc[new])
new_lab = np.where(d[:, 0] <= eps, core_lab[i[:, 0]], -1)
print(round((new_lab == -1).mean(), 3))
0.181
full_noise = labels[new.to_numpy()] == -1
print(round(((new_lab == -1) == full_noise).mean(), 3))
0.986

Your turn#

Use the core-point rule on the most easterly district. Cluster or noise?

east = housing["longitude"].idxmax()
d_e, _ = nn1.kneighbors(coords.loc[[east]])
km_away = d_e[0, 0] * EARTH_KM
print(round(km_away, 1), "noise" if km_away > 8 else "cluster")
257.5 noise

Mini-project: are crowded districts pricier?#

Do districts inside dense clusters cost more than the scattered noise?

housing["setting"] = np.where(housing["cluster"] == -1,
                              "scattered", "clustered")
housing.groupby("setting")[["median_income",
                            "median_house_value"]].median()
median_income median_house_value
setting
clustered 3.6995 193900.0
scattered 2.8562 109700.0
pd.crosstab(housing["setting"], housing["ocean_proximity"],
            normalize="index").round(2)
ocean_proximity <1H OCEAN INLAND ISLAND NEAR BAY NEAR OCEAN
setting
clustered 0.50 0.22 0.0 0.13 0.14
scattered 0.16 0.76 0.0 0.00 0.08
housing.boxplot(column="median_house_value", by="setting",
                showfliers=False)
plt.title("House value: clustered vs scattered districts")
plt.suptitle("")
plt.xlabel("")
plt.ylabel("Median house value ($)")
plt.show()
No description has been provided for this image
inland = housing[housing["ocean_proximity"] == "INLAND"]
v = inland.groupby("setting")["median_house_value"].median()
gap = v["clustered"] / v["scattered"] - 1
print(v.round(), f"premium: {gap:.0%}", sep="\n")
setting
clustered    117800.0
scattered     95300.0
Name: median_house_value, dtype: float64
premium: 24%

Recap#

  • DBSCAN grows clusters from core points in crowded areas
  • Points near no crowd are noise, labelled -1; no k needed
  • Map data: haversine distance, radians, latitude first
  • Pick eps from a k-distance plot; min_samples sets the crowd size
  • Profile noise separately; never treat -1 as a cluster

Challenges#

  1. Run DBSCAN with eps = 3 km. Does Los Angeles split into several clusters?
  2. Scale median_income and median_house_value and cluster them with DBSCAN.
  3. Profile the noise districts by housing_median_age. Are they older or newer?

What's next#

Lesson 31: Cluster Evaluation Metrics and Interpretation Score and compare clusterings, with and without known labels.

Data Mining 31: Cluster Evaluation Metrics and Interpretation#

In this lesson you'll be able to:

  • Read inertia and the elbow, and know their limits
  • Compute and plot the silhouette score, by hand and with sklearn
  • Compare k with Davies-Bouldin and Calinski-Harabasz
  • Check clusters against known labels with the adjusted Rand index
  • Turn cluster profiles into plain-English descriptions Dataset: 178 Italian wines, 13 chemical measurements (data/mining/wine.csv)

Warm-up: DBSCAN from lesson 30#

Lesson 30 grouped points by density. Here it is on our wines.

import pandas as pd
from sklearn.preprocessing import StandardScaler
w = pd.read_csv("wine.csv")
features = w.columns.drop("cultivar")
X = StandardScaler().fit_transform(w[features])
print(X.shape)
(178, 13)
from sklearn.cluster import DBSCAN
db = DBSCAN(eps=2.2, min_samples=5).fit_predict(X)
pd.Series(db).value_counts()
 0    86
-1    55
 1    37
Name: count, dtype: int64

Inertia and the elbow: how tight are the clusters?#

Problem: we need a number that says how compact each cluster is.

  • Inertia: total squared distance from each row to its centre
  • Lower means tighter clusters, but it always falls as k grows
from sklearn.cluster import KMeans
km3 = KMeans(n_clusters=3, n_init=10, random_state=42).fit(X)
print(round(km3.inertia_, 1))
1277.9
centres = km3.cluster_centers_[km3.labels_]
print(round(((X - centres) ** 2).sum(), 1))
1277.9
import numpy as np
order = km3.cluster_centers_[:, features.get_loc("proline")]
lab = order.argsort().argsort()[km3.labels_]
np.bincount(lab)
array([65, 51, 62])
from sklearn.decomposition import PCA
xy = PCA(n_components=2).fit_transform(X)
plt.scatter(xy[:, 0], xy[:, 1], c=lab, cmap="viridis", alpha=0.7)
plt.title("Three k-means clusters, seen through PCA")
plt.xlabel("First principal component")
plt.ylabel("Second principal component")
plt.show()
No description has been provided for this image
inertia = {}
for k in range(1, 9):
    km = KMeans(n_clusters=k, n_init=10, random_state=42).fit(X)
    inertia[k] = km.inertia_
inertia = pd.Series(inertia).round(1)
inertia
1    2314.0
2    1658.8
3    1277.9
4    1175.4
5    1109.5
6    1046.0
7     981.6
8     935.2
dtype: float64
import matplotlib.pyplot as plt
inertia.plot(marker="o")
plt.title("Elbow chart for the wines")
plt.xlabel("Number of clusters k")
plt.ylabel("Inertia")
plt.show()
No description has been provided for this image

Quick check: if inertia always falls, why not pick k = 178?

km_all = KMeans(n_clusters=178, n_init=1, random_state=42).fit(X)
print(km_all.inertia_)
0.0

Silhouette: tight inside, far from the neighbours#

Problem: inertia ignores how far apart the clusters are.

  • a: average distance to the other members of my cluster
  • b: average distance to the nearest other cluster
  • silhouette = (b - a) / max(a, b): near 1 good, 0 on the border, below 0 misplaced
from sklearn.metrics import pairwise_distances
d = pairwise_distances(X[:1], X)[0]
own = lab == lab[0]
a = d[own].sum() / (own.sum() - 1)
others = [d[lab == c].mean() for c in range(3) if c != lab[0]]
b = min(others)
print(round(a, 3), round(b, 3), round((b - a) / max(a, b), 3))
3.157 5.723 0.448
from sklearn.metrics import silhouette_samples, silhouette_score
sil = silhouette_samples(X, lab)
print(round(sil[0], 3))
print(round(silhouette_score(X, lab), 3))
0.448
0.285
per = pd.Series(sil).groupby(lab).agg(["mean", "min", "size"])
per.round(3)
mean min size
0 0.177 -0.023 65
1 0.351 0.061 51
2 0.343 0.035 62
fig, ax = plt.subplots(figsize=(7, 4))
y = 0
for c in range(3):
    vals = np.sort(sil[lab == c])
    ax.barh(range(y, y + len(vals)), vals, height=1,
            label=f"Cluster {c}")
    y += len(vals) + 5
ax.axvline(sil.mean(), color="black", linestyle="--")
ax.set_title("Silhouette plot, k-means with k = 3")
ax.set_xlabel("Silhouette value")
ax.set_ylabel("Wines, sorted within cluster")
ax.legend()
No description has been provided for this image

Your turn: find the wine with the lowest silhouette. Which cluster is it nearer to?

worst = sil.argmin()
dw = pairwise_distances(X[[worst]], X)[0]
means = pd.Series(dw).groupby(lab).mean().round(3)
print(worst, lab[worst], round(sil[worst], 3))
means
71 0 -0.023
0    4.703
1    6.500
2    4.668
dtype: float64

Common mistake: asking for a silhouette with only one cluster

one = KMeans(n_clusters=1, n_init=10, random_state=42).fit_predict(X)
silhouette_score(X, one)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[16], line 2
      1 one = KMeans(n_clusters=1, n_init=10, random_state=42).fit_predict(X)
----> 2 silhouette_score(X, one)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_param_validation.py:218, in validate_params.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    212 try:
    213     with config_context(
    214         skip_parameter_validation=(
    215             prefer_skip_nested_validation or global_skip_validation
    216         )
    217     ):
--> 218         return func(*args, **kwargs)
    219 except InvalidParameterError as e:
    220     # When the function is just a wrapper around an estimator, we allow
    221     # the function to delegate validation to the estimator, but we replace
    222     # the name of the estimator by the name of the function in the error
    223     # message to avoid confusion.
    224     msg = re.sub(
    225         r"parameter of \w+ must be",
    226         f"parameter of {func.__qualname__} must be",
    227         str(e),
    228     )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\metrics\cluster\_unsupervised.py:146, in silhouette_score(X, labels, metric, sample_size, random_state, **kwds)
    144     else:
    145         X, labels = X[indices], labels[indices]
--> 146 return float(np.mean(silhouette_samples(X, labels, metric=metric, **kwds)))

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_param_validation.py:191, in validate_params.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    189 global_skip_validation = get_config()["skip_parameter_validation"]
    190 if global_skip_validation:
--> 191     return func(*args, **kwargs)
    193 func_sig = signature(func)
    195 # Map *args/**kwargs to the function signature

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\metrics\cluster\_unsupervised.py:304, in silhouette_samples(X, labels, metric, **kwds)
    302 n_samples = len(labels)
    303 label_freqs = np.bincount(labels)
--> 304 check_number_of_labels(len(le.classes_), n_samples)
    306 kwds["metric"] = metric
    307 reduce_func = functools.partial(
    308     _silhouette_reduce, labels=labels, label_freqs=label_freqs
    309 )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\metrics\cluster\_unsupervised.py:43, in check_number_of_labels(n_labels, n_samples)
     32 """Check that number of labels are valid.
     33 
     34 Parameters
   (...)
     40     Number of samples.
     41 """
     42 if not 1 < n_labels < n_samples:
---> 43     raise ValueError(
     44         "Number of labels is %d. Valid values are 2 to n_samples - 1 (inclusive)"
     45         % n_labels
     46     )

ValueError: Number of labels is 1. Valid values are 2 to n_samples - 1 (inclusive)
sil_k = {}
for k in range(2, 9):
    labels_k = KMeans(n_clusters=k, n_init=10,
                      random_state=42).fit_predict(X)
    sil_k[k] = silhouette_score(X, labels_k)
sil_k = pd.Series(sil_k).round(3)
sil_k
2    0.259
3    0.285
4    0.260
5    0.202
6    0.237
7    0.204
8    0.157
dtype: float64
sil_k.plot(marker="o")
plt.title("Average silhouette by k")
plt.xlabel("Number of clusters k")
plt.ylabel("Average silhouette")
plt.show()
No description has been provided for this image

Two more referees: Davies-Bouldin and Calinski-Harabasz#

Problem: one score can be fooled, so we ask for a second opinion.

  • Davies-Bouldin: how much each cluster resembles its closest rival. Lower is better
  • Calinski-Harabasz: spread between clusters over spread within. Higher is better
from sklearn.metrics import davies_bouldin_score
from sklearn.metrics import calinski_harabasz_score
print(round(davies_bouldin_score(X, lab), 3))
print(round(calinski_harabasz_score(X, lab), 1))
1.389
70.9

Your turn: build one table of all three scores for k = 2 to 6. Which k does each score prefer?

rows = []
for k in range(2, 7):
    lk = KMeans(n_clusters=k, n_init=10,
                random_state=42).fit_predict(X)
    rows.append({"k": k, "silhouette": silhouette_score(X, lk),
                 "davies_bouldin": davies_bouldin_score(X, lk),
                 "calinski": calinski_harabasz_score(X, lk)})
scores = pd.DataFrame(rows).set_index("k").round(3)
scores
silhouette davies_bouldin calinski
k
2 0.259 1.526 69.523
3 0.285 1.389 70.940
4 0.260 1.797 56.181
5 0.202 1.808 46.952
6 0.237 1.554 41.701
print("silhouette:", scores["silhouette"].idxmax())
print("davies_bouldin:", scores["davies_bouldin"].idxmin())
print("calinski:", scores["calinski"].idxmax())
silhouette: 3
davies_bouldin: 3
calinski: 3

Common mistake: scoring labels from a sample against the full table

idx = w.sample(100, random_state=42).index
km_part = KMeans(n_clusters=3, n_init=10, random_state=42)
part = km_part.fit_predict(X[idx])
davies_bouldin_score(X, part)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[22], line 4
      2 km_part = KMeans(n_clusters=3, n_init=10, random_state=42)
      3 part = km_part.fit_predict(X[idx])
----> 4 davies_bouldin_score(X, part)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_param_validation.py:218, in validate_params.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    212 try:
    213     with config_context(
    214         skip_parameter_validation=(
    215             prefer_skip_nested_validation or global_skip_validation
    216         )
    217     ):
--> 218         return func(*args, **kwargs)
    219 except InvalidParameterError as e:
    220     # When the function is just a wrapper around an estimator, we allow
    221     # the function to delegate validation to the estimator, but we replace
    222     # the name of the estimator by the name of the function in the error
    223     # message to avoid confusion.
    224     msg = re.sub(
    225         r"parameter of \w+ must be",
    226         f"parameter of {func.__qualname__} must be",
    227         str(e),
    228     )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\metrics\cluster\_unsupervised.py:458, in davies_bouldin_score(X, labels)
    414 """Compute the Davies-Bouldin score.
    415 
    416 The score is defined as the average similarity measure of each cluster with
   (...)
    455 0.12...
    456 """
    457 xp, _, device_ = get_namespace_and_device(X, labels)
--> 458 X, labels = check_X_y(X, labels)
    459 le = LabelEncoder()
    460 labels = le.fit_transform(labels)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:1349, in check_X_y(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, ensure_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, estimator)
   1330 X = check_array(
   1331     X,
   1332     accept_sparse=accept_sparse,
   (...)
   1344     input_name="X",
   1345 )
   1347 y = _check_y(y, multi_output=multi_output, y_numeric=y_numeric, estimator=estimator)
-> 1349 check_consistent_length(X, y)
   1351 return X, y

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:458, in check_consistent_length(*arrays)
    456 lengths = [_num_samples(X) for X in arrays if X is not None]
    457 if len(set(lengths)) > 1:
--> 458     raise ValueError(
    459         "Found input variables with inconsistent numbers of samples: %r"
    460         % [int(l) for l in lengths]
    461     )

ValueError: Found input variables with inconsistent numbers of samples: [178, 100]
full = km_part.predict(X)
print(len(full), round(davies_bouldin_score(X, full), 3))
178 1.389

When labels exist: the adjusted Rand index#

Problem: the wines do have a cultivar column. Do clusters find it?

truth = w["cultivar"].str[-1].astype(int)
pd.crosstab(truth, lab, rownames=["cultivar"],
            colnames=["cluster"])
cluster 0 1 2
cultivar
0 0 0 59
1 65 3 3
2 0 48 0

Quick check: cluster numbers are arbitrary. Will plain accuracy be fair here?

from sklearn.metrics import accuracy_score, adjusted_rand_score
print(round(accuracy_score(truth, lab), 3))
print(round(adjusted_rand_score(truth, lab), 3))
0.017
0.897
renamed = (lab + 1) % 3
print(round(accuracy_score(truth, renamed), 3))
print(round(adjusted_rand_score(truth, renamed), 3))
0.966
0.897
mixed = pd.Series(lab).sample(frac=1, random_state=42)
shuffled = mixed.to_numpy()
print(round(adjusted_rand_score(truth, shuffled), 3))
-0.006

Adjusted Rand: 1 is a perfect match, 0 is no better than random.

Interpreting clusters: from numbers to words#

Problem: a score says the groups are distinct, not what they are.

cols = ["alcohol", "flavanoids", "color_intensity", "hue",
        "proline"]
w.groupby(lab)[cols].mean().round(2)
alcohol flavanoids color_intensity hue proline
0 12.25 2.05 2.97 1.06 510.17
1 13.13 0.82 7.23 0.69 619.06
2 13.68 3.00 5.45 1.07 1100.23
z = pd.DataFrame(X, columns=features).groupby(lab).mean()
import seaborn as sns
sns.heatmap(z.T.round(1), annot=True, cmap="RdBu_r", center=0)
plt.title("Cluster profiles in standard deviations")
plt.xlabel("Cluster")
plt.show()
No description has been provided for this image
w.assign(cluster=lab).boxplot(column="proline", by="cluster")
plt.suptitle("")
plt.title("Proline by cluster")
plt.xlabel("Cluster")
plt.ylabel("Proline")
plt.show()
No description has been provided for this image
for c in z.index:
    top = z.loc[c].abs().nlargest(3).index
    print(c, list(top))
0 ['alcohol', 'color_intensity', 'proline']
1 ['od280/od315_of_diluted_wines', 'flavanoids', 'hue']
2 ['proline', 'flavanoids', 'total_phenols']

Your turn: give each cluster a short plain-English name.

names = {0: "light, low alcohol", 1: "deep colour, low flavanoids",
         2: "rich, high proline"}
pd.Series(lab).map(names).value_counts()
light, low alcohol             65
rich, high proline             62
deep colour, low flavanoids    51
Name: count, dtype: int64

Quick check: which cultivar does each named cluster match?

named = pd.Series(lab, name="cluster").map(names)
pd.crosstab(named, w["cultivar"])
cultivar class_0 class_1 class_2
cluster
deep colour, low flavanoids 0 3 48
light, low alcohol 0 65 0
rich, high proline 59 3 0

Mini-project: which clustering should we trust?#

DBSCAN, hierarchical or k-means: grade all three on the same wines.

from sklearn.cluster import AgglomerativeClustering
ward = AgglomerativeClustering(n_clusters=3).fit_predict(X)
methods = {"DBSCAN": db, "Hierarchical": ward, "k-means": lab}

DBSCAN labels noise as -1. Score only the clustered wines, and report the noise share.

rows = []
for name, l in methods.items():
    keep = l != -1
    rows.append({"method": name, "clusters": len(set(l[keep])),
                 "noise_%": round(100 * (~keep).mean(), 1),
                 "silhouette": silhouette_score(X[keep], l[keep]),
                 "ARI": adjusted_rand_score(truth[keep], l[keep])})
report = pd.DataFrame(rows).set_index("method").round(3)
report
clusters noise_% silhouette ARI
method
DBSCAN 2 30.9 0.348 0.525
Hierarchical 3 0.0 0.277 0.790
k-means 3 0.0 0.285 0.897
report[["silhouette", "ARI"]].plot(kind="bar", rot=0)
plt.title("Grading three clusterings of the wines")
plt.xlabel("Method")
plt.ylabel("Score")
plt.show()
No description has been provided for this image

Finding: k-means with k = 3 is the one to trust: every wine clustered, ARI 0.897. DBSCAN's higher silhouette is bought by discarding 31% of the wines as noise.

Recap#

  • Inertia measures tightness but always falls as k grows
  • Silhouette compares distance inside (a) with the nearest neighbour (b)
  • Davies-Bouldin: lower is better. Calinski-Harabasz: higher is better
  • Adjusted Rand checks clusters against true labels, ignoring names
  • A profile turns clusters into words; the scores only grade them

Challenges#

  1. Repeat the score table using only alcohol, flavanoids and proline. Does k = 3 still win?
  2. Score k-means on the unscaled wines. What happens to the adjusted Rand index?
  3. Try DBSCAN with eps from 2.0 to 3.0. Which setting gives the best silhouette?

What's next#

Lesson 32: Hands-on Clustering Project: Socio-Economic Segmentation Cluster the world's countries by wealth, health and progress.

Data Mining 32: Hands-on Clustering Project: Socio-Economic Segmentation#

In this lesson you'll be able to:

  • Turn a long yearly table into one row of features per country
  • Engineer wealth, health and progress features, and log skewed money
  • Choose k with scores, then make the business call
  • Profile, name and map the segments
  • Follow segments over time to answer a real question Dataset: Gapminder, 142 countries every 5 years from 1952 to 2007 (data/pandas/gapminder.csv)

Warm-up: grading clusters, from lesson 31#

A first, naive clustering of the 2007 countries, and its silhouette.

import pandas as pd
import numpy as np
g = pd.read_csv("gapminder.csv")
g7 = g[g["year"] == 2007]
print(g.shape, g7.shape)
(1704, 6) (142, 6)
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
Xq = StandardScaler().fit_transform(g7[["lifeExp", "gdpPercap"]])
km = KMeans(n_clusters=3, n_init=10, random_state=42)
print(round(silhouette_score(Xq, km.fit_predict(Xq)), 3))
0.594

Frame the question and meet the data#

Problem: which countries share a development path, not just a snapshot?

g.head(3)
country year pop continent lifeExp gdpPercap
0 Afghanistan 1952 8425333.0 Asia 28.801 779.445314
1 Afghanistan 1957 9240934.0 Asia 30.332 820.853030
2 Afghanistan 1962 10267083.0 Asia 31.997 853.100710
print(g["country"].nunique(), g["year"].nunique())
print(g["year"].min(), g["year"].max())
142 12
1952 2007
g7[["lifeExp", "gdpPercap", "pop"]].describe().round(1)
lifeExp gdpPercap pop
count 142.0 142.0 1.420000e+02
mean 67.0 11680.1 4.402122e+07
std 12.1 12859.9 1.476214e+08
min 39.6 277.6 1.995790e+05
25% 57.2 1624.8 4.508034e+06
50% 71.9 6124.4 1.051753e+07
75% 76.4 18008.8 3.121004e+07
max 82.6 49357.2 1.318683e+09
import matplotlib.pyplot as plt
g7["gdpPercap"].plot(kind="hist", bins=30)
plt.title("Income per person, 2007")
plt.xlabel("GDP per person (US dollars)")
plt.ylabel("Number of countries")
plt.show()
No description has been provided for this image

Common mistake: passing the whole table, text columns and all, to the scaler

StandardScaler().fit_transform(g7)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_26132\450071016.py in ?()
----> 1 StandardScaler().fit_transform(g7)

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_set_output.py in ?(self, X, *args, **kwargs)
    325     @wraps(f)
    326     def wrapped(self, X, *args, **kwargs):
--> 327         data_to_wrap = f(self, X, *args, **kwargs)
    328         if isinstance(data_to_wrap, tuple):
    329             # only wrap the first output for cross decomposition
    330             return_tuple = (

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py in ?(self, X, y, **fit_params)
    965                 )
    966 
    967         if y is None:
    968             # fit method of arity 1 (unsupervised transformation)
--> 969             return self.fit(X, **fit_params).transform(X)
    970         else:
    971             # fit method of arity 2 (supervised transformation)
    972             return self.fit(X, y, **fit_params).transform(X)

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\preprocessing\_data.py in ?(self, X, y, sample_weight)
    924             Fitted scaler.
    925         """
    926         # Reset internal state before fitting
    927         self._reset()
--> 928         return self.partial_fit(X, y, sample_weight)

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py in ?(estimator, *args, **kwargs)
   1399                     )
   1400                 ),
   1401                 callback_management_context(estimator),
   1402             ):
-> 1403                 return fit_method(estimator, *args, **kwargs)

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\preprocessing\_data.py in ?(self, X, y, sample_weight)
    961             Fitted scaler.
    962         """
    963         xp, _, X_device = get_namespace_and_device(X)
    964         first_call = not hasattr(self, "n_samples_seen_")
--> 965         X = validate_data(
    966             self,
    967             X,
    968             accept_sparse=("csr", "csc"),

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py in ?(_estimator, X, y, reset, validate_separately, skip_check_array, **check_params)
   3037             out = y
   3038         else:
   3039             out = X, y
   3040     elif not no_val_X and no_val_y:
-> 3041         out = check_array(X, input_name="X", **check_params)
   3042     elif no_val_X and not no_val_y:
   3043         out = _check_y(y, **check_params)
   3044     else:

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py in ?(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, ensure_all_finite, ensure_non_negative, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)
   1035                         )
   1036                     array = xp.astype(array, dtype, copy=False)
   1037                 else:
   1038                     array = _asarray_with_order(array, order=order, dtype=dtype, xp=xp)
-> 1039             except ComplexWarning as complex_warning:
   1040                 raise ValueError(
   1041                     "Complex data not supported\n{}\n".format(array)
   1042                 ) from complex_warning

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_array_api.py in ?(array, dtype, order, copy, xp, device)
    979         # Use NumPy API to support order
    980         if copy is True:
    981             array = numpy.array(array, order=order, dtype=dtype)
    982         else:
--> 983             array = numpy.asarray(array, order=order, dtype=dtype)
    984 
    985         # At this point array is a NumPy ndarray. We convert it to an array
    986         # container that is consistent with the input's namespace.

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\generic.py in ?(self, dtype, copy)
   2023             )
   2024         values = self._values
   2025         if copy is None:
   2026             # Note: branch avoids `copy=None` for NumPy 1.x support
-> 2027             arr = np.asarray(values, dtype=dtype)
   2028         else:
   2029             arr = np.array(values, dtype=dtype, copy=copy)
   2030 

ValueError: could not convert string to float: 'Afghanistan'
num = g7.select_dtypes("number").drop(columns="year")
print(num.columns.tolist())
['pop', 'lifeExp', 'gdpPercap']

Engineer features: wealth, health and progress#

Problem: the data is long, one row per country per year. We need one row per country.

  • Wealth: log of GDP per person in 2007
  • Health: life expectancy in 2007
  • Progress: life years gained and GDP growth since 1952

Common mistake: pivoting on a column that is not unique

g.pivot(index="continent", columns="year", values="lifeExp")
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[10], line 1
----> 1 g.pivot(index="continent", columns="year", values="lifeExp")

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\frame.py:10988, in DataFrame.pivot(self, columns, index, values)
  10981 @Substitution("")
  10982 @Appender(_shared_docs["pivot"])
  10983 def pivot(
  10984     self, *, columns, index=lib.no_default, values=lib.no_default
  10985 ) -> DataFrame:
  10986     from pandas.core.reshape.pivot import pivot
> 10988     return pivot(self, index=index, columns=columns, values=values)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\reshape\pivot.py:910, in pivot(data, columns, index, values)
    905         indexed = data._constructor_sliced(data[values]._values, index=multiindex)
    906 # error: Argument 1 to "unstack" of "DataFrame" has incompatible type "Union
    907 # [List[Any], ExtensionArray, ndarray[Any, Any], Index, Series]"; expected
    908 # "Hashable"
    909 # unstack with a MultiIndex returns a DataFrame
--> 910 result = cast("DataFrame", indexed.unstack(columns_listlike))  # type: ignore[arg-type]
    911 result.index.names = [
    912     name if name is not lib.no_default else None for name in result.index.names
    913 ]
    915 return result

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\series.py:4501, in Series.unstack(self, level, fill_value, sort)
   4451 """
   4452 Unstack, also known as pivot, Series with MultiIndex to produce DataFrame.
   4453 
   (...)
   4497 b    2    4
   4498 """
   4499 from pandas.core.reshape.reshape import unstack
-> 4501 return unstack(self, level, fill_value, sort)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\reshape\reshape.py:596, in unstack(obj, level, fill_value, sort)
    594 if is_1d_only_ea_dtype(obj.dtype):
    595     return _unstack_extension_series(obj, level, fill_value, sort=sort)
--> 596 unstacker = _Unstacker(
    597     obj.index, level=level, constructor=obj._constructor_expanddim, sort=sort
    598 )
    599 return unstacker.get_result(obj, value_columns=None, fill_value=fill_value)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\reshape\reshape.py:176, in _Unstacker.__init__(self, index, level, constructor, sort)
    168     if num_cells > np.iinfo(np.int32).max:
    169         warnings.warn(
    170             f"The following operation may generate {num_cells} cells "
    171             f"in the resulting pandas object.",
    172             PerformanceWarning,
    173             stacklevel=find_stack_level(),
    174         )
--> 176 self._make_selectors()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\reshape\reshape.py:232, in _Unstacker._make_selectors(self)
    229 mask.put(selector, True)
    231 if mask.sum() < len(self.index):
--> 232     raise ValueError("Index contains duplicate entries, cannot reshape")
    234 self.group_index = comp_index
    235 self.mask = mask

ValueError: Index contains duplicate entries, cannot reshape
life = g.pivot(index="country", columns="year", values="lifeExp")
life.iloc[:3, [0, -1]]
year 1952 2007
country
Afghanistan 28.801 43.828
Albania 55.230 76.423
Algeria 43.077 72.301
gdp = g.pivot(index="country", columns="year", values="gdpPercap")
print(life.shape, gdp.shape)
(142, 12) (142, 12)
four = ["China", "India", "Nigeria", "United States"]
life.loc[four].T.plot(marker=".")
plt.title("Life expectancy, 1952 to 2007")
plt.xlabel("Year")
plt.ylabel("Life expectancy (years)")
plt.show()
No description has been provided for this image
feats = pd.DataFrame({
    "life_2007": life[2007],
    "log_gdp_2007": np.log10(gdp[2007]),
    "life_gain": life[2007] - life[1952],
    "gdp_growth": np.log10(gdp[2007] / gdp[1952]),
})
feats.describe().round(2)
life_2007 log_gdp_2007 life_gain gdp_growth
count 142.00 142.00 142.00 142.00
mean 67.01 3.74 17.95 0.45
std 12.07 0.59 8.12 0.34
min 39.61 2.44 -4.96 -0.45
25% 57.16 3.21 11.91 0.24
50% 71.94 3.79 18.39 0.46
75% 76.41 4.26 24.31 0.64
max 82.60 4.69 38.06 1.51

Quick check: why use the log of GDP rather than GDP itself?

print(round(gdp[2007].skew(), 2))
print(round(feats["log_gdp_2007"].skew(), 2))
1.22
-0.15
feats.nsmallest(4, "life_gain").round(2)
life_2007 log_gdp_2007 life_gain gdp_growth
country
Zimbabwe 43.49 2.67 -4.96 0.06
Swaziland 39.61 3.65 -1.79 0.59
Zambia 42.38 3.10 0.35 0.04
Lesotho 42.59 3.20 0.45 0.72

Your turn: which three countries grew their income the most since 1952?

fast = feats.nlargest(3, "gdp_growth")
fast["times_richer"] = 10 ** fast["gdp_growth"]
fast.round(2)
life_2007 log_gdp_2007 life_gain gdp_growth times_richer
country
Equatorial Guinea 51.58 4.08 17.10 1.51 32.36
Taiwan 78.40 4.46 19.90 1.38 23.79
Korea, Rep. 78.62 4.37 31.17 1.36 22.66

Choose k: scores advise, the business decides#

Problem: how many segments should we report?

X = StandardScaler().fit_transform(feats)
print(X.shape, X.mean().round(2), X.std().round(2))
(142, 4) 0.0 1.0
from sklearn.metrics import davies_bouldin_score
rows = []
for k in range(2, 8):
    lk = KMeans(n_clusters=k, n_init=10,
                random_state=42).fit_predict(X)
    rows.append({"k": k, "silhouette": silhouette_score(X, lk),
                 "davies_bouldin": davies_bouldin_score(X, lk),
                 "smallest": np.bincount(lk).min()})
scores = pd.DataFrame(rows).set_index("k").round(3)
scores
silhouette davies_bouldin smallest
k
2 0.414 0.964 52
3 0.378 0.965 43
4 0.344 1.014 26
5 0.357 0.984 11
6 0.337 1.025 11
7 0.326 1.044 4
scores["silhouette"].plot(marker="o")
plt.title("Silhouette by number of segments")
plt.xlabel("Number of clusters k")
plt.ylabel("Average silhouette")
plt.show()
No description has been provided for this image
inertia = {}
for k in range(1, 8):
    km = KMeans(n_clusters=k, n_init=10, random_state=42).fit(X)
    inertia[k] = km.inertia_
pd.Series(inertia).plot(marker="o")
plt.title("Elbow chart for the countries")
plt.xlabel("Number of clusters k")
plt.ylabel("Inertia")
plt.show()
No description has been provided for this image

Quick check: the scores prefer k = 2. Should we report two segments?

l2 = KMeans(n_clusters=2, n_init=10, random_state=42).fit_predict(X)
feats.groupby(l2).median().round(2)
life_2007 log_gdp_2007 life_gain gdp_growth
0 74.92 4.08 19.57 0.58
1 53.53 3.08 15.10 0.18
l3 = KMeans(n_clusters=3, n_init=10, random_state=42).fit_predict(X)
pd.crosstab(l2, l3, rownames=["k=2"], colnames=["k=3"])
k=3 0 1 2
k=2
0 41 0 49
1 9 43 0

Two clusters just says rich and poor. k = 3 costs a little silhouette but adds a story.

Fit, profile and name the segments#

Problem: turn cluster numbers into segments people can use.

km3 = KMeans(n_clusters=3, n_init=10, random_state=42).fit(X)
wealth = km3.cluster_centers_[:, feats.columns.get_loc("log_gdp_2007")]
seg = wealth.argsort().argsort()[km3.labels_]
np.bincount(seg)
array([43, 50, 49])
prof = feats.groupby(seg).median()
prof["gdp_2007"] = 10 ** prof["log_gdp_2007"]
prof.round(2)
life_2007 log_gdp_2007 life_gain gdp_growth gdp_2007
0 51.54 3.04 12.72 0.16 1107.48
1 72.35 3.77 26.44 0.48 5831.76
2 78.40 4.41 12.83 0.64 25523.28

Quick check: why profile with the median rather than the mean?

raw = gdp[2007].groupby(seg)
pd.DataFrame({"mean": raw.mean(), "median": raw.median()}).round(0)
mean median
0 1704.0 1107.0
1 7884.0 5833.0
2 24308.0 25523.0
import seaborn as sns
centres = pd.DataFrame(km3.cluster_centers_, columns=feats.columns)
centres.index = wealth.argsort().argsort()
sns.heatmap(centres.sort_index().round(2), annot=True, cmap="RdBu_r",
            center=0)
plt.title("Segment centres in standard deviations")
plt.ylabel("Segment")
plt.show()
No description has been provided for this image

Your turn: give each segment a short, honest name.

names = {0: "Struggling", 1: "Health risers", 2: "Prosperous"}
feats["segment"] = pd.Series(seg, index=feats.index).map(names)
feats["segment"].value_counts()
segment
Health risers    50
Prosperous       49
Struggling       43
Name: count, dtype: int64
pop07 = g7.set_index("country")["pop"]
for name, grp in feats.groupby("segment"):
    big = pop07[grp.index].nlargest(4).index
    print(name, list(big))
Health risers ['China', 'India', 'Indonesia', 'Brazil']
Prosperous ['United States', 'Japan', 'Germany', 'Thailand']
Struggling ['Nigeria', 'Ethiopia', 'Congo, Dem. Rep.', 'South Africa']
cont = g7.set_index("country")["continent"]
pd.crosstab(cont, feats["segment"])
segment Health risers Prosperous Struggling
continent
Africa 11 2 39
Americas 16 8 1
Asia 21 9 3
Europe 2 28 0
Oceania 0 2 0

See the segments and check the borderline countries#

Problem: are the segments visible, and which countries sit on a fence?

for name, grp in feats.groupby("segment"):
    plt.scatter(grp["log_gdp_2007"], grp["life_2007"], label=name,
                alpha=0.7)
plt.title("Segments by wealth and health, 2007")
plt.xlabel("Log10 GDP per person")
plt.ylabel("Life expectancy (years)")
plt.legend()
plt.show()
No description has been provided for this image
for name, grp in feats.groupby("segment"):
    plt.scatter(grp["gdp_growth"], grp["life_gain"], label=name,
                alpha=0.7)
plt.title("Segments by progress since 1952")
plt.xlabel("Log10 GDP growth, 1952 to 2007")
plt.ylabel("Life years gained")
plt.legend()
plt.show()
No description has been provided for this image
from sklearn.metrics import silhouette_samples
feats["sil"] = silhouette_samples(X, seg)
feats.groupby("segment")["sil"].mean().round(3)
segment
Health risers    0.332
Prosperous       0.400
Struggling       0.408
Name: sil, dtype: float64
feats.nsmallest(5, "sil")[["segment", "life_2007", "sil"]].round(2)
segment life_2007 sil
country
Korea, Rep. Prosperous 78.62 -0.00
Bosnia and Herzegovina Prosperous 74.85 0.00
Sao Tome and Principe Struggling 65.53 0.01
Panama Health risers 75.54 0.05
Sri Lanka Prosperous 72.40 0.05

Your turn: look up where five large countries fall.

picks = ["China", "India", "United States", "Nigeria", "Brazil"]
cols = ["segment", "life_2007", "life_gain", "sil"]
feats.loc[picks, cols].round(2)
segment life_2007 life_gain sil
country
China Health risers 72.96 28.96 0.20
India Health risers 64.70 27.32 0.48
United States Prosperous 78.24 9.80 0.53
Nigeria Struggling 46.86 10.54 0.55
Brazil Health risers 72.39 21.47 0.18

Stability check: would a slightly different sample give the same segments?

from sklearn.metrics import adjusted_rand_score
part = feats.sample(frac=0.8, random_state=42).index
rows = feats.index.get_indexer(part)
km_s = KMeans(n_clusters=3, n_init=10, random_state=42).fit(X[rows])
again = km_s.predict(X)
print(len(rows), round(adjusted_rand_score(seg, again), 3))
114 0.895

Mini-project: did the health risers close the gap?#

Follow each 2007 segment back to 1952. How did the life expectancy gap change?

g["segment"] = g["country"].map(feats["segment"])
trend = g.groupby(["year", "segment"])["lifeExp"].median()
trend = trend.unstack()
trend.loc[[1952, 1982, 2007]].round(1)
segment Health risers Prosperous Struggling
year
1952 43.3 64.4 38.5
1982 61.7 73.1 49.5
2007 72.3 78.4 51.5
trend.plot(marker="o")
plt.title("Median life expectancy by 2007 segment")
plt.xlabel("Year")
plt.ylabel("Life expectancy (years)")
plt.show()
No description has been provided for this image
gap = trend["Prosperous"] - trend["Health risers"]
print(round(gap[1952], 1), round(gap[2007], 1))
gap2 = trend["Prosperous"] - trend["Struggling"]
print(round(gap2[1952], 1), round(gap2[2007], 1))
21.1 6.1
25.8 26.9
top = feats.sort_values("life_gain", ascending=False)
top.groupby("segment").head(1)[["segment", "life_gain"]].round(2)
segment life_gain
country
Oman Health risers 38.06
Korea, Rep. Prosperous 31.17
Comoros Struggling 24.44
import os
out = feats[["segment", "life_2007", "life_gain", "sil"]].round(3)
out.to_csv("m32_segments.csv")
back = pd.read_csv("m32_segments.csv", index_col=0)
print(back.shape)
back["segment"].value_counts()
(142, 4)
segment
Health risers    50
Prosperous       49
Struggling       43
Name: count, dtype: int64

Finding: the health risers cut their gap to the prosperous from 21.1 to 6.1 years. The struggling segment fell further behind: 25.8 years then, 26.9 now.

Recap#

  • Pivot long yearly data into one row per country
  • Log skewed money values before scaling and clustering
  • Scores advise on k; usefulness to people decides
  • Relabel clusters by a feature, then profile and name them
  • Low-silhouette rows are the borderline cases to discuss

Challenges#

  1. Add log population as a fifth feature. Do the segments change?
  2. Build the features for 1982 instead of 2007. Who were the health risers then?
  3. Try agglomerative clustering with 3 clusters. Use the adjusted Rand index to compare.

What's next#

Lesson 33: Association Rule Mining with Apriori Algorithm From grouping rows to finding items that are bought together.

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.