Mathew K Analytics

Lesson 55 · Pandas Full Course

Clustering and Outlier Detection in Python: KMeans + Pandas | Pandas Course #55

In this lesson you'll be able to: Prepare and scale numeric features for clustering Group similar rows with KMeans and profile each cluster in pandas Choose…

📓 Full notebook

Download .ipynb

Pandas 55: Clustering outlier detection#

In this lesson you'll be able to:

  • Prepare and scale numeric features for clustering
  • Group similar rows with KMeans and profile each cluster in pandas
  • Choose the number of clusters with inertia and silhouette
  • Flag unusual rows with z-scores, IQR and Local Outlier Factor
  • Check clusters and outliers against what you already know Dataset: Palmer penguins (data/pandas/penguins.csv)

Warm-up: features ready for scikit-learn#

Lesson 54 prepared data for scikit-learn. Same habits, new data.

import pandas as pd
df = pd.read_csv("data/pandas/penguins.csv")
print(df.shape)
df.head()
(344, 7)
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g sex
0 Adelie Torgersen 39.1 18.7 181.0 3750.0 MALE
1 Adelie Torgersen 39.5 17.4 186.0 3800.0 FEMALE
2 Adelie Torgersen 40.3 18.0 195.0 3250.0 FEMALE
3 Adelie Torgersen NaN NaN NaN NaN NaN
4 Adelie Torgersen 36.7 19.3 193.0 3450.0 FEMALE
df.isna().sum()
species               0
island                0
bill_length_mm        2
bill_depth_mm         2
flipper_length_mm     2
body_mass_g           2
sex                  11
dtype: int64

Prepare the features#

Problem: clustering needs complete numbers on similar scales.

num = ["bill_length_mm", "bill_depth_mm", "flipper_length_mm",
       "body_mass_g"]
df[num].describe().round(1)
bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
count 342.0 342.0 342.0 342.0
mean 43.9 17.2 200.9 4201.8
std 5.5 2.0 14.1 802.0
min 32.1 13.1 172.0 2700.0
25% 39.2 15.6 190.0 3550.0
50% 44.4 17.3 197.0 4050.0
75% 48.5 18.7 213.0 4750.0
max 59.6 21.5 231.0 6300.0

Common mistake: clustering a table that still has gaps

from sklearn.cluster import KMeans
KMeans(n_clusters=3, n_init=10, random_state=42).fit(df[num])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[5], line 2
      1 from sklearn.cluster import KMeans
----> 2 KMeans(n_clusters=3, n_init=10, random_state=42).fit(df[num])

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py:1403, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1393     estimator._validate_params()
   1395 with (
   1396     config_context(
   1397         skip_parameter_validation=(
   (...)
   1401     callback_management_context(estimator),
   1402 ):
-> 1403     return fit_method(estimator, *args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\cluster\_kmeans.py:1459, in KMeans.fit(self, X, y, sample_weight)
   1431 @_fit_context(prefer_skip_nested_validation=True)
   1432 def fit(self, X, y=None, sample_weight=None):
   1433     """Compute k-means clustering.
   1434 
   1435     Parameters
   (...)
   1457         Fitted estimator.
   1458     """
-> 1459     X = validate_data(
   1460         self,
   1461         X,
   1462         accept_sparse="csr",
   1463         dtype=[np.float64, np.float32],
   1464         order="C",
   1465         copy=self.copy_x,
   1466         accept_large_sparse=False,
   1467     )
   1469     self._check_params_vs_input(X)
   1471     random_state = check_random_state(self.random_state)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:3041, in validate_data(_estimator, X, y, reset, validate_separately, skip_check_array, **check_params)
   3039         out = X, y
   3040 elif not no_val_X and no_val_y:
-> 3041     out = check_array(X, input_name="X", **check_params)
   3042 elif no_val_X and not no_val_y:
   3043     out = _check_y(y, **check_params)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:1090, in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, ensure_all_finite, ensure_non_negative, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)
   1084     raise ValueError(
   1085         f"Found array with dim {array.ndim},"
   1086         f" while dim <= 2 is required{context}."
   1087     )
   1089 if ensure_all_finite:
-> 1090     _assert_all_finite(
   1091         array,
   1092         input_name=input_name,
   1093         estimator_name=estimator_name,
   1094         allow_nan=ensure_all_finite == "allow-nan",
   1095     )
   1097 if copy:
   1098     if _is_numpy_namespace(xp):
   1099         # only make a copy if `array` and `array_orig` may share memory`

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:137, in _assert_all_finite(X, allow_nan, msg_dtype, estimator_name, input_name)
    134 if first_pass_isfinite:
    135     return
--> 137 _assert_all_finite_element_wise(
    138     X,
    139     xp=xp,
    140     allow_nan=allow_nan,
    141     msg_dtype=msg_dtype,
    142     estimator_name=estimator_name,
    143     input_name=input_name,
    144 )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:186, in _assert_all_finite_element_wise(X, xp, allow_nan, msg_dtype, estimator_name, input_name)
    169 if estimator_name and input_name == "X" and has_nan_error:
    170     # Improve the error message on how to handle missing values in
    171     # scikit-learn.
    172     msg_err += (
    173         f"\n{estimator_name} does not accept missing values"
    174         " encoded as NaN natively. For supervised learning, you might want"
   (...)
    184         "#estimators-that-handle-nan-values"
    185     )
--> 186 raise ValueError(msg_err)

ValueError: Input X contains NaN.
KMeans does not accept missing values encoded as NaN natively. For supervised learning, you might want to consider sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept missing values encoded as NaNs natively. Alternatively, it is possible to preprocess the data, for instance by using an imputer transformer in a pipeline or drop samples with missing values. See https://scikit-learn.org/stable/modules/impute.html You can find a list of all estimators that handle NaN values at the following page: https://scikit-learn.org/stable/modules/impute.html#estimators-that-handle-nan-values
p = df.dropna(subset=num).copy()
print(len(df), len(p))
344 342

Quick check: body mass runs into thousands, bill depth stays near 17. Which one would dominate a distance?

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler().set_output(transform="pandas")
X = scaler.fit_transform(p[num])
X.describe().loc[["mean", "std", "min", "max"]].round(2)
bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
mean 0.00 -0.00 -0.00 0.00
std 1.00 1.00 1.00 1.00
min -2.17 -2.05 -2.06 -1.88
max 2.88 2.21 2.14 2.62

Group penguins with KMeans#

Problem: find natural groups without being told the answer.

km = KMeans(n_clusters=3, n_init=10, random_state=42)
p["cluster"] = km.fit_predict(X)
p["cluster"].value_counts().sort_index()
cluster
0     87
1    123
2    132
Name: count, dtype: int64
order = p.groupby("cluster")["flipper_length_mm"].mean().rank()
p["cluster"] = p["cluster"].map(order.astype(int))
p.groupby("cluster")[num].mean().round(1)
bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
cluster
1 38.2 18.1 188.4 3584.7
2 47.5 18.8 196.9 3902.0
3 47.5 15.0 217.2 5076.0
p.groupby("cluster")["island"].value_counts().unstack(fill_value=0)
island Biscoe Dream Torgersen
cluster
1 37 55 40
2 7 69 11
3 123 0 0

Quick check: we never showed KMeans the species. Did it find them anyway?

pd.crosstab(p["species"], p["cluster"])
cluster 1 2 3
species
Adelie 127 24 0
Chinstrap 5 63 0
Gentoo 0 0 123
match = pd.crosstab(p["species"], p["cluster"]).max(axis=1)
print(match.sum(), len(p), round(match.sum() / len(p), 3))
313 342 0.915
raw = KMeans(n_clusters=3, n_init=10, random_state=42)
p["raw_cluster"] = raw.fit_predict(p[num])
pd.crosstab(p["species"], p["raw_cluster"])
raw_cluster 0 1 2
species
Adelie 112 0 39
Chinstrap 52 0 16
Gentoo 1 68 54

Choose the number of clusters#

Problem: in real work nobody tells you k.

from sklearn.metrics import silhouette_score
rows = []
for k in range(2, 8):
    model = KMeans(n_clusters=k, n_init=10, random_state=42)
    labels = model.fit_predict(X)
    sil = silhouette_score(X, labels)
    rows.append({"k": k, "inertia": model.inertia_, "silhouette": sil})
scores = pd.DataFrame(rows).set_index("k").round(3)
scores
inertia silhouette
k
2 565.708 0.532
3 379.393 0.447
4 300.400 0.400
5 232.605 0.379
6 204.393 0.368
7 187.029 0.331
import matplotlib.pyplot as plt
fig, axes = plt.subplots(1, 2, figsize=(9, 3.5))
scores["inertia"].plot(ax=axes[0], marker="o")
axes[0].set_title("Inertia: lower is tighter")
scores["silhouette"].plot(ax=axes[1], marker="o", color="green")
axes[1].set_title("Silhouette: higher is better")
for ax in axes:
    ax.set_xlabel("Number of clusters (k)")
axes[0].set_ylabel("Score")
plt.tight_layout()
No description has been provided for this image
from sklearn.metrics import silhouette_samples
p["sil"] = silhouette_samples(X, p["cluster"])
print(p.groupby("cluster")["sil"].agg(["mean", "min"]).round(3))
print((p["sil"] < 0).sum())
          mean    min
cluster              
1        0.431  0.079
2        0.301 -0.051
3        0.568  0.317
5

Your turn#

Silhouette prefers k = 2. Fit it and crosstab against species. What merged?

two = KMeans(n_clusters=2, n_init=10, random_state=42)
labels2 = two.fit_predict(X)
pd.crosstab(p["species"], labels2, colnames=["cluster"])
cluster 0 1
species
Adelie 151 0
Chinstrap 68 0
Gentoo 0 123

Scores guide the choice of k; domain knowledge makes the final call.

See the clusters in two dimensions with PCA#

Problem: four measurements can't be drawn on a flat screen.

from sklearn.decomposition import PCA
pca = PCA(n_components=2).set_output(transform="pandas")
proj = pca.fit_transform(X)
proj.head().round(2)
pca0 pca1
0 -1.84 0.05
1 -1.31 -0.43
2 -1.37 -0.15
4 -1.88 -0.00
5 -1.91 0.83

Quick check: how much of the original spread do two axes keep?

share = pd.Series(pca.explained_variance_ratio_,
                  index=["PC1", "PC2"])
print(share.round(3), round(share.sum(), 3))
PC1    0.688
PC2    0.193
dtype: float64 0.882
ax = proj.plot.scatter(x="pca0", y="pca1", c=p["cluster"],
                       cmap="viridis", figsize=(7, 4.5))
ax.set_title("Penguin clusters on the first two PCA axes")
ax.set_xlabel("First principal component")
ax.set_ylabel("Second principal component")
plt.show()
No description has been provided for this image

Your turn#

Draw the same PCA scatter, coloured by species instead of cluster.

palette = {"Adelie": "darkorange", "Chinstrap": "purple",
           "Gentoo": "teal"}
ax = proj.plot.scatter(x="pca0", y="pca1",
                       c=p["species"].map(palette), figsize=(7, 4.5))
ax.set_title("Penguin species on the first two PCA axes")
ax.set_xlabel("First principal component")
ax.set_ylabel("Second principal component")
plt.show()
No description has been provided for this image

Flag outliers one column at a time#

Problem: find measurements that are far from typical.

mass = p["body_mass_g"]
z = (mass - mass.mean()) / mass.std(ddof=0)
print((z - X["body_mass_g"]).abs().max().round(6))
0.0
z_out = (X.abs() > 3).sum()
print(z_out)
bill_length_mm       0
bill_depth_mm        0
flipper_length_mm    0
body_mass_g          0
dtype: int64
q1, q3 = p[num].quantile(0.25), p[num].quantile(0.75)
iqr = q3 - q1
low, high = q1 - 1.5 * iqr, q3 + 1.5 * iqr
((p[num] < low) | (p[num] > high)).sum()
bill_length_mm       0
bill_depth_mm        0
flipper_length_mm    0
body_mass_g          0
dtype: int64

Your turn#

Loosen the rule to 2.5 standard units. Which penguins break it in any column?

flag = (X.abs() > 2.5).any(axis=1)
print(flag.sum())
p.loc[flag, ["species"] + num]
3
species bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
169 Chinstrap 58.0 17.8 181.0 3700.0
237 Gentoo 49.2 15.2 221.0 6300.0
253 Gentoo 59.6 17.0 230.0 6050.0

Flag outliers with Local Outlier Factor#

Problem: a penguin can be normal in every column but odd overall.

from sklearn.neighbors import LocalOutlierFactor
lof = LocalOutlierFactor(n_neighbors=20, contamination=0.03)
p["outlier"] = lof.fit_predict(X) == -1
p["lof_score"] = -lof.negative_outlier_factor_
print(p["outlier"].sum())
11
p["lof_score"].describe().round(2)
count    342.00
mean       1.06
std        0.13
min        0.94
25%        0.99
50%        1.02
75%        1.09
max        2.15
Name: lof_score, dtype: float64
ax = p["lof_score"].plot.hist(bins=30, figsize=(7, 3.5))
ax.set_title("Local Outlier Factor scores for 342 penguins")
ax.set_xlabel("LOF score (around 1 is normal)")
ax.set_ylabel("Number of penguins")
plt.show()
No description has been provided for this image
cols = ["species", "bill_length_mm", "flipper_length_mm",
        "body_mass_g", "lof_score"]
top = p.sort_values("lof_score", ascending=False)
top[cols].head().round(2)
species bill_length_mm flipper_length_mm body_mass_g lof_score
169 Chinstrap 58.0 181.0 3700.0 2.15
253 Gentoo 59.6 230.0 6050.0 2.08
257 Gentoo 44.4 219.0 5250.0 1.59
142 Adelie 32.1 188.0 3050.0 1.52
189 Chinstrap 52.0 210.0 4800.0 1.50
X.loc[169].round(2)
bill_length_mm       2.58
bill_depth_mm        0.33
flipper_length_mm   -1.42
body_mass_g         -0.63
Name: 169, dtype: float64

Common mistake: calling predict on a fitted LOF

lof.predict(X)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_available_if.py:32, in _AvailableIfDescriptor._check(self, obj, owner)
     31 try:
---> 32     check_result = self.check(obj)
     33 except Exception as e:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\neighbors\_lof.py:342, in LocalOutlierFactor._check_novelty_predict(self)
    336     msg = (
    337         "predict is not available when novelty=False, use "
    338         "fit_predict if you want to predict on training data. Use "
    339         "novelty=True if you want to use LOF for novelty detection "
    340         "and predict on new unseen data."
    341     )
--> 342     raise AttributeError(msg)
    343 return True

AttributeError: predict is not available when novelty=False, use fit_predict if you want to predict on training data. Use novelty=True if you want to use LOF for novelty detection and predict on new unseen data.

The above exception was the direct cause of the following exception:

AttributeError                            Traceback (most recent call last)
Cell In[31], line 1
----> 1 lof.predict(X)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_available_if.py:43, in _AvailableIfDescriptor.__get__(self, obj, owner)
     39 def __get__(self, obj, owner=None):
     40     if obj is not None:
     41         # delegate only on instances, not the classes.
     42         # this is to allow access to the docstrings.
---> 43         self._check(obj, owner=owner)
     44         out = MethodType(self.fn, obj)
     46     else:
     47         # This makes it possible to use the decorated method as an unbound method,
     48         # for instance when monkeypatching.

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_available_if.py:34, in _AvailableIfDescriptor._check(self, obj, owner)
     32     check_result = self.check(obj)
     33 except Exception as e:
---> 34     raise AttributeError(attr_err_msg) from e
     36 if not check_result:
     37     raise AttributeError(attr_err_msg)

AttributeError: This 'LocalOutlierFactor' has no attribute 'predict'
new_lof = LocalOutlierFactor(n_neighbors=20, novelty=True)
new_lof.fit(X.iloc[:300].to_numpy())
flags = new_lof.predict(X.iloc[300:].to_numpy())
print(len(flags), (flags == -1).sum())
42 1

Your turn#

Which species has the most LOF outliers, and which cluster?

by_species = p.groupby("species")["outlier"].sum()
by_cluster = p.groupby("cluster")["outlier"].sum()
print(by_species, by_cluster, sep="\n\n")
species
Adelie       5
Chinstrap    3
Gentoo       3
Name: outlier, dtype: int64

cluster
1    4
2    4
3    3
Name: outlier, dtype: int64

Mini-project: which penguins don't fit their species?#

Misfit = a penguin outside its species' main cluster.

table = pd.crosstab(p["species"], p["cluster"])
home = table.idxmax(axis=1)
p["misfit"] = p["cluster"] != p["species"].map(home)
p.groupby("species")["misfit"].agg(["sum", "mean"]).round(3)
sum mean
species
Adelie 24 0.159
Chinstrap 5 0.074
Gentoo 0 0.000
adelie = p[p["species"] == "Adelie"]
adelie.groupby("misfit")[num].mean().round(1)
bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
misfit
False 38.0 18.2 188.5 3596.7
True 42.7 19.3 197.6 4251.0
adelie.groupby("misfit")["sex"].value_counts().unstack(fill_value=0)
sex FEMALE MALE
misfit
False 73 50
True 0 23
adelie.groupby("misfit")["island"].value_counts().unstack()
island Biscoe Dream Torgersen
misfit
False 37 50 40
True 7 6 11
pd.crosstab(p["misfit"], p["outlier"])
outlier False True
misfit
False 303 10
True 28 1
colors = p["misfit"].map({False: "lightgrey", True: "crimson"})
ax = p.plot.scatter(x="bill_length_mm", y="bill_depth_mm",
                    c=colors, figsize=(7, 4.5))
ax.set_title("Penguins outside their species' main cluster")
ax.set_xlabel("Bill length (mm)")
ax.set_ylabel("Bill depth (mm)")
plt.show()
No description has been provided for this image
n_mis = p["misfit"].sum()
both = (p["misfit"] & p["outlier"]).sum()
print(f"{n_mis} misfits, of which {both} also flagged by LOF")
29 misfits, of which 1 also flagged by LOF

Recap#

  • Clustering needs complete, scaled numeric features
  • KMeans groups rows; profile clusters with groupby and crosstab
  • Inertia and silhouette guide k; domain knowledge decides
  • z-scores and IQR check one column; LOF checks all together
  • LOF needs novelty=True to score new rows

Challenges#

  1. Cluster on bill length and bill depth only. Does agreement improve?
  2. Try LOF with n_neighbors=10 and 40. Which penguins stay flagged?
  3. Cluster each sex separately with k = 3 and compare the crosstabs.

What's next#

Lesson 56: Exploring Real Datasets with Pandas A full first look at a messy real dataset, start to finish.

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.