Lesson 10 · ML algorithms deep dive

Hierarchical Clustering & Dendrograms | ML Algorithms #10

Video ten of the 12-part series: revisiting real mall customers with a genuinely different clustering approach. Same real customers, same real question, no…

⬇ Download notebook ⬇ Notebook + data (.zip)Open in Colab ↗

CourseML algorithms deep dive
Lesson10 of 12
Video15 min
FormatJupyter notebook · 9 code cells
Data1 dataset

SciPy scikit-learn pandas NumPy Matplotlib

What you'll learn

A Genuinely Different Approach
Reading a Dendrogram
Cutting the Tree
Comparing Linkage Methods
Cophenetic Correlation
A Real Rematch Against K-Means
Visualizing the Real Agreement
Strengths, Weaknesses, and When to Use It

Datasets used in this lesson

Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.

mall_customers.csv4.3 KB

📓 Full notebook

Download .ipynb

ML Algorithms Deep-Dive, Video 10: Hierarchical Clustering#

Video ten of the 12-part series: revisiting real mall customers with a genuinely different clustering approach.
Same real customers, same real question, no K chosen in advance this time.
Let's get into it.

Before You Start#

Open a new Jupyter Notebook in VS Code and select your Python interpreter as the kernel.
You'll need pandas, NumPy, Matplotlib, SciPy, and scikit-learn.
Place mall_customers.csv in the same folder as this notebook.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from scipy.cluster.hierarchy import linkage, dendrogram, fcluster, cophenet
from scipy.spatial.distance import pdist
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score, adjusted_rand_score

customers = pd.read_csv('mall_customers.csv')
X = customers[['Annual Income (k$)', 'Spending Score (1-100)']].values
print(f'Real customers in this dataset: {len(customers)}')

Real customers in this dataset: 200

Part 1: A Genuinely Different Approach#

linkage_matrix = linkage(X, method='ward')
print(f'Real shape of the linkage matrix: {linkage_matrix.shape}')
print('Real first few merges (cluster A, cluster B, distance, size):')
print(linkage_matrix[:5])

Real shape of the linkage matrix: (199, 4)
Real first few merges (cluster A, cluster B, distance, size):
[[ 65.  68.   0.   2.]
 [ 48.  49.   0.   2.]
 [156. 158.   0.   2.]
 [129. 131.   0.   2.]
 [ 21.  23.   1.   2.]]

Part 2: Reading a Dendrogram#

plt.figure(figsize=(12, 6))
dendrogram(linkage_matrix)
plt.axhline(179.77, color='darkred', linestyle='--', label='Cut for K=5')
plt.title('Real Hierarchical Clustering Dendrogram (Ward Linkage)')
plt.xlabel('Customer Index')
plt.ylabel('Distance')
plt.legend()
plt.savefig('hierarchical_dendrogram.png', dpi=150)
plt.show()

No description has been provided for this image

Part 3: Cutting the Tree#

labels_ward = fcluster(linkage_matrix, t=5, criterion='maxclust')
cluster_sizes = pd.Series(labels_ward).value_counts().sort_index()
print('Real cluster sizes at K=5:')
print(cluster_sizes)

Real cluster sizes at K=5:
1    21
2    23
3    85
4    39
5    32
Name: count, dtype: int64

Part 4: Comparing Linkage Methods#

for method in ['ward', 'complete', 'average', 'single']:
    Z = linkage(X, method=method)
    labels = fcluster(Z, t=5, criterion='maxclust')
    score = silhouette_score(X, labels)
    print(f'method={method}: real K=5 silhouette={score:.4f}')

method=ward: real K=5 silhouette=0.5530
method=complete: real K=5 silhouette=0.5530
method=average: real K=5 silhouette=0.4792
method=single: real K=5 silhouette=0.2695

Single linkage's real weakness has a name: chaining. It merges clusters based on their single real closest pair of points, so one real bridge of nearby customers can string two genuinely different groups together into one long real chain instead of two compact ones. Ward and complete linkage, which look at overall real cluster shape rather than just the nearest edge, avoid that real trap here.

Part 5: Cophenetic Correlation#

original_distances = pdist(X)
for method in ['ward', 'complete', 'average', 'single']:
    Z = linkage(X, method=method)
    cophenetic_corr, _ = cophenet(Z, original_distances)
    print(f'method={method}: real cophenetic correlation={cophenetic_corr:.4f}')

method=ward: real cophenetic correlation=0.7179
method=complete: real cophenetic correlation=0.6794
method=average: real cophenetic correlation=0.7213
method=single: real cophenetic correlation=0.7231

Part 6: A Real Rematch Against K-Means#

kmeans_model = KMeans(n_clusters=5, random_state=42, n_init=10).fit(X)
agreement = adjusted_rand_score(labels_ward, kmeans_model.labels_)
print(f'Real adjusted Rand index between Ward hierarchical and K-Means: {agreement:.4f}')

Real adjusted Rand index between Ward hierarchical and K-Means: 0.9420

That real agreement is itself valuable evidence: it means video nine's five real segments weren't an artifact of K-Means' particular assumptions, a genuinely different real method, with no concept of centroids or K chosen in advance, found essentially the same real structure in the data.

Part 7: Visualizing the Real Agreement#

fig, axes = plt.subplots(1, 2, figsize=(14, 6))
axes[0].scatter(X[:, 0], X[:, 1], c=labels_ward, cmap='viridis')
axes[0].set_title('Real Hierarchical Clustering (Ward)')
axes[0].set_xlabel('Annual Income (k$)')
axes[0].set_ylabel('Spending Score (1-100)')
axes[1].scatter(X[:, 0], X[:, 1], c=kmeans_model.labels_, cmap='viridis')
axes[1].set_title('Real K-Means')
axes[1].set_xlabel('Annual Income (k$)')
axes[1].set_ylabel('Spending Score (1-100)')
plt.tight_layout()
plt.show()

Part 8: Strengths, Weaknesses, and When to Use It#

Strength: no need to commit to K before fitting, the full real hierarchy is built once and cut afterward at any level.
Strength: the real dendrogram itself is a genuinely useful visualization, showing how clusters nest inside each other.
Weakness: single linkage's real chaining problem, demonstrated directly in part four.
Weakness: real computational cost grows faster than K-Means as the real dataset gets larger.
Use it when the real number of natural groups is genuinely unknown, or when the nested real structure itself, not just a flat partition, is worth seeing.

Part 9: Saving Your Work#

np.save('linkage_matrix.npy', linkage_matrix)
reloaded_linkage = np.load('linkage_matrix.npy')
print(f'Real original linkage matrix shape: {linkage_matrix.shape}')
print(f'Real reloaded linkage matrix shape: {reloaded_linkage.shape}')
print(f'Real arrays match exactly: {np.array_equal(linkage_matrix, reloaded_linkage)}')

Real original linkage matrix shape: (199, 4)
Real reloaded linkage matrix shape: (199, 4)
Real arrays match exactly: True

Wrap-Up: What You Learned#

Building a real, full hierarchy of clusters, with K chosen afterward instead of in advance.
Reading a real dendrogram, and cutting it at a chosen real height or real cluster count.
Four real linkage methods compared directly, including single linkage's real chaining weakness.
Cophenetic correlation, a genuinely different real quality measure than silhouette score.
A real, independent rematch against video nine's K-Means, converging on nearly the same real customer segments.
Video eleven moves to PCA, compressing real high-dimensional handwritten digit images down to something you can actually plot. Subscribe so it lands automatically see you there.

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.

☕ Buy me a coffee Other ways to help