Lesson 3 · Data Mining Full Course
Data Mining Full Course Part 3 (3.5 Hours): Classification Algorithms & Model Evaluation
All 9 lesson notebooks from Part 3 of the Data Mining Full Course in Python, in teaching order. Each lesson starts with a heading; the timestamps below jump…
- CourseData Mining Full Course
- Lesson3 of 6
- Video3 h 39 min
- FormatJupyter notebook · 328 code cells
- Data5 datasets
What you'll learn
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
- titanic.csv preview58.9 KB
- telco_churn.csv preview947.7 KB
- penguins.csv preview13.2 KB
- sms_spam.csv preview469.5 KB
- pima_diabetes.csv preview22.7 KB
📓 Full notebook
Download .ipynbData Mining Full Course, Part 3: Classification Algorithms and Model Evaluation#
All 9 lesson notebooks from Part 3 of the Data Mining Full Course in Python, in teaching order. Each lesson starts with a heading; the timestamps below jump to it in the video.
- 0:00 Lesson 14: Decision Trees- ID3, C4.5, CART
- 0:24:18 Lesson 15: k-Nearest Neighbors kNN Algorithm
- 0:48:24 Lesson 16: Naive Bayes Classifier
- 1:12:45 Lesson 17: Logistic Regression for Classification
- 1:36:03 Lesson 18: Support Vector Machines SVM
- 1:59:56 Lesson 19: Introduction to Neural Networks for Data Mining
- 2:27:03 Lesson 20: Evaluating Classifiers- Accuracy, Precision, Recall, F1-Score
- 2:51:02 Lesson 21: ROC Curves, AUC, and Confusion Matrix Analysis
- 3:15:34 Lesson 22: Hands-on Classification Project- Predicting Customer Churn
Data files: save the datasets from the lesson page in the same folder as this notebook. Files the lessons create are written to that folder too.
Data Mining 14: Decision Trees - ID3, C4.5, CART#
In this lesson you'll be able to:
- Explain how a tree chooses its yes/no questions
- Calculate entropy, Gini, information gain and gain ratio by hand
- Tell ID3, C4.5 and CART apart, and know which one scikit-learn uses
- Grow, prune and read a tree that predicts customer churn
- Turn tree rules into a target list for a business Datasets: Titanic passengers (data/pandas/titanic.csv) and IBM Telco churn (data/mining/telco_churn.csv)
Warm-up: the Titanic, cleaned#
Lesson 13 prepared the Titanic data. One column stood out.
import pandas as pd
import numpy as np
titanic = pd.read_csv("titanic.csv")
titanic["IsFemale"] = (titanic["Sex"] == "female").astype(int)
titanic.groupby("IsFemale")["Survived"].mean().round(3)
titanic.groupby("Pclass")["Survived"].mean().round(3)
How a tree asks questions#
Problem: turn a table into a short chain of yes/no rules.
- Each node asks one question about one column
- Each answer sends the row left or right
- A leaf at the bottom predicts the majority class
Quick check: which question will the tree ask first?
from sklearn.tree import DecisionTreeClassifier, export_text
cols = ["Pclass", "IsFemale", "Fare", "SibSp", "Parch"]
stump = DecisionTreeClassifier(max_depth=1, random_state=42)
stump.fit(titanic[cols], titanic["Survived"])
print(export_text(stump, feature_names=cols))
import matplotlib.pyplot as plt
from sklearn.tree import plot_tree
small = DecisionTreeClassifier(max_depth=2, random_state=42)
small.fit(titanic[cols], titanic["Survived"])
fig, ax = plt.subplots(figsize=(11, 5))
plot_tree(small, feature_names=cols, class_names=["Died", "Lived"],
filled=True, impurity=False, ax=ax)
ax.set_title("Two levels of questions on the Titanic")
plt.show()
Measure impurity: entropy and Gini#
Problem: score how mixed a group is, so splits can be compared.
- Entropy (ID3, C4.5): 0 when pure, 1 for a 50/50 split of two classes
- Gini (CART): 0 when pure, 0.5 for a 50/50 split of two classes
def entropy(labels):
p = labels.value_counts(normalize=True)
return float(-(p * np.log2(p)).sum())
def gini(labels):
p = labels.value_counts(normalize=True)
return float(1 - (p ** 2).sum())
survived = titanic["Survived"]
print(round(entropy(survived), 3), round(gini(survived), 3))
by_sex = titanic.groupby("Sex")["Survived"]
by_sex.agg([entropy, gini, "size"]).round(3)
third_women = titanic[(titanic["Sex"] == "female")
& (titanic["Pclass"] == 3)]["Survived"]
print(third_women.value_counts().to_dict())
print(entropy(third_women), gini(third_women))
def info_gain(data, col, target="Survived"):
weights = data[col].value_counts(normalize=True)
child = data.groupby(col)[target].apply(entropy)
after = (weights * child).sum()
return entropy(data[target]) - after
for col in ["Sex", "Pclass", "Embarked"]:
print(col, round(info_gain(titanic, col), 3))
Your turn: work out the weighted Gini after splitting by Pclass.
w = titanic["Pclass"].value_counts(normalize=True)
g = titanic.groupby("Pclass")["Survived"].apply(gini)
print(g.round(3).to_dict())
print(round((w * g).sum(), 3))
ID3, C4.5 and CART: why gain ratio exists#
Problem: plain information gain loves columns with many values.
print(titanic["PassengerId"].nunique())
print(round(info_gain(titanic, "PassengerId"), 3))
def gain_ratio(data, col):
split_info = entropy(data[col])
return info_gain(data, col) / split_info
rows = ["Sex", "Pclass", "Embarked", "Ticket", "PassengerId"]
table = pd.DataFrame({
"values": [titanic[c].nunique() for c in rows],
"gain": [info_gain(titanic, c) for c in rows],
"ratio": [gain_ratio(titanic, c) for c in rows]},
index=rows)
table.round(3).sort_values("ratio", ascending=False)
| ID3 | C4.5 | CART | |
|---|---|---|---|
| Split score | information gain | gain ratio | Gini |
| Branches | one per value | one per value | always two |
| Numbers, gaps, pruning | no | yes | yes |
Common mistake: asking scikit-learn for a C4.5 tree.
c45 = DecisionTreeClassifier(criterion="gain_ratio")
c45.fit(titanic[cols], titanic["Survived"])
id3_like = DecisionTreeClassifier(criterion="entropy", max_depth=1,
random_state=42)
id3_like.fit(titanic[cols], titanic["Survived"])
print(export_text(id3_like, feature_names=cols))
Grow a CART tree on telecom churn#
Problem: predict which of 7,043 customers will leave.
telco = pd.read_csv("telco_churn.csv")
print(telco.shape)
print(telco["Churn"].value_counts(normalize=True).round(3))
Your turn: find the churn rate for each contract type.
is_churn = telco["Churn"] == "Yes"
is_churn.groupby(telco["Contract"]).mean().round(3)
Common mistake: converting a text column that hides blanks.
pd.to_numeric(telco["TotalCharges"])
total = pd.to_numeric(telco["TotalCharges"], errors="coerce")
print(total.isna().sum())
print(telco.loc[total.isna(), "tenure"].unique())
telco["TotalCharges"] = total.fillna(0)
y = (telco["Churn"] == "Yes").astype(int)
X_raw = telco.drop(columns=["customerID", "Churn"])
X = pd.get_dummies(X_raw, drop_first=True, dtype=int)
print(X.shape)
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, stratify=y, random_state=42)
print(X_train.shape, X_test.shape)
Quick check: a tree with no limits scores how well on its own training data?
full = DecisionTreeClassifier(random_state=42)
full.fit(X_train, y_train)
print(round(full.score(X_train, y_train), 3))
print(round(full.score(X_test, y_test), 3))
print(full.get_depth(), full.get_n_leaves())
Prune the tree with depth and leaf size#
Problem: stop the tree memorising, and tune it without touching the test set.
from sklearn.model_selection import cross_val_score
depths = range(1, 13)
cv_acc = []
for d in depths:
tree = DecisionTreeClassifier(max_depth=d, random_state=42)
s = cross_val_score(tree, X_train, y_train, cv=5)
cv_acc.append(s.mean())
cv = pd.Series(cv_acc, index=depths)
cv.round(3).head(8)
ax = cv.plot(marker="o")
ax.set_title("Cross-validated accuracy by tree depth")
ax.set_xlabel("max_depth")
ax.set_ylabel("Mean accuracy (5 folds)")
plt.show()
best_depth = int(cv.idxmax())
pruned = DecisionTreeClassifier(max_depth=best_depth,
min_samples_leaf=20,
random_state=42)
pruned.fit(X_train, y_train)
print(best_depth, pruned.get_n_leaves())
print(round(pruned.score(X_test, y_test), 3))
for leaf in [1, 20, 100]:
tree = DecisionTreeClassifier(min_samples_leaf=leaf,
random_state=42)
s = cross_val_score(tree, X_train, y_train, cv=5).mean()
n = tree.fit(X_train, y_train).get_n_leaves()
print(leaf, round(s, 3), n)
Quick check: is 77.7% accuracy impressive if 73.5% of customers stay anyway?
from sklearn.metrics import classification_report
pred = pruned.predict(X_test)
print(classification_report(y_test, pred, digits=3))
Read the tree and its feature importances#
Problem: explain the model in words a manager can use.
rules = export_text(pruned, feature_names=list(X.columns),
max_depth=2)
print(rules)
imp = pd.Series(pruned.feature_importances_, index=X.columns)
top = imp.sort_values(ascending=False).head(6)
print(top.round(3))
ax = top.sort_values().plot(kind="barh", color="teal")
ax.set_title("Top features in the pruned churn tree")
ax.set_xlabel("Importance (share of impurity reduction)")
plt.show()
leaf_id = pd.Series(pruned.apply(X_train), index=X_train.index)
seg = y_train.groupby(leaf_id).agg(["mean", "size"])
seg.round(3).sort_values("mean", ascending=False)
new = X_test.head(3)
print(pruned.predict_proba(new)[:, 1].round(3))
print(y_test.head(3).to_list())
Mini-project: who should the retention team call?#
Use the pruned tree to rank test customers and check the call list.
risk = pd.Series(pruned.predict_proba(X_test)[:, 1],
index=X_test.index)
cutoff = risk.quantile(0.8)
called = risk >= cutoff
print(called.sum(), round(cutoff, 3))
hit = y_test[called].mean()
base = y_test.mean()
caught = y_test[called].sum() / y_test.sum()
print(f"List {hit:.1%}, base {base:.1%}, caught {caught:.1%}")
charges = telco.loc[X_test.index, "MonthlyCharges"]
charges.groupby(called).median()
listed = telco.loc[called[called].index]
listed["Contract"].value_counts(normalize=True).round(3)
Recap#
- A tree asks the question that most reduces impurity, then repeats
- Entropy and Gini both measure how mixed a group is
- ID3 uses information gain; C4.5's gain ratio penalises ID-like columns
- scikit-learn builds CART trees; criterion only changes the score
- Prune with max_depth and min_samples_leaf, tuned by cross-validation
Challenges#
- Tune min_samples_leaf (5, 20, 50, 100) with cross-validation.
- Refit the churn tree with criterion="entropy". Do the top features change?
- Compute the gain ratio of Cabin on the Titanic. What does it say?
What's next#
Lesson 15: k-Nearest Neighbors (kNN) Algorithm Classify by looking at the most similar examples, and see why scaling matters.
Data Mining 15: k-Nearest Neighbors (kNN) Algorithm#
In this lesson you'll be able to:
- Explain kNN as a vote among the most similar examples
- Find nearest neighbours by hand with Euclidean distance
- Show why kNN needs scaled features, and fix it with a pipeline
- Choose k with cross-validation, not the test set
- Read kNN's mistakes and vote shares Dataset: Palmer penguins, 344 real birds from Antarctica (data/pandas/penguins.csv)
Warm-up: a decision tree on the penguins#
Lesson 14 grew decision trees. Here is one on today's data.
import pandas as pd
import numpy as np
penguins = pd.read_csv("penguins.csv")
print(penguins.shape)
penguins.head(3)
from sklearn.tree import DecisionTreeClassifier, export_text
num = ["bill_length_mm", "bill_depth_mm",
"flipper_length_mm", "body_mass_g"]
known = penguins.dropna(subset=num)
tree = DecisionTreeClassifier(max_depth=2, random_state=42)
tree.fit(known[num], known["species"])
print(export_text(tree, feature_names=num))
Classify by asking the neighbours#
Problem: label a new bird using the birds most like it.
- Measure the distance from the new bird to every known bird
- Keep the k closest: its nearest neighbours
- Let them vote; the majority species wins
import matplotlib.pyplot as plt
import seaborn as sns
ax = sns.scatterplot(data=known, x="bill_length_mm",
y="flipper_length_mm", hue="species")
ax.set_title("Penguins by bill and flipper length")
ax.set_xlabel("Bill length (mm)")
ax.set_ylabel("Flipper length (mm)")
plt.show()
from sklearn.model_selection import train_test_split
X = known[num]
y = known["species"]
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, stratify=y, random_state=42)
print(X_train.shape, X_test.shape)
query = X_test.iloc[[0]]
query
diff = X_train - query.to_numpy()
dist = np.sqrt((diff ** 2).sum(axis=1))
nearest = dist.nsmallest(5)
nearest.round(1)
Quick check: how will these 5 neighbours vote?
y_train[nearest.index].value_counts()
print(y_test.iloc[0])
manhattan = diff.abs().sum(axis=1)
y_train[manhattan.nsmallest(5).index].value_counts()
Fit kNN with scikit-learn#
Problem: repeat the vote for every test bird, safely.
Common mistake: fitting kNN on rows with missing measurements.
from sklearn.neighbors import KNeighborsClassifier
raw = KNeighborsClassifier(n_neighbors=5)
raw.fit(penguins[num], penguins["species"])
print(penguins[num].isna().any(axis=1).sum())
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train, y_train)
print(knn.predict(query))
print(round(knn.score(X_test, y_test), 3))
d, idx = knn.kneighbors(query)
print(d.round(1))
Quick check: what did fit() actually learn?
print(knn.n_samples_fit_)
print(knn.classes_)
- kNN is a lazy learner: fit stores the data, predict does the work
- Fast to train, slower to predict on big tables
raw_pred = knn.predict(X_test)
pd.crosstab(y_test, raw_pred, rownames=["real"],
colnames=["predicted"])
Why scaling decides who your neighbours are#
Problem: one big-number column can drown out the others.
X_train.agg(["min", "max", "std"]).round(1)
X_train.loc[nearest.index]
ax = sns.scatterplot(data=X_train.assign(species=y_train),
x="bill_length_mm", y="body_mass_g",
hue="species", alpha=0.4)
nb = X_train.loc[nearest.index]
ax.scatter(nb["bill_length_mm"], nb["body_mass_g"], color="black",
marker="x", s=80, label="raw neighbours")
ax.set_title("Unscaled neighbours all share one body mass")
ax.set_xlabel("Bill length (mm)")
ax.set_ylabel("Body mass (g)")
ax.legend()
plt.show()
share = (diff ** 2).div((diff ** 2).sum(axis=1), axis=0)
share.mean().round(3)
Quick check: the neighbours were chosen almost entirely by weight. Is that fair?
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
scaled_knn = make_pipeline(StandardScaler(),
KNeighborsClassifier(n_neighbors=5))
scaled_knn.fit(X_train, y_train)
print(round(scaled_knn.score(X_test, y_test), 3))
scaler = scaled_knn.named_steps["standardscaler"]
model = scaled_knn.named_steps["kneighborsclassifier"]
d, idx = model.kneighbors(scaler.transform(query))
y_train.iloc[idx[0]].value_counts()
Your turn: fit an unscaled kNN on bill length and bill depth only. Both are in millimetres.
bills = ["bill_length_mm", "bill_depth_mm"]
bill_knn = KNeighborsClassifier(n_neighbors=5)
bill_knn.fit(X_train[bills], y_train)
print(round(bill_knn.score(X_test[bills], y_test), 3))
Choose k with cross-validation#
Problem: too few neighbours is noisy, too many blurs the classes.
from sklearn.model_selection import cross_val_score
ks = range(1, 31, 2)
scores = []
for k in ks:
pipe = make_pipeline(StandardScaler(),
KNeighborsClassifier(n_neighbors=k))
scores.append(cross_val_score(pipe, X_train, y_train,
cv=5).mean())
cv = pd.Series(scores, index=ks)
cv.round(3).head(8)
ax = cv.plot(marker="o")
ax.set_title("Cross-validated accuracy by number of neighbours")
ax.set_xlabel("k (neighbours)")
ax.set_ylabel("Mean accuracy (5 folds)")
plt.show()
one = make_pipeline(StandardScaler(),
KNeighborsClassifier(n_neighbors=1))
one.fit(X_train, y_train)
print(one.score(X_train, y_train), round(cv[1], 3))
train_acc = []
for k in ks:
pipe = make_pipeline(StandardScaler(),
KNeighborsClassifier(n_neighbors=k))
train_acc.append(pipe.fit(X_train, y_train)
.score(X_train, y_train))
gap = pd.DataFrame({"train": train_acc, "cv": scores}, index=ks)
gap.round(3).head(5)
best_k = int(cv.idxmax())
final = make_pipeline(StandardScaler(),
KNeighborsClassifier(n_neighbors=best_k))
final.fit(X_train, y_train)
print(best_k, round(final.score(X_test, y_test), 3))
Your turn: try weights="distance" so closer neighbours count more. Score it with cross-validation.
weighted = make_pipeline(StandardScaler(), KNeighborsClassifier(
n_neighbors=best_k, weights="distance"))
s = cross_val_score(weighted, X_train, y_train, cv=5).mean()
print(round(s, 3), round(cv[best_k], 3))
Read the mistakes and the vote shares#
Problem: find which species get confused, and how sure kNN was.
from sklearn.metrics import confusion_matrix
pred = final.predict(X_test)
labels = final.classes_
cm = confusion_matrix(y_test, pred, labels=labels)
pd.DataFrame(cm, index=labels, columns=labels)
from sklearn.metrics import classification_report
print(classification_report(y_test, pred, digits=3))
wrong = X_test[pred != y_test]
wrong.assign(real=y_test[wrong.index],
predicted=pred[pred != y_test])
proba = pd.DataFrame(final.predict_proba(X_test),
columns=labels, index=X_test.index)
unsure = proba.max(axis=1) < 1
print(unsure.sum(), len(proba))
proba[unsure].round(2).head()
Common mistake: asking for k bigger than the data allows.
tiny = X_train.head(4)
small = KNeighborsClassifier(n_neighbors=5)
small.fit(tiny, y_train.head(4)).predict(query)
k = min(5, len(tiny))
small = KNeighborsClassifier(n_neighbors=k)
print(small.fit(tiny, y_train.head(4)).predict(query))
Mini-project: can kNN tell a penguin's sex?#
Researchers often can't sex a bird in the field. Do the measurements give it away?
sexed = known.dropna(subset=["sex"])
Xs = pd.get_dummies(sexed[num + ["species"]], dtype=int)
ys = sexed["sex"]
print(Xs.shape)
ys.value_counts()
Xs_tr, Xs_te, ys_tr, ys_te = train_test_split(
Xs, ys, test_size=0.25, stratify=ys, random_state=42)
sex_cv = []
for k in ks:
pipe = make_pipeline(StandardScaler(),
KNeighborsClassifier(n_neighbors=k))
sex_cv.append(cross_val_score(pipe, Xs_tr, ys_tr, cv=5).mean())
sex_k = ks[int(np.argmax(sex_cv))]
print(sex_k, round(max(sex_cv), 3))
ax = pd.Series(sex_cv, index=ks).plot(marker="o", color="purple")
ax.set_title("Sexing penguins: accuracy by number of neighbours")
ax.set_xlabel("k (neighbours)")
ax.set_ylabel("Mean accuracy (5 folds)")
plt.show()
sex_model = make_pipeline(StandardScaler(),
KNeighborsClassifier(n_neighbors=sex_k))
sex_model.fit(Xs_tr, ys_tr)
acc = sex_model.score(Xs_te, ys_te)
base = ys_te.value_counts(normalize=True).max()
print(f"kNN {acc:.1%} vs guessing {base:.1%}")
three = Xs_te.head(3)
print(sex_model.predict_proba(three).round(2))
print(ys_te.head(3).to_list())
right = sex_model.predict(Xs_te) == ys_te
right.groupby(sexed.loc[Xs_te.index, "species"]).mean().round(3)
sex_tree = DecisionTreeClassifier(max_depth=4, random_state=42)
t = cross_val_score(sex_tree, Xs_tr, ys_tr, cv=5).mean()
print(round(t, 3), round(max(sex_cv), 3))
print(f"Species: {final.score(X_test, y_test):.1%} with k={best_k}")
print(f"Sex: {acc:.1%} with k={sex_k}, guessing {base:.1%}")
Recap#
- kNN stores the training data and votes among the k closest rows
- Distance is dominated by big-number columns, so always scale
- Put the scaler in a pipeline so it learns from training rows only
- Choose k with cross-validation; odd k avoids two-way ties
- predict_proba shows the vote shares behind each prediction
Challenges#
- Re-run the species model using only bill length and depth.
- Swap StandardScaler for MinMaxScaler. Does the best k change?
- Try metric="manhattan" in the sex model and compare scores.
What's next#
Lesson 16: Naive Bayes Classifier Classify with probabilities: fast, simple, and great with text.
Data Mining 16: Naive Bayes Classifier#
In this lesson you'll be able to:
- Apply Bayes' rule to real counts by hand
- Train Gaussian Naive Bayes on numeric measurements
- Turn text into word counts and train Multinomial Naive Bayes
- Fix the zero-frequency problem with smoothing
- Choose a spam threshold that protects real messages Dataset: 5,572 real SMS messages labelled ham or spam (data/mining/sms_spam.csv)
Warm-up: kNN on text messages#
Lesson 15 classified with k-nearest neighbours. Same idea, today's data.
import pandas as pd
sms = pd.read_csv("sms_spam.csv")
sms["spam"] = (sms["label"] == "spam").astype(int)
print(sms.shape)
sms["label"].value_counts()
sms["length"] = sms["message"].str.len()
sms["digits"] = sms["message"].str.count(r"\d")
sms["capitals"] = sms["message"].str.count(r"[A-Z]")
nums = ["length", "digits", "capitals"]
sms.groupby("label")[nums].mean().round(1)
from sklearn.model_selection import train_test_split
train, test = train_test_split(
sms, test_size=0.25, stratify=sms["spam"], random_state=42)
print(len(train), len(test))
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
knn = make_pipeline(StandardScaler(), KNeighborsClassifier(5))
knn.fit(train[nums], train["spam"])
print(round(knn.score(test[nums], test["spam"]), 3))
Bayes' rule with real counts#
Problem: turn 'how often spam says free' into 'how likely this message is spam'.
- Prior: how common spam is before we read a word
- Likelihood: how often spam (or ham) contains the clue
- Posterior = prior x likelihood / how common the clue is
p_spam = sms["spam"].mean()
print(round(p_spam, 3))
has_free = sms["message"].str.contains(r"\bfree\b", case=False)
pd.crosstab(has_free, sms["label"])
p_free_spam = has_free[sms["spam"] == 1].mean()
p_free = has_free.mean()
posterior = p_free_spam * p_spam / p_free
print(round(p_free_spam, 3), round(p_free, 3))
print(round(posterior, 3))
Quick check: Bayes said 0.742. What share of messages that contain 'free' really are spam?
print(round(sms.loc[has_free, "spam"].mean(), 3))
Your turn: use Bayes' rule to find the spam chance for messages that say 'call'.
has_call = sms["message"].str.contains(r"\bcall\b", case=False)
p_call_spam = has_call[sms["spam"] == 1].mean()
print(round(p_call_spam * p_spam / has_call.mean(), 3))
Why it is called naive#
Naive Bayes multiplies the evidence from each clue as if the clues were independent.
def lift(flag):
return flag[sms["spam"] == 1].mean() / flag[sms["spam"] == 0].mean()
has_txt = sms["message"].str.contains(r"\btxt\b", case=False)
odds = p_spam / (1 - p_spam) * lift(has_free) * lift(has_txt)
print(round(lift(has_free), 1), round(lift(has_txt), 1))
print(round(odds / (1 + odds), 3))
both = has_free & has_txt
print(both.sum(), round(sms.loc[both, "spam"].mean(), 3))
Gaussian Naive Bayes on message measurements#
Problem: our measurements are numbers like length, not yes-or-no clues.
from sklearn.naive_bayes import GaussianNB
gnb = GaussianNB()
gnb.fit(train[nums], train["spam"])
print(round(gnb.score(test[nums], test["spam"]), 3))
means = pd.DataFrame(gnb.theta_, index=["ham", "spam"], columns=nums)
means.round(1)
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(7, 4))
for name, grp in train.groupby("label"):
ax.hist(grp["digits"], bins=30, alpha=0.6, label=name)
ax.set_title("Digits per message, ham vs spam")
ax.set_xlabel("Digits in the message")
ax.set_ylabel("Number of messages")
ax.legend()
plt.show()
probs = gnb.predict_proba(test[nums].head(3))
print(probs.round(3))
print(test["label"].head(3).to_list())
Your turn: train GaussianNB on the digits column alone. How close does it get to 0.961?
solo = GaussianNB().fit(train[["digits"]], train["spam"])
print(round(solo.score(test[["digits"]], test["spam"]), 3))
Common mistake: giving MultinomialNB scaled features
from sklearn.naive_bayes import MultinomialNB
scaled = StandardScaler().fit_transform(train[nums])
MultinomialNB().fit(scaled, train["spam"])
mnb_nums = MultinomialNB().fit(train[nums], train["spam"])
print(round(mnb_nums.score(test[nums], test["spam"]), 3))
From words to counts#
Problem: a model needs numbers, but the real evidence is in the words.
from sklearn.feature_extraction.text import CountVectorizer
demo = CountVectorizer()
counts = demo.fit_transform(sms["message"].head(2))
words = demo.get_feature_names_out()
print(counts.shape)
pd.DataFrame(counts.toarray(), columns=words).iloc[:, :8]
vec = CountVectorizer()
Xw_train = vec.fit_transform(train["message"])
Xw_test = vec.transform(test["message"])
print(Xw_train.shape, Xw_test.shape)
print(round(Xw_train.nnz / (Xw_train.shape[0] * Xw_train.shape[1]), 4))
Train a word-based spam filter#
Problem: score every word in a message at once, and decide.
mnb = MultinomialNB()
mnb.fit(Xw_train, train["spam"])
print(round(mnb.score(Xw_test, test["spam"]), 3))
from sklearn.metrics import confusion_matrix, classification_report
pred = mnb.predict(Xw_test)
print(confusion_matrix(test["spam"], pred))
print(classification_report(test["spam"], pred,
target_names=["ham", "spam"], digits=3))
names = vec.get_feature_names_out()
log_p = mnb.feature_log_prob_
spam_lift = pd.Series(log_p[1] - log_p[0], index=names)
spam_lift.nlargest(8).round(2)
top = spam_lift.nlargest(10).sort_values()
ax = top.plot(kind="barh", figsize=(7, 4))
ax.set_title("Words most typical of spam")
ax.set_xlabel("Log probability lift, spam vs ham")
ax.set_ylabel("Word")
plt.show()
Your turn: list the 8 words that point most strongly to ham.
spam_lift.nsmallest(8).round(2)
Common mistake: fitting a new vectorizer on the test messages
Xw_bad = CountVectorizer().fit_transform(test["message"])
mnb.predict(Xw_bad)
new_msgs = test["message"].head(3)
print(mnb.predict(vec.transform(new_msgs)))
print(test["spam"].head(3).to_list())
Smoothing: the zero-frequency problem#
Problem: one word never seen in ham would make the ham chance exactly zero.
wc = mnb.feature_count_
only_spam = names[(wc[0] == 0) & (wc[1] >= 15)]
print(len(only_spam))
print(only_spam[:8])
import numpy as np
i = vec.vocabulary_[only_spam[0]]
print(wc[0, i] / wc[0].sum())
print(f"{np.exp(log_p[0, i]):.7f}")
from sklearn.model_selection import cross_val_score
for a in [0.01, 0.1, 0.5, 1.0]:
model = MultinomialNB(alpha=a)
f1 = cross_val_score(model, Xw_train, train["spam"], cv=5,
scoring="f1").mean()
print(a, round(f1, 3))
Quick check: which alpha would you keep, and why not pick it on the test set?
best = MultinomialNB(alpha=0.01)
best.fit(Xw_train, train["spam"])
print(round(best.score(Xw_test, test["spam"]), 3))
Mini-project: a filter that doesn't lose real messages#
Question: how strict must the filter be so almost no genuine message is blocked?
from sklearn.model_selection import cross_val_predict
pipe = make_pipeline(CountVectorizer(), MultinomialNB(alpha=0.01))
p_cv = cross_val_predict(pipe, train["message"], train["spam"],
cv=5, method="predict_proba")[:, 1]
is_ham = (train["spam"] == 0).to_numpy()
for t in [0.5, 0.9, 0.99]:
blocked = ((p_cv >= t) & is_ham).sum()
caught = (p_cv[~is_ham] >= t).mean()
print(t, blocked, round(caught, 3))
pipe.fit(train["message"], train["spam"])
p_test = pipe.predict_proba(test["message"])[:, 1]
flag = (p_test >= 0.99).astype(int)
print(confusion_matrix(test["spam"], flag))
sp = test.assign(flag=flag).query("spam == 1")
sp.groupby("flag")[["length", "digits"]].median()
blocked = ((flag == 1) & (test["spam"] == 0)).sum()
caught = flag[test["spam"] == 1].mean()
print(f"Blocked ham: {blocked}, spam caught: {caught:.1%}")
Recap#
- Bayes' rule: posterior = prior x likelihood / evidence
- Naive means clues are treated as independent and multiplied
- GaussianNB for measurements; MultinomialNB for counts, never negatives
- Fit the vectorizer on training text only; transform the rest
- Tune alpha and the threshold with cross-validation, not the test set
Challenges#
- Add the three measurements as extra columns. Does accuracy improve?
- Try CountVectorizer(ngram_range=(1, 2)) so word pairs count too.
- Find the threshold where blocked ham first rises above zero.
What's next#
Lesson 17: Logistic Regression for Classification Learn weights for every feature and read them as odds.
Data Mining 17: Logistic Regression for Classification#
In this lesson you'll be able to:
- Explain how the sigmoid turns a score into a probability
- Clean and encode real churn data for a linear model
- Train and evaluate LogisticRegression on a held-out test set
- Read coefficients as odds ratios a manager understands
- Build a retention call list and measure its lift Dataset: IBM Telco customer churn, 7,043 customers (data/mining/telco_churn.csv)
Warm-up: Naive Bayes on churn#
Lesson 16 built Naive Bayes classifiers. Here's one on today's data.
import pandas as pd
df = pd.read_csv("telco_churn.csv")
df["churn"] = (df["Churn"] == "Yes").astype(int)
print(df.shape)
print(round(df["churn"].mean(), 3))
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB
train, test = train_test_split(
df, test_size=0.25, stratify=df["churn"], random_state=42)
two = ["tenure", "MonthlyCharges"]
nb = GaussianNB().fit(train[two], train["churn"])
print(round(nb.score(test[two], test["churn"]), 3))
From a score to a probability#
Problem: a straight line can predict -0.3 or 1.4, but a probability must stay between 0 and 1.
- Add up the features, each times a weight, into a score z
- Squash z through the S-shaped sigmoid: p = 1 / (1 + e^-z)
- Predict churn when p is 0.5 or more
bands = pd.cut(df["tenure"], bins=[-1, 12, 24, 48, 72],
labels=["0-12", "13-24", "25-48", "49-72"])
df.groupby(bands, observed=True)["churn"].mean().round(3)
from sklearn.linear_model import LogisticRegression
one = LogisticRegression()
one.fit(train[["tenure"]], train["churn"])
print(one.coef_.round(4), one.intercept_.round(4))
import matplotlib.pyplot as plt
rate = train.groupby("tenure")["churn"].mean()
grid = rate.index.to_frame(index=False)
curve = one.predict_proba(grid)[:, 1]
fig, ax = plt.subplots(figsize=(7, 4))
ax.scatter(rate.index, rate, s=12, label="Actual churn rate")
ax.plot(grid["tenure"], curve, color="red", label="Logistic curve")
ax.set_title("Churn probability falls with tenure")
ax.set_xlabel("Tenure (months)")
ax.set_ylabel("Share who churned")
ax.legend()
import numpy as np
z = one.intercept_[0] + one.coef_[0, 0] * 12
print(round(1 / (1 + np.exp(-z)), 3))
row = test.loc[test["tenure"] == 12, ["tenure"]].head(1)
print(one.predict_proba(row)[:, 1].round(3))
Quick check: what churn chance does the curve give a loyal customer at 72 months?
row72 = test.loc[test["tenure"] == 72, ["tenure"]].head(1)
print(one.predict_proba(row72)[:, 1].round(3))
Your turn: fit a one-feature model on MonthlyCharges. Do higher bills raise churn?
bill = LogisticRegression()
bill.fit(train[["MonthlyCharges"]], train["churn"])
print(bill.coef_.round(4))
Prepare real data for the model#
Problem: the model needs complete numbers, but this table holds text and blanks.
print(df["TotalCharges"].dtype)
print(df["TotalCharges"].head(3).to_list())
Common mistake: converting a text column that hides blanks
pd.to_numeric(df["TotalCharges"])
tc = pd.to_numeric(df["TotalCharges"], errors="coerce")
blank = tc.isna()
print(blank.sum())
print(df.loc[blank, "tenure"].unique())
df["TotalCharges"] = tc.fillna(0)
print(df["TotalCharges"].isna().sum())
Quick check: Contract has three levels. How many 0 or 1 columns does it really need?
print(pd.get_dummies(df["Contract"]).shape)
print(pd.get_dummies(df["Contract"], drop_first=True).shape)
nums = ["SeniorCitizen", "tenure", "MonthlyCharges", "TotalCharges"]
cats = ["Contract", "InternetService", "PaymentMethod",
"PaperlessBilling", "Partner", "Dependents"]
X = pd.get_dummies(df[nums + cats], drop_first=True, dtype=int)
print(X.shape)
print(list(X.columns[4:7]))
X_train, X_test = X.loc[train.index], X.loc[test.index]
y_train, y_test = train["churn"], test["churn"]
print(X_train.shape, X_test.shape)
Your turn: which Contract level did drop_first remove? Check the levels.
print(sorted(df["Contract"].unique()))
Train and evaluate the full model#
Problem: does the full model beat a simple guess on customers it hasn't seen?
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
model = make_pipeline(StandardScaler(),
LogisticRegression(max_iter=1000))
model.fit(X_train, y_train)
print(round(model.score(X_test, y_test), 3))
print(round(1 - y_test.mean(), 3))
from sklearn.metrics import confusion_matrix, classification_report
pred = model.predict(X_test)
print(confusion_matrix(y_test, pred))
print(classification_report(y_test, pred, digits=3))
probs = model.predict_proba(X_test.head(5))[:, 1]
print(probs.round(3))
print(y_test.head(5).to_list())
p_all = model.predict_proba(X_test)[:, 1]
fig, ax = plt.subplots(figsize=(7, 4))
ax.hist(p_all[y_test == 0], bins=20, alpha=0.6, label="Stayed")
ax.hist(p_all[y_test == 1], bins=20, alpha=0.6, label="Left")
ax.set_title("Predicted churn probability by outcome")
ax.set_xlabel("Predicted probability of leaving")
ax.set_ylabel("Number of customers")
ax.legend()
print(round(np.median(p_all[y_test == 0]), 3))
print(round(np.median(p_all[y_test == 1]), 3))
Common mistake: scoring a raw customer row that was never encoded
model.predict_proba(test.head(1))
model.predict_proba(X_test.head(1)).round(3)
Read the coefficients as odds#
Problem: a manager asks why customers leave, not just who.
coefs = pd.Series(model[-1].coef_[0], index=X.columns)
coefs.sort_values().round(2)
Quick check: tenure lowers churn, yet TotalCharges raises it. How can both be true?
df[["tenure", "MonthlyCharges", "TotalCharges"]].corr().round(2)
X_train2 = X_train.drop(columns="TotalCharges")
X_test2 = X_test.drop(columns="TotalCharges")
model2 = make_pipeline(StandardScaler(),
LogisticRegression(max_iter=1000))
model2.fit(X_train2, y_train)
print(round(model2.score(X_test2, y_test), 3))
coefs2 = pd.Series(model2[-1].coef_[0], index=X_train2.columns)
print(round(coefs2["tenure"], 2))
- Odds = p / (1 - p): a 0.2 chance is odds of 1 to 4, or 0.25
- Each weight multiplies the odds by exp(weight)
p = model2.predict_proba(X_test2.head(1))[0, 1]
print(round(p, 3), round(p / (1 - p), 3))
per_unit = model2[-1].coef_[0] / model2[0].scale_
odds = pd.Series(np.exp(per_unit), index=X_train2.columns)
odds.sort_values().round(3)
ax = odds.sort_values().plot(kind="barh", figsize=(7, 5))
ax.axvline(1, color="grey", linestyle="--")
ax.set_title("What moves the odds of churn")
ax.set_xlabel("Odds ratio (below 1 protects, above 1 risks)")
ax.set_ylabel("Feature")
plt.show()
Your turn: tenure's odds ratio is per month. What is it for a whole year?
print(round(odds["tenure"] ** 12, 3))
Regularisation with C#
Problem: a model can lean too hard on noisy features. C sets how far it may.
from sklearn.model_selection import cross_val_score
for c in [0.01, 0.1, 1, 10]:
lr = LogisticRegression(C=c, max_iter=1000)
pipe = make_pipeline(StandardScaler(), lr)
f1 = cross_val_score(pipe, X_train2, y_train, cv=5,
scoring="f1").mean()
print(c, round(f1, 3))
Quick check: is it worth moving away from the default C of 1?
strict = make_pipeline(StandardScaler(),
LogisticRegression(C=0.01, max_iter=1000))
strict.fit(X_train2, y_train)
print(round(np.abs(strict[-1].coef_).sum(), 2))
print(round(np.abs(model2[-1].coef_).sum(), 2))
Mini-project: who gets a retention call?#
Question: the team can call 1 test customer in 5. How many leavers does the list reach?
p_test = model2.predict_proba(X_test2)[:, 1]
cut = np.quantile(p_test, 0.8)
call = p_test >= cut
print(call.sum(), round(cut, 3))
hit = y_test[call].mean()
base = y_test.mean()
print(round(hit, 3), round(base, 3), round(hit / base, 2))
print(y_test[call].sum(), y_test.sum())
test.loc[call, "Contract"].value_counts(normalize=True).round(3)
groups = pd.qcut(p_test, 5, labels=["1 low", "2", "3", "4", "5 high"])
by_group = y_test.groupby(groups, observed=True).mean()
ax = by_group.plot(kind="bar", rot=0, figsize=(7, 4))
ax.set_title("Real churn rate by predicted risk group")
ax.set_xlabel("Risk group (fifths of customers)")
ax.set_ylabel("Share who actually left")
print(by_group.round(3))
reached = y_test[call].sum() / y_test.sum()
print(f"Calling 20% of customers reaches {reached:.0%} of leavers,")
print(f"{hit / base:.1f} times better than calling at random.")
Recap#
- The sigmoid squashes a weighted score into a probability
- Coerce messy text to numbers, then fill gaps with an honest value
- Scale inside a pipeline; score rows as tables, double brackets
- exp(weight) is an odds ratio; drop copycat columns to read them
- Tune C with cross-validation; rank by probability to target action
Challenges#
- Add TechSupport and OnlineSecurity. Do they protect against churn?
- Try class_weight='balanced'. What happens to recall for leavers?
- Build a call list for the riskiest 10%. What is its lift?
What's next#
Lesson 18: Support Vector Machines (SVM) Find the widest boundary between classes, and bend it with kernels.
Data Mining 18: Support Vector Machines (SVM)#
In this lesson you'll be able to:
- Explain margins and support vectors with a real picture
- Handle hidden missing values and scale features for an SVM
- Compare linear and RBF kernels
- Tune C and the kernel with cross-validation, then test once
- Turn SVM scores into a screening rule for a clinic Dataset: Pima Indians diabetes, 768 women (data/mining/pima_diabetes.csv)
Warm-up: logistic regression on diabetes#
Lesson 17 built logistic regression. Here's one on today's data.
import pandas as pd
df = pd.read_csv("pima_diabetes.csv")
print(df.shape)
print(round(df["outcome"].mean(), 3))
from sklearn.model_selection import train_test_split
train, test = train_test_split(
df, test_size=0.25, stratify=df["outcome"], random_state=42)
feats = list(df.columns[:-1])
print(len(train), len(test))
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
lr = make_pipeline(StandardScaler(), LogisticRegression())
lr.fit(train[feats], train["outcome"])
print(round(lr.score(test[feats], test["outcome"]), 3))
Find the missing values hiding as zeros#
Problem: a blood pressure of 0 isn't a measurement. It's a missing one.
hidden = ["glucose", "blood_pressure", "skin_thickness", "insulin",
"bmi"]
(df[hidden] == 0).sum()
import numpy as np
X = df[feats].copy()
X[hidden] = X[hidden].replace(0, np.nan)
X_train, X_test = X.loc[train.index], X.loc[test.index]
y_train, y_test = train["outcome"], test["outcome"]
print(X_train.isna().sum().sum())
print(round(df["insulin"].mean(), 1))
print(round(X["insulin"].mean(), 1))
Common mistake: fitting an SVM on data that still has gaps
from sklearn.svm import SVC
SVC().fit(X_train, y_train)
from sklearn.impute import SimpleImputer
from sklearn.model_selection import cross_val_score
svm = make_pipeline(SimpleImputer(strategy="median"),
StandardScaler(), SVC())
f1 = cross_val_score(svm, X_train, y_train, cv=5, scoring="f1")
print(round(f1.mean(), 3))
Quick check: why fill medians inside the pipeline, not on the whole table first?
print(X_train["insulin"].median())
print(X_test["insulin"].median())
The widest street: margins and support vectors#
Problem: many lines can split two classes. Which one should we trust?
- The SVM picks the boundary with the widest margin, a street between classes
- Patients on the edge of the street are the support vectors
- C sets how many patients may stand inside the street
two = ["glucose", "bmi"]
d2 = X_train[two].dropna()
y2 = y_train.loc[d2.index]
lin2 = make_pipeline(StandardScaler(), SVC(kernel="linear"))
lin2.fit(d2, y2)
print(len(d2), lin2[-1].n_support_)
print(lin2.decision_function(d2.head(3)).round(2))
print(y2.head(3).to_list())
import matplotlib.pyplot as plt
from sklearn.inspection import DecisionBoundaryDisplay
fig, ax = plt.subplots(figsize=(7, 5))
DecisionBoundaryDisplay.from_estimator(
lin2, d2, response_method="decision_function", ax=ax,
plot_method="contour", levels=[-1, 0, 1], colors="k",
linestyles=["--", "-", "--"])
ax.scatter(d2["glucose"], d2["bmi"], c=y2, cmap="coolwarm", s=10)
ax.set_title("Linear SVM: the street between the classes")
ax.set_xlabel("Glucose")
ax.set_ylabel("BMI")
Quick check: if C gets smaller, does the street get wider or narrower?
for c in [0.01, 1, 100]:
lin = SVC(kernel="linear", C=c)
m = make_pipeline(StandardScaler(), lin).fit(d2, y2)
print(c, m[-1].n_support_.sum())
Your turn: fit an RBF SVM on glucose and BMI. How many support vectors does it keep?
rbf2 = make_pipeline(StandardScaler(), SVC(kernel="rbf"))
rbf2.fit(d2, y2)
print(rbf2[-1].n_support_, rbf2[-1].n_support_.sum())
Why scaling matters for SVMs#
Problem: the SVM measures distance, so big-number features drown out small ones.
X_train[["insulin", "diabetes_pedigree"]].describe().round(2)
raw = make_pipeline(SimpleImputer(strategy="median"), SVC())
for name, m in [("unscaled", raw), ("scaled", svm)]:
s = cross_val_score(m, X_train, y_train, cv=5, scoring="f1")
print(name, round(s.mean(), 3))
Your turn: which raw feature has the biggest spread? Sort the standard deviations.
X_train.std().round(1).sort_values()
Kernels: bending the boundary#
Problem: sometimes no straight street can separate the classes.
- Linear kernel: a straight boundary
- RBF kernel: similarity to nearby patients, so the boundary can curve
- gamma sets how far each patient's influence reaches
fig, axes = plt.subplots(1, 2, figsize=(10, 4))
for ax, k in zip(axes, ["linear", "rbf"]):
m = make_pipeline(StandardScaler(), SVC(kernel=k)).fit(d2, y2)
DecisionBoundaryDisplay.from_estimator(m, d2, ax=ax, alpha=0.3,
response_method="predict", cmap="coolwarm")
ax.scatter(d2["glucose"], d2["bmi"], c=y2, cmap="coolwarm", s=8)
ax.set_title(f"{k} kernel")
ax.set_xlabel("Glucose")
ax.set_ylabel("BMI")
plt.tight_layout()
for k in ["linear", "rbf", "poly"]:
m = make_pipeline(SimpleImputer(strategy="median"),
StandardScaler(), SVC(kernel=k))
s = cross_val_score(m, X_train, y_train, cv=5, scoring="f1")
print(k, round(s.mean(), 3))
Your turn: set gamma=1 on the RBF SVM. Compare training accuracy with cross-validated accuracy.
wiggly = make_pipeline(SimpleImputer(strategy="median"),
StandardScaler(), SVC(gamma=1))
wiggly.fit(X_train, y_train)
print(round(wiggly.score(X_train, y_train), 3))
print(round(cross_val_score(wiggly, X_train, y_train, cv=5).mean(), 3))
gammas = [0.01, 0.1, 1, 10]
fit_acc, cv_acc = [], []
for g in gammas:
m = make_pipeline(SimpleImputer(strategy="median"),
StandardScaler(), SVC(gamma=g))
fit_acc.append(m.fit(X_train, y_train).score(X_train, y_train))
cv_acc.append(cross_val_score(m, X_train, y_train, cv=5).mean())
ax = pd.DataFrame({"training": fit_acc, "cross-validated": cv_acc},
index=gammas).plot(marker="o", logx=True)
ax.set_title("Larger gamma memorises the training patients")
ax.set_xlabel("gamma (log scale)")
ax.set_ylabel("Accuracy")
Tune C and the kernel, then test once#
Problem: choose the settings without ever peeking at the test set.
from sklearn.model_selection import GridSearchCV
options = dict(svc__kernel=["linear", "rbf"], svc__C=[0.1, 1, 10])
grid = GridSearchCV(svm, options, cv=5, scoring="f1")
grid.fit(X_train, y_train)
print(grid.best_params_, round(grid.best_score_, 3))
res = pd.DataFrame(grid.cv_results_)
cols = ["param_svc__kernel", "param_svc__C", "mean_test_score"]
res.sort_values("rank_test_score")[cols].head(4).round(3)
Quick check: the top three differ by 0.004. Is that difference real?
top = res.sort_values("rank_test_score").head(3)
top[["mean_test_score", "std_test_score"]].round(3)
best = grid.best_estimator_
print(round(best.score(X_test, y_test), 3))
print(round(lr.score(test[feats], test["outcome"]), 3))
from sklearn.metrics import confusion_matrix, classification_report
pred = best.predict(X_test)
print(confusion_matrix(y_test, pred))
print(classification_report(y_test, pred, digits=3))
Common mistake: asking an SVM for probabilities
best.predict_proba(X_test.head(3))
print(best.decision_function(X_test.head(3)).round(2))
print(y_test.head(3).to_list())
Mini-project: a screening rule for a clinic#
Question: catch at least 80% of diabetic patients. How many must be sent for a test?
from sklearn.model_selection import cross_val_predict
s_cv = cross_val_predict(best, X_train, y_train, cv=5,
method="decision_function")
cut = np.quantile(s_cv[(y_train == 1).to_numpy()], 0.2)
print(round(cut, 3))
fig, ax = plt.subplots(figsize=(7, 4))
ax.hist(s_cv[(y_train == 0).to_numpy()], bins=30, alpha=0.6,
label="No diabetes")
ax.hist(s_cv[(y_train == 1).to_numpy()], bins=30, alpha=0.6,
label="Diabetes")
ax.axvline(0, color="grey", linestyle="--", label="Default")
ax.axvline(cut, color="black", label="Screening cut-off")
ax.set_title("Moving the cut-off left catches more diabetes")
ax.set_xlabel("SVM decision score")
ax.set_ylabel("Number of patients")
ax.legend()
s_test = best.decision_function(X_test)
has = (y_test == 1).to_numpy()
for name, line in [("default", 0), ("screening", cut)]:
flag = s_test >= line
print(name, round(flag[has].mean(), 3), round(flag.mean(), 3))
flag = s_test >= cut
print(confusion_matrix(y_test, flag.astype(int)))
print(X_test.loc[flag, "glucose"].median())
print(X_test.loc[~flag, "glucose"].median())
tests = flag.sum()
found = flag[has].sum()
print(tests, found, round(tests / found, 2))
caught = flag[has].mean()
sent = flag.mean()
print(f"Screening catches {caught:.0%} of diabetic patients")
print(f"by sending {sent:.0%} of patients for a test.")
Recap#
- An SVM picks the boundary with the widest margin; support vectors hold it
- Turn impossible zeros into NaN and impute inside the pipeline
- Always scale: SVMs measure distance
- Kernels bend the boundary; C and gamma control how tightly it fits
- Tune with GridSearchCV, test once, and move the cut-off to match the cost
Challenges#
- Add gamma values [0.01, 0.1, 1] to the grid for the RBF kernel.
- Find the cut-off that catches 90% of diabetic training patients.
- Try SVC(class_weight='balanced'). How does recall change?
What's next#
Lesson 19: Introduction to Neural Networks for Data Mining Stack simple units into layers that learn their own features.
Data Mining 19: Introduction to Neural Networks for Data Mining#
In this lesson you'll be able to:
- Explain a neuron, a layer and training in plain words
- Train a neural network with scikit-learn's MLPClassifier
- Look inside a trained network and redo one prediction by hand
- Spot overfitting and tame it with regularisation
- Score new patients and compare the network with simpler models Dataset: Pima Indians diabetes, 768 women (data/mining/pima_diabetes.csv)
Warm-up: the SVM from lesson 18#
Lesson 18 drew a boundary with a support vector machine. Let's set the score to beat.
import numpy as np
import pandas as pd
df = pd.read_csv("pima_diabetes.csv")
print(df.shape)
df.head()
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score
quick_svm = make_pipeline(StandardScaler(), SVC())
raw_X = df.drop(columns="outcome")
cv = cross_val_score(quick_svm, raw_X, df["outcome"], cv=5)
print(cv.round(3), round(cv.mean(), 3))
Prepare data a network can learn from#
Problem: zeros that really mean 'not measured', and features on very different scales.
print(df["outcome"].value_counts())
print((df == 0).sum())
fake_zero = ["glucose", "blood_pressure", "skin_thickness",
"insulin", "bmi"]
df[fake_zero] = df[fake_zero].replace(0, np.nan)
df[fake_zero].isna().sum()
new_patients = df.tail(5)
df = df.iloc[:-5]
X = df.drop(columns="outcome")
y = df["outcome"]
print(X.shape, y.shape)
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=42, stratify=y)
print(len(X_train), len(X_test))
print(round(y_train.mean(), 3), round(y_test.mean(), 3))
Common mistake: feeding missing values straight into a network
from sklearn.neural_network import MLPClassifier
net = MLPClassifier(random_state=42)
net.fit(X_train, y_train)
from sklearn.impute import SimpleImputer
prep = make_pipeline(SimpleImputer(strategy="median"),
StandardScaler())
X_train_ready = prep.fit_transform(X_train)
print(np.isnan(X_train_ready).sum())
print(X_train_ready.mean(axis=0).round(2) + 0)
print(X_train_ready.std(axis=0).round(2))
Quick check: why does scaling matter so much for a network?
X_train[["insulin", "diabetes_pedigree"]].describe().round(2)
Train your first neural network#
Problem: let the model invent its own useful combinations of features.
import matplotlib.pyplot as plt
colours = y_train.map({0: "tab:blue", 1: "tab:red"})
plt.figure(figsize=(7, 4))
plt.scatter(X_train["glucose"], X_train["bmi"], c=colours, s=12)
plt.title("Glucose vs BMI: no single clean dividing line")
plt.xlabel("Glucose")
plt.ylabel("BMI")
plt.show()
- Neuron: weighted sum of inputs, then an on-off style switch
- Layer: several neurons side by side, each finding a pattern
- Training: nudge every weight to shrink the prediction error
mlp = make_pipeline(
SimpleImputer(strategy="median"), StandardScaler(),
MLPClassifier(hidden_layer_sizes=(8,), max_iter=2000,
random_state=42))
mlp.fit(X_train, y_train)
print("train:", round(mlp.score(X_train, y_train), 3))
print("test: ", round(mlp.score(X_test, y_test), 3))
svm = make_pipeline(
SimpleImputer(strategy="median"), StandardScaler(),
SVC(random_state=42))
svm.fit(X_train, y_train)
print("SVM test:", round(svm.score(X_test, y_test), 3))
probs = mlp.predict_proba(X_test.head())[:, 1]
print(probs.round(3))
print(mlp.predict(X_test.head()))
print(y_test.head().to_numpy())
net = mlp[-1]
plt.figure(figsize=(7, 4))
plt.plot(net.loss_curve_)
plt.title("How the network learned: training loss per round")
plt.xlabel("Training round (epoch)")
plt.ylabel("Loss (lower is better)")
plt.show()
Quick check: the network scored higher on training than on test patients. Is that a warning sign?
gap = mlp.score(X_train, y_train) - mlp.score(X_test, y_test)
print("gap:", round(gap, 3))
Your turn#
Train a network with just 4 hidden neurons. Report its train and test accuracy.
mlp4 = make_pipeline(
SimpleImputer(strategy="median"), StandardScaler(),
MLPClassifier(hidden_layer_sizes=(4,), max_iter=2000,
random_state=42))
mlp4.fit(X_train, y_train)
print("train:", round(mlp4.score(X_train, y_train), 3))
print("test: ", round(mlp4.score(X_test, y_test), 3))
Look inside the network#
Problem: a network feels like a black box. It is really just weights and sums.
for i, w in enumerate(net.coefs_):
print("weights", i, w.shape)
for i, b in enumerate(net.intercepts_):
print("biases ", i, b.shape)
Quick check: how many numbers did training adjust in total?
n_weights = sum(w.size for w in net.coefs_)
n_biases = sum(b.size for b in net.intercepts_)
print(n_weights, n_biases, n_weights + n_biases)
w_in = pd.DataFrame(net.coefs_[0], index=X.columns)
w_in.columns = [f"n{i}" for i in range(8)]
w_in.round(2)
strength = w_in.abs().sum(axis=1)
strength.sort_values(ascending=False).round(2)
A forward pass, step by step:
- Hidden layer: weighted sum + bias, then ReLU (negatives become 0)
- Output: weighted sum of hidden values + bias, then the logistic squash
x = prep.transform(X_test.head(1))
hidden = np.maximum(0, x @ net.coefs_[0] + net.intercepts_[0])
print(hidden.round(2))
z = hidden @ net.coefs_[1] + net.intercepts_[1]
p_hand = 1 / (1 + np.exp(-z))
print("by hand:", p_hand.round(4))
print("sklearn:", mlp.predict_proba(X_test.head(1))[:, 1].round(4))
Size, overfitting and regularisation#
Problem: bigger networks can memorise the training patients instead of learning.
X_fit, X_val, y_fit, y_val = train_test_split(
X_train, y_train, test_size=0.25, random_state=42,
stratify=y_train)
print(len(X_fit), len(X_val), len(X_test))
def build(sizes, alpha=0.0001):
return make_pipeline(
SimpleImputer(strategy="median"), StandardScaler(),
MLPClassifier(hidden_layer_sizes=sizes, alpha=alpha,
max_iter=2000, random_state=42))
rows = []
for sizes in [(2,), (8,), (64, 64)]:
model = build(sizes).fit(X_fit, y_fit)
rows.append({"layers": str(sizes),
"fit": model.score(X_fit, y_fit),
"validation": model.score(X_val, y_val)})
sizes_df = pd.DataFrame(rows).round(3)
sizes_df
ax = sizes_df.plot(x="layers", kind="bar", rot=0, figsize=(7, 4))
ax.set_title("Bigger networks memorise: fit vs validation accuracy")
ax.set_xlabel("Hidden layers")
ax.set_ylabel("Accuracy")
ax.set_ylim(0.5, 1.0)
plt.show()
- alpha adds a penalty for large weights (L2 regularisation)
- early_stopping stops training when a validation slice stops improving
for a in [0.0001, 1, 3, 10]:
model = build((64, 64), alpha=a).fit(X_fit, y_fit)
tr = model.score(X_fit, y_fit)
va = model.score(X_val, y_val)
print(f"alpha={a}: fit {tr:.3f}, validation {va:.3f}")
Quick check: which alpha would you pick, and why not decide using the test set?
final = build((64, 64), alpha=1).fit(X_train, y_train)
print("test:", round(final.score(X_test, y_test), 3))
Your turn#
Train the (64, 64) network with early_stopping=True on X_fit. Compare fit and validation accuracy.
early = make_pipeline(
SimpleImputer(strategy="median"), StandardScaler(),
MLPClassifier(hidden_layer_sizes=(64, 64), early_stopping=True,
max_iter=2000, random_state=42))
early.fit(X_fit, y_fit)
print("fit: ", round(early.score(X_fit, y_fit), 3))
print("validation:", round(early.score(X_val, y_val), 3))
Score new patients#
Problem: use the trained network on people it has never seen.
Common mistake: passing the new rows with the answer column still attached
final.predict(new_patients)
new_X = new_patients.drop(columns="outcome")
risk = final.predict_proba(new_X)[:, 1]
report = new_X[["glucose", "bmi", "age"]].copy()
report["risk"] = risk.round(3)
report["actual"] = new_patients["outcome"]
report
Quick check: which patient would you call in first, and would the default 0.5 cut-off flag her?
top = report["risk"].idxmax()
print(report.loc[top])
print("flagged at 0.5:", report.loc[top, "risk"] >= 0.5)
test_pred = final.predict(X_test)
pd.crosstab(y_test, test_pred, rownames=["actual"],
colnames=["predicted"])
Mini-project: is the network worth it here?#
Compare logistic regression, SVM and a regularised network with 5-fold cross-validation on the training set.
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_validate
models = {
"logistic": make_pipeline(SimpleImputer(strategy="median"),
StandardScaler(),
LogisticRegression(max_iter=1000)),
"svm": svm,
"network": build((8,), alpha=1),
}
scores = {}
for name, model in models.items():
cv = cross_validate(model, X_train, y_train, cv=5,
scoring=["accuracy", "recall"])
scores[name] = {"accuracy": cv["test_accuracy"].mean(),
"recall": cv["test_recall"].mean()}
results = pd.DataFrame(scores).T.round(3)
results
ax = results.plot(kind="bar", rot=0, figsize=(7, 4))
ax.set_title("Cross-validated scores on the Pima training set")
ax.set_xlabel("Model")
ax.set_ylabel("Average score over 5 folds")
ax.set_ylim(0.4, 0.9)
plt.show()
best = results["accuracy"].idxmax()
edge = results.loc["network"] - results.loc["logistic"]
print("most accurate:", best)
print(edge.round(3))
Recap#
- A neuron is a weighted sum plus a switch; layers stack neurons
- Networks need complete, scaled inputs: use a pipeline
- A trained network is just weights you can inspect and recompute
- Bigger is not better: watch the train-test gap, use alpha
- On small tables, simple models often match a network
Challenges#
- Try
hidden_layer_sizes=(16, 8)withalpha=1. Does cross-validated recall improve? - Swap
activationto"tanh"and compare the loss curves. - Rebuild the hand calculation for the second test patient.
What's next#
Lesson 20: Evaluating Classifiers - Accuracy, Precision, Recall, F1-Score Accuracy alone hides costly mistakes. Learn the metrics that don't.
Data Mining 20: Evaluating Classifiers- Accuracy, Precision, Recall, F1-Score#
In this lesson you'll be able to:
- Explain why accuracy alone can hide a useless model
- Read a confusion matrix and name its four cells
- Calculate precision, recall and F1 by hand and with scikit-learn
- Read a classification report, including macro and weighted averages
- Move the decision threshold to hit a recall target, honestly Dataset: Pima Indians diabetes, 768 women (data/mining/pima_diabetes.csv)
Warm-up: the network from lesson 19#
Rebuild lesson 19's prepared data and its regularised network.
import numpy as np
import pandas as pd
df = pd.read_csv("pima_diabetes.csv")
fake_zero = ["glucose", "blood_pressure", "skin_thickness",
"insulin", "bmi"]
df[fake_zero] = df[fake_zero].replace(0, np.nan)
X = df.drop(columns="outcome")
y = df["outcome"]
print(X.shape, round(y.mean(), 3))
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=42, stratify=y)
print(len(X_train), len(X_test), y_test.sum())
from sklearn.pipeline import make_pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.neural_network import MLPClassifier
net = make_pipeline(SimpleImputer(strategy="median"),
StandardScaler(),
MLPClassifier(hidden_layer_sizes=(8,), alpha=1,
max_iter=2000, random_state=42))
net.fit(X_train, y_train)
print("network accuracy:", round(net.score(X_test, y_test), 3))
Why accuracy alone can mislead#
Problem: a high accuracy can come from a model that never finds a single case.
from sklearn.linear_model import LogisticRegression
model = make_pipeline(SimpleImputer(strategy="median"),
StandardScaler(),
LogisticRegression(max_iter=1000))
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
print(y_pred[:10])
print(y_test.head(10).to_numpy())
- Accuracy = correct predictions / all predictions
from sklearn.metrics import accuracy_score
acc = accuracy_score(y_test, y_pred)
by_hand = (y_pred == y_test).mean()
print(round(acc, 3), round(by_hand, 3))
Quick check: what accuracy would a 'model' get by telling every woman she does NOT have diabetes?
from sklearn.dummy import DummyClassifier
lazy = DummyClassifier(strategy="most_frequent")
lazy.fit(X_train, y_train)
lazy_pred = lazy.predict(X_test)
print("lazy accuracy:", round(accuracy_score(y_test, lazy_pred), 3))
print("cases found:", lazy_pred.sum())
print(y_test.value_counts(normalize=True).round(3))
Count the four outcomes with a confusion matrix#
Problem: see exactly which kinds of mistakes the model makes.
- True positive (TP): has diabetes, flagged
- False negative (FN): has diabetes, missed
- False positive (FP): healthy, flagged (false alarm)
- True negative (TN): healthy, not flagged
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_test, y_pred)
print(cm)
labels = pd.DataFrame(cm, index=["actual no", "actual yes"],
columns=["predicted no", "predicted yes"])
labels
Common mistake: unpacking the matrix into four names without flattening it
tn, fp, fn, tp = confusion_matrix(y_test, y_pred)
tn, fp, fn, tp = confusion_matrix(y_test, y_pred).ravel()
print("TN", tn, "FP", fp, "FN", fn, "TP", tp)
print(round((tp + tn) / (tp + tn + fp + fn), 3))
import matplotlib.pyplot as plt
from sklearn.metrics import ConfusionMatrixDisplay
disp = ConfusionMatrixDisplay(cm, display_labels=["no", "yes"])
disp.plot(cmap="Blues")
plt.title("Logistic regression on 192 test patients")
plt.xlabel("Predicted diabetes")
plt.ylabel("Actual diabetes")
plt.show()
Your turn#
Build the confusion matrix for the lazy model's predictions, lazy_pred. Which cell holds every diabetic woman?
lazy_cm = confusion_matrix(y_test, lazy_pred)
pd.DataFrame(lazy_cm, index=labels.index, columns=labels.columns)
Precision and recall: two different questions#
Problem: measure false alarms and missed cases separately.
- Precision = TP / (TP + FP): when we say yes, how often are we right?
- Recall = TP / (TP + FN): of the real cases, how many did we catch?
from sklearn.metrics import precision_score, recall_score
print(round(tp / (tp + fp), 3), round(precision_score(y_test, y_pred), 3))
print(round(tp / (tp + fn), 3), round(recall_score(y_test, y_pred), 3))
Quick check: for a diabetes screening test, which mistake is worse, a false alarm or a missed case?
print("missed cases:", fn, "of", tp + fn)
print("false alarms:", fp, "of", tp + fp, "flagged")
print(round(recall_score(y_test, y_pred, pos_label=0), 3))
print(round(tn / (tn + fp), 3))
Common mistake: passing probabilities where labels are expected
probs = model.predict_proba(X_test)[:, 1]
recall_score(y_test, probs)
labels_05 = (probs >= 0.5).astype(int)
print(round(recall_score(y_test, labels_05), 3))
print((labels_05 == y_pred).all())
One number for both: F1 and the classification report#
Problem: compare models with one score that punishes a weak precision or recall.
- F1 = 2 x precision x recall / (precision + recall)
- A harmonic mean is dragged down by the weaker of the two
from sklearn.metrics import f1_score
p = precision_score(y_test, y_pred)
r = recall_score(y_test, y_pred)
print("plain average:", round((p + r) / 2, 3))
print("F1 by hand: ", round(2 * p * r / (p + r), 3))
print("f1_score: ", round(f1_score(y_test, y_pred), 3))
print("lazy F1:", f1_score(y_test, lazy_pred, zero_division=0))
from sklearn.metrics import classification_report
print(classification_report(y_test, y_pred,
target_names=["no", "yes"]))
- support: how many real cases of each class
- macro avg: plain average of the two classes
- weighted avg: average weighted by support
Your turn#
Print a classification report for the warm-up network net. Does it beat logistic regression on recall?
net_pred = net.predict(X_test)
print(classification_report(y_test, net_pred,
target_names=["no", "yes"]))
Check the scores are stable with cross-validation#
Problem: one test set of 192 women is a small sample. How much could the scores wobble?
from sklearn.model_selection import cross_validate
metrics = ["accuracy", "precision", "recall", "f1"]
cv = cross_validate(model, X_train, y_train, cv=5,
scoring=metrics)
folds = pd.DataFrame({m: cv[f"test_{m}"] for m in metrics})
folds.round(3)
folds.agg(["mean", "std"]).round(3)
Quick check: which metric wobbles most from fold to fold, and why might that be?
print(folds.std().idxmax(), round(folds.std().max(), 3))
print(int(y_train.sum() / 5))
Move the threshold to trade precision for recall#
Problem: 0.5 is just a default. Pick the cut-off that suits the real cost of mistakes.
from sklearn.model_selection import cross_val_predict
train_probs = cross_val_predict(model, X_train, y_train, cv=5,
method="predict_proba")[:, 1]
print(train_probs[:5].round(3))
rows = []
for t in [0.2, 0.3, 0.4, 0.5, 0.6, 0.7]:
flag = (train_probs >= t).astype(int)
rows.append({"threshold": t,
"precision": precision_score(y_train, flag),
"recall": recall_score(y_train, flag),
"flagged": flag.sum()})
trade = pd.DataFrame(rows).set_index("threshold").round(3)
trade
ax = trade[["precision", "recall"]].plot(marker="o", figsize=(7, 4))
ax.set_title("Lower threshold: more recall, less precision")
ax.set_xlabel("Decision threshold")
ax.set_ylabel("Score on training folds")
plt.show()
Quick check: at what threshold do precision and recall roughly meet?
gap = (trade["precision"] - trade["recall"]).abs()
print(gap.idxmin(), gap.min().round(3))
Your turn#
Add an f1 column to trade and find the threshold with the best F1.
p, r = trade["precision"], trade["recall"]
trade["f1"] = (2 * p * r / (p + r)).round(3)
print(trade["f1"])
print("best F1 at", trade["f1"].idxmax())
Mini-project: a screening rule that catches 80% of cases#
A clinic wants to catch at least 80% of diabetic women. What threshold, and what does it cost in follow-ups?
cutoffs = np.arange(0.05, 0.95, 0.01).round(2)
ok = [t for t in cutoffs
if recall_score(y_train, train_probs >= t) >= 0.8]
best_t = max(ok)
print("chosen threshold:", best_t)
screen = (probs >= best_t).astype(int)
tn2, fp2, fn2, tp2 = confusion_matrix(y_test, screen).ravel()
print("caught", tp2, "of", tp2 + fn2, "- missed", fn2)
print("follow-ups:", tp2 + fp2, "- false alarms:", fp2)
print("recall:", round(recall_score(y_test, screen), 3))
print("precision:", round(precision_score(y_test, screen), 3))
summary = pd.DataFrame({
"default 0.5": [tp, fn, tp + fp],
f"screening {best_t}": [tp2, fn2, tp2 + fp2]},
index=["caught", "missed", "follow-ups"])
summary
extra_caught = tp2 - tp
extra_tests = (tp2 + fp2) - (tp + fp)
print(f"{extra_caught} more cases caught for {extra_tests} more "
f"follow-ups: {extra_tests / extra_caught:.1f} tests each")
Recap#
- Always compare accuracy with a lazy baseline
- The confusion matrix splits mistakes into false alarms and misses
- Precision punishes false alarms; recall punishes missed cases
- F1 is high only when both are high
- Choose thresholds on training folds, then test once
Challenges#
- Repeat the 80% screening rule with the network
net. Which model needs fewer follow-ups? - Use
fbeta_scorewithbeta=2to weight recall twice as much as precision. - Write a function that prints accuracy, precision, recall and F1 for any model.
What's next#
Lesson 21: ROC Curves, AUC, and Confusion Matrix Analysis See every threshold at once, and compare models with one number.
Data Mining 21: ROC Curves, AUC, and Confusion Matrix Analysis#
In this lesson you'll be able to:
- Turn a confusion matrix into true and false positive rates
- Draw an ROC curve and read any threshold off it
- Explain AUC in plain words, and check it by hand
- Compare models fairly with ROC curves and cross-validated AUC
- Break a model's performance down by customer segment Dataset: IBM Telco customer churn, 7,043 customers (data/mining/telco_churn.csv)
Warm-up: precision and recall from lesson 20#
A fast churn model on three numeric columns, judged the lesson 20 way.
import numpy as np
import pandas as pd
df = pd.read_csv("telco_churn.csv")
print(df.shape)
print(df["Churn"].value_counts(normalize=True).round(3))
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score
few = df[["tenure", "MonthlyCharges", "SeniorCitizen"]]
churned = (df["Churn"] == "Yes").astype(int)
a, b, c, d = train_test_split(few, churned, random_state=42)
pred = make_pipeline(StandardScaler(),
LogisticRegression()).fit(a, c).predict(b)
print(round(precision_score(d, pred), 3),
round(recall_score(d, pred), 3))
Clean the churn data before modelling#
Problem: one numeric column is secretly text, and dropna can't see the gaps.
print(df.isna().sum().sum())
print(df["TotalCharges"].dtype)
print(df["TotalCharges"].head(3).to_list())
Common mistake: forcing text into numbers with astype(float)
df["TotalCharges"].astype(float)
total = pd.to_numeric(df["TotalCharges"], errors="coerce")
print(total.isna().sum())
print(df.loc[total.isna(), "tenure"].unique())
df["TotalCharges"] = total
df = df.dropna(subset=["TotalCharges"])
print(df.shape)
y = (df["Churn"] == "Yes").astype(int)
raw = df.drop(columns=["customerID", "Churn"])
X = pd.get_dummies(raw, drop_first=True, dtype=int)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=42, stratify=y)
print(X.shape, len(X_test), y_test.sum())
model = make_pipeline(StandardScaler(),
LogisticRegression(max_iter=1000))
model.fit(X_train, y_train)
probs = model.predict_proba(X_test)[:, 1]
print(probs[:6].round(3))
Confusion matrix analysis: rates, not just counts#
Problem: compare thresholds fairly when churners and stayers come in very different numbers.
- True positive rate (TPR) = TP / (TP + FN), the same as recall
- False positive rate (FPR) = FP / (FP + TN)
from sklearn.metrics import confusion_matrix
tn, fp, fn, tp = confusion_matrix(y_test, probs >= 0.5).ravel()
print("TN", tn, "FP", fp, "FN", fn, "TP", tp)
print("TPR", round(tp / (tp + fn), 3))
print("FPR", round(fp / (fp + tn), 3))
import matplotlib.pyplot as plt
from sklearn.metrics import ConfusionMatrixDisplay
ConfusionMatrixDisplay.from_predictions(
y_test, probs >= 0.5, normalize="true",
display_labels=["stayed", "churned"], cmap="Blues")
plt.title("Each row sums to 1: rates at threshold 0.5")
plt.xlabel("Predicted")
plt.ylabel("Actual")
plt.show()
def rates(t):
tn, fp, fn, tp = confusion_matrix(y_test, probs >= t).ravel()
return {"threshold": t, "TP": tp, "FP": fp,
"TPR": tp / (tp + fn), "FPR": fp / (fp + tn)}
grid = pd.DataFrame([rates(t) for t in [0.2, 0.3, 0.5, 0.7]])
grid.round(3)
Quick check: as the threshold drops from 0.7 to 0.2, what happens to both rates?
print(grid["TPR"].is_monotonic_decreasing)
print(grid["FPR"].is_monotonic_decreasing)
Your turn#
Add a precision column to grid using its TP and FP columns. Which cut-off is most precise?
grid["precision"] = grid["TP"] / (grid["TP"] + grid["FP"])
grid[["threshold", "TPR", "FPR", "precision"]].round(3)
Draw the ROC curve#
Problem: see every possible threshold at once, not four of them.
from sklearn.metrics import roc_curve
fpr, tpr, cuts = roc_curve(y_test, probs)
roc = pd.DataFrame({"cut": cuts, "FPR": fpr, "TPR": tpr})
print(len(roc) > 100)
roc.head().round(3)
plt.figure(figsize=(6, 5))
plt.plot(fpr, tpr, label="logistic regression")
plt.plot([0, 1], [0, 1], "k--", label="random guessing")
plt.scatter(fp / (fp + tn), tp / (tp + fn), color="red",
zorder=3, label="threshold 0.5")
plt.title("ROC curve: churn model on 1,758 test customers")
plt.xlabel("False positive rate (loyal customers flagged)")
plt.ylabel("True positive rate (churners caught)")
plt.legend()
plt.show()
- Bottom-left corner: threshold so high nobody is flagged
- Top-right corner: threshold so low everybody is flagged
- The closer the curve hugs the top-left, the better
Common mistake: passing the original Yes/No text labels
churn_text = df.loc[y_test.index, "Churn"]
roc_curve(churn_text, probs)
fpr2, tpr2, _ = roc_curve(churn_text, probs, pos_label="Yes")
print(np.allclose(fpr, fpr2), np.allclose(tpr, tpr2))
Your turn#
Find the first point on the curve that catches at least 80% of churners. What false positive rate does it cost?
hit = roc[roc["TPR"] >= 0.8].iloc[0]
hit.round(3)
AUC: one number for ranking quality#
Problem: summarise the whole curve in one number that doesn't depend on any threshold.
from sklearn.metrics import roc_auc_score, auc
print(round(roc_auc_score(y_test, probs), 3))
print(round(auc(fpr, tpr), 3))
AUC in plain words: pick one churner and one loyal customer at random. AUC is the chance the churner gets the higher score.
plt.figure(figsize=(7, 4))
plt.hist(probs[y_test == 0], bins=30, alpha=0.6, label="stayed")
plt.hist(probs[y_test == 1], bins=30, alpha=0.6, label="churned")
plt.title("Churners mostly score higher, but the groups overlap")
plt.xlabel("Predicted churn probability")
plt.ylabel("Number of customers")
plt.legend()
plt.show()
churn_scores = probs[y_test.to_numpy() == 1]
stay_scores = probs[y_test.to_numpy() == 0]
wins = churn_scores[:, None] > stay_scores[None, :]
print(wins.size)
print(round(wins.mean(), 3))
for col in ["MonthlyCharges", "tenure"]:
print(col, round(roc_auc_score(y_test, X_test[col]), 3))
Quick check: tenure scores below 0.5. Is it useless, or something else?
print(round(roc_auc_score(y_test, -X_test["tenure"]), 3))
Common mistake: computing AUC from yes/no predictions
hard = model.predict(X_test)
print("AUC from labels:", round(roc_auc_score(y_test, hard), 3))
print("AUC from scores:", round(roc_auc_score(y_test, probs), 3))
Compare models with ROC curves#
Problem: decide which of two models ranks customers better, without cheating on the test set.
from sklearn.tree import DecisionTreeClassifier
tree = DecisionTreeClassifier(max_depth=5, random_state=42)
tree.fit(X_train, y_train)
tree_probs = tree.predict_proba(X_test)[:, 1]
print(len(np.unique(tree_probs)))
t_fpr, t_tpr, _ = roc_curve(y_test, tree_probs)
plt.figure(figsize=(6, 5))
plt.plot(fpr, tpr, label="logistic regression")
plt.plot(t_fpr, t_tpr, label="decision tree")
plt.plot([0, 1], [0, 1], "k--", label="random guessing")
plt.title("Two churn models, one ROC chart")
plt.xlabel("False positive rate")
plt.ylabel("True positive rate")
plt.legend()
plt.show()
from sklearn.model_selection import cross_val_score
for name, m in [("logistic", model), ("tree", tree)]:
s = cross_val_score(m, X_train, y_train, cv=5, scoring="roc_auc")
print(name, s.round(3), "mean", round(s.mean(), 3))
Quick check: does logistic regression win in every fold, or only on average?
lr_cv = cross_val_score(model, X_train, y_train, cv=5,
scoring="roc_auc")
tr_cv = cross_val_score(tree, X_train, y_train, cv=5,
scoring="roc_auc")
print((lr_cv > tr_cv).sum(), "of 5 folds")
print("logistic test AUC:", round(roc_auc_score(y_test, probs), 3))
print("tree test AUC: ", round(roc_auc_score(y_test, tree_probs), 3))
Your turn#
The retention team accepts a 10% false positive rate. Which model catches more churners there? Hint: np.interp.
lr_at_10 = np.interp(0.10, fpr, tpr)
tree_at_10 = np.interp(0.10, t_fpr, t_tpr)
print(round(lr_at_10, 3), round(tree_at_10, 3))
Mini-project: where does the churn model fail?#
Analyse AUC and the confusion matrix separately for each contract type.
contract = df.loc[y_test.index, "Contract"]
rows = []
for kind in contract.unique():
m = (contract == kind).to_numpy()
tn, fp, fn, tp = confusion_matrix(y_test[m], probs[m] >= 0.5,
labels=[0, 1]).ravel()
rows.append({"contract": kind, "customers": m.sum(),
"churn_rate": y_test[m].mean(), "caught": tp,
"missed": fn,
"auc": roc_auc_score(y_test[m], probs[m])})
seg = pd.DataFrame(rows).set_index("contract").round(3)
seg
ax = seg["auc"].plot(kind="bar", rot=0, figsize=(7, 4))
ax.axhline(roc_auc_score(y_test, probs), color="red", ls="--")
ax.set_title("AUC inside each contract type vs overall")
ax.set_xlabel("Contract type")
ax.set_ylabel("AUC")
ax.set_ylim(0.5, 0.9)
plt.show()
scores = pd.DataFrame({"contract": contract, "churned": y_test,
"score": probs})
avg = scores.groupby(["contract", "churned"])["score"].mean()
avg.unstack().round(3)
long = seg.drop(index="Month-to-month")
print("long-contract churners:", long["caught"].sum()
+ long["missed"].sum())
print("caught at 0.5:", long["caught"].sum())
Recap#
- TPR is recall; FPR is the share of negatives wrongly flagged
- The ROC curve plots both rates for every threshold
- AUC = chance a random positive outranks a random negative
- AUC needs scores, not labels; below 0.5 means flipped
- Compare models with cross-validated AUC, then check segments
Challenges#
- Build a
Pipelineof StandardScaler and LogisticRegression on onlytenureandMonthlyCharges. Compare its test AUC. - Draw ROC curves for the tree at max_depth 3 and 8. Which bends closer to the top-left?
- Use
RocCurveDisplay.from_estimatorto redraw the curve in one line.
What's next#
Lesson 22: Hands-on Classification Project- Predicting Customer Churn Put every classifier and metric together on one business problem.
Data Mining 22: Hands-on Classification Project- Predicting Customer Churn#
In this lesson you'll be able to:
- Clean and encode a real customer table for modelling
- Train five classic classifiers on one honest train/test split
- Compare them on precision, recall, F1 and AUC, not just accuracy
- Turn the winner into a ranked call list for a retention team Dataset: IBM Telco customer churn, 7,043 customers (data/mining/telco_churn.csv)
Warm-up: the confusion matrix from lesson 21#
Lesson 21 judged models with ROC curves, AUC and the confusion matrix.
import pandas as pd
df = pd.read_csv("telco_churn.csv")
print(df.shape)
df["Churn"].value_counts(normalize=True).round(3)
from sklearn.metrics import confusion_matrix
stay = pd.Series("No", index=df.index)
print((stay == df["Churn"]).mean().round(3))
confusion_matrix(df["Churn"], stay)
Get to know the churners#
Problem: find who leaves, and fix the messy columns, before any model.
print(df.dtypes.value_counts())
df[["tenure", "MonthlyCharges", "TotalCharges", "Contract"]].head(3)
Common mistake: doing maths on a column that only looks numeric
df["TotalCharges"].mean()
total = pd.to_numeric(df["TotalCharges"], errors="coerce")
print(total.isna().sum())
blank = total.isna()
df.loc[blank, ["tenure", "MonthlyCharges", "TotalCharges"]].head()
df["TotalCharges"] = total.fillna(0)
print(round(df["TotalCharges"].mean(), 2))
churned = df["Churn"].eq("Yes")
churned.groupby(df["Contract"]).mean().round(3)
Quick check: do brand-new customers churn more or less than loyal ones?
edges = [0, 12, 24, 48, 72]
labels = ["0-12", "13-24", "25-48", "49-72"]
bands = pd.cut(df["tenure"], bins=edges, labels=labels,
include_lowest=True)
churned.groupby(bands, observed=True).mean().round(3)
import matplotlib.pyplot as plt
rate = churned.groupby(bands, observed=True).mean()
ax = rate.plot(kind="bar", rot=0, figsize=(7, 4), color="tab:red")
ax.set_title("Churn rate falls the longer customers stay")
ax.set_xlabel("Tenure (months)")
ax.set_ylabel("Share who churned")
plt.show()
Turn the table into model-ready numbers#
Problem: models need numbers only, and a test set they never see.
y = churned.astype(int)
X = df.drop(columns=["customerID", "Churn"])
print(X.shape)
Common mistake: handing text columns straight to a model
from sklearn.tree import DecisionTreeClassifier
DecisionTreeClassifier(random_state=42).fit(X, y)
X = pd.get_dummies(X, drop_first=True, dtype=int)
print(X.shape)
X.columns[-6:].tolist()
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, stratify=y, random_state=42)
print(len(X_train), len(X_test))
print(round(y_train.mean(), 3), round(y_test.mean(), 3))
Quick check: why fit the scaler on the training rows only?
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler().fit(X_train)
X_train_s = scaler.transform(X_train)
X_test_s = scaler.transform(X_test)
print(X_train_s.std().round(3), X_test_s.std().round(3))
Line up five classic classifiers#
Problem: which family of model suits this data? Train them all on one split.
- Logistic regression: a weighted score turned into a probability
- Decision tree: a flowchart of yes or no questions
- kNN: ask the most similar past customers
- Naive Bayes: combine each column's evidence as if independent
- SVM: draw the widest possible boundary between the groups
from sklearn.dummy import DummyClassifier
base = DummyClassifier(strategy="most_frequent")
base.fit(X_train_s, y_train)
print(round(base.score(X_test_s, y_test), 3))
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC
models = {
"Logistic regression": LogisticRegression(max_iter=1000),
"Decision tree": DecisionTreeClassifier(max_depth=5,
random_state=42),
"kNN": KNeighborsClassifier(n_neighbors=25),
"Naive Bayes": GaussianNB(),
"SVM": SVC(random_state=42),
}
for name, model in models.items():
model.fit(X_train_s, y_train)
acc = model.score(X_test_s, y_test)
print(f"{name:20} {acc:.3f}")
Your turn#
Fit a decision tree with no depth limit. Compare its training and test accuracy.
deep = DecisionTreeClassifier(random_state=42)
deep.fit(X_train_s, y_train)
print(round(deep.score(X_train_s, y_train), 3))
print(round(deep.score(X_test_s, y_test), 3))
Compare beyond accuracy#
Problem: accuracy hides how many churners each model actually catches.
from sklearn.metrics import accuracy_score, precision_score
from sklearn.metrics import recall_score, f1_score, roc_auc_score
def churn_scores(model):
pred = model.predict(X_test_s)
if hasattr(model, "predict_proba"):
risk = model.predict_proba(X_test_s)[:, 1]
else:
risk = model.decision_function(X_test_s)
return {"accuracy": accuracy_score(y_test, pred),
"precision": precision_score(y_test, pred),
"recall": recall_score(y_test, pred),
"f1": f1_score(y_test, pred),
"auc": roc_auc_score(y_test, risk)}
scores = {name: churn_scores(m) for name, m in models.items()}
table = pd.DataFrame(scores).T
table.sort_values("auc", ascending=False).round(3)
Quick check: the retention team wants to reach as many leavers as possible. Which column matters most?
table[["recall", "precision"]].sort_values("recall").round(3)
from sklearn.metrics import RocCurveDisplay
fig, ax = plt.subplots(figsize=(7, 5))
for name, model in models.items():
RocCurveDisplay.from_estimator(model, X_test_s, y_test,
name=name, ax=ax)
ax.plot([0, 1], [0, 1], "k--", label="Random guess")
ax.set_title("ROC curves for five churn classifiers")
ax.set_xlabel("False positive rate (stayers flagged)")
ax.set_ylabel("True positive rate (leavers caught)")
ax.legend()
plt.show()
Catch more churners with class weights#
Problem: leavers are only a quarter of customers, so models play safe.
Setting class_weight="balanced" makes each mistake on a rare leaver cost more. We'll apply it to logistic regression, the best ranker.
logreg = models["Logistic regression"]
confusion_matrix(y_test, logreg.predict(X_test_s))
balanced = LogisticRegression(max_iter=1000,
class_weight="balanced")
balanced.fit(X_train_s, y_train)
confusion_matrix(y_test, balanced.predict(X_test_s))
both = pd.DataFrame({
"plain": churn_scores(logreg),
"balanced": churn_scores(balanced)})
both.round(3)
Your turn#
Give the depth-5 decision tree balanced class weights. What happens to its recall?
tree_bal = DecisionTreeClassifier(max_depth=5, random_state=42,
class_weight="balanced")
tree_bal.fit(X_train_s, y_train)
print(round(churn_scores(models["Decision tree"])["recall"], 3))
print(round(churn_scores(tree_bal)["recall"], 3))
Explain what drives churn#
Problem: a score is not enough; the business needs reasons to act on.
weights = pd.Series(balanced.coef_[0], index=X.columns)
weights.nlargest(5).round(2)
Quick check: will a two-year contract raise or lower the churn risk?
weights.nsmallest(5).round(2)
top = weights.abs().nlargest(10).index
ax = weights[top].sort_values().plot(kind="barh", figsize=(7, 5))
ax.set_title("What pushes customers to leave or stay")
ax.set_xlabel("Weight (positive = more likely to churn)")
ax.set_ylabel("Feature")
plt.show()
Weights describe this model's pattern, not proof of cause.
Mini-project: build the retention call list#
The team can call 10% of customers. How many leavers would our list reach?
calls = df.loc[X_test.index, ["customerID", "Contract"]].copy()
calls["risk"] = balanced.predict_proba(X_test_s)[:, 1]
calls["churned"] = y_test
calls = calls.sort_values("risk", ascending=False)
calls.head().round(3)
n_calls = len(calls) // 10
called = calls.head(n_calls)
hit = called["churned"].mean()
base_rate = calls["churned"].mean()
print(n_calls, round(hit, 3), round(base_rate, 3))
calls["decile"] = pd.qcut(calls["risk"], 10, labels=False) + 1
by_decile = calls.groupby("decile")["churned"].mean()
ax = by_decile.plot(kind="bar", rot=0, figsize=(7, 4))
ax.axhline(base_rate, color="k", ls="--", label="Average")
ax.set_title("Real churn rate by predicted risk group")
ax.set_xlabel("Risk group (10 = highest predicted risk)")
ax.set_ylabel("Share who churned")
ax.legend()
plt.show()
caught = called["churned"].sum()
all_leavers = calls["churned"].sum()
lift = hit / base_rate
print(f"Calls reach {caught} of {all_leavers} leavers, "
f"lift {lift:.1f}x")
Recap#
- Fix hidden text values (TotalCharges) before any maths or model
- Encode text with get_dummies; split with stratify; fit the scaler on train only
- Always compare models against a do-nothing baseline
- Choose the model by recall, precision and AUC, not accuracy alone
- class_weight='balanced' trades precision for many more caught churners
Challenges#
- Change kNN's n_neighbors to 5 and 75. How do recall and AUC change?
- Drop TotalCharges and refit the balanced logistic regression. Does AUC drop?
- Build the call list for 20% of customers. What share are real leavers?
What's next#
Lesson 23: Introduction to Ensemble Learning Combine many models so their mistakes cancel out.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.



