Mathew K Analytics

Lesson 3 · Data Mining Full Course

Data Mining Full Course Part 3 (3.5 Hours): Classification Algorithms & Model Evaluation

All 9 lesson notebooks from Part 3 of the Data Mining Full Course in Python, in teaching order. Each lesson starts with a heading; the timestamps below jump…

📓 Full notebook

Download .ipynb

Data Mining Full Course, Part 3: Classification Algorithms and Model Evaluation#

All 9 lesson notebooks from Part 3 of the Data Mining Full Course in Python, in teaching order. Each lesson starts with a heading; the timestamps below jump to it in the video.

  • 0:00 Lesson 14: Decision Trees- ID3, C4.5, CART
  • 0:24:18 Lesson 15: k-Nearest Neighbors kNN Algorithm
  • 0:48:24 Lesson 16: Naive Bayes Classifier
  • 1:12:45 Lesson 17: Logistic Regression for Classification
  • 1:36:03 Lesson 18: Support Vector Machines SVM
  • 1:59:56 Lesson 19: Introduction to Neural Networks for Data Mining
  • 2:27:03 Lesson 20: Evaluating Classifiers- Accuracy, Precision, Recall, F1-Score
  • 2:51:02 Lesson 21: ROC Curves, AUC, and Confusion Matrix Analysis
  • 3:15:34 Lesson 22: Hands-on Classification Project- Predicting Customer Churn

Data files: save the datasets from the lesson page in the same folder as this notebook. Files the lessons create are written to that folder too.

Data Mining 14: Decision Trees - ID3, C4.5, CART#

In this lesson you'll be able to:

  • Explain how a tree chooses its yes/no questions
  • Calculate entropy, Gini, information gain and gain ratio by hand
  • Tell ID3, C4.5 and CART apart, and know which one scikit-learn uses
  • Grow, prune and read a tree that predicts customer churn
  • Turn tree rules into a target list for a business Datasets: Titanic passengers (data/pandas/titanic.csv) and IBM Telco churn (data/mining/telco_churn.csv)

Warm-up: the Titanic, cleaned#

Lesson 13 prepared the Titanic data. One column stood out.

import pandas as pd
import numpy as np
titanic = pd.read_csv("titanic.csv")
titanic["IsFemale"] = (titanic["Sex"] == "female").astype(int)
titanic.groupby("IsFemale")["Survived"].mean().round(3)
IsFemale
0    0.189
1    0.742
Name: Survived, dtype: float64
titanic.groupby("Pclass")["Survived"].mean().round(3)
Pclass
1    0.630
2    0.473
3    0.242
Name: Survived, dtype: float64

How a tree asks questions#

Problem: turn a table into a short chain of yes/no rules.

  • Each node asks one question about one column
  • Each answer sends the row left or right
  • A leaf at the bottom predicts the majority class

Quick check: which question will the tree ask first?

from sklearn.tree import DecisionTreeClassifier, export_text
cols = ["Pclass", "IsFemale", "Fare", "SibSp", "Parch"]
stump = DecisionTreeClassifier(max_depth=1, random_state=42)
stump.fit(titanic[cols], titanic["Survived"])
print(export_text(stump, feature_names=cols))
|--- IsFemale <= 0.50
|   |--- class: 0
|--- IsFemale >  0.50
|   |--- class: 1

import matplotlib.pyplot as plt
from sklearn.tree import plot_tree
small = DecisionTreeClassifier(max_depth=2, random_state=42)
small.fit(titanic[cols], titanic["Survived"])
fig, ax = plt.subplots(figsize=(11, 5))
plot_tree(small, feature_names=cols, class_names=["Died", "Lived"],
          filled=True, impurity=False, ax=ax)
ax.set_title("Two levels of questions on the Titanic")
plt.show()
No description has been provided for this image

Measure impurity: entropy and Gini#

Problem: score how mixed a group is, so splits can be compared.

  • Entropy (ID3, C4.5): 0 when pure, 1 for a 50/50 split of two classes
  • Gini (CART): 0 when pure, 0.5 for a 50/50 split of two classes
def entropy(labels):
    p = labels.value_counts(normalize=True)
    return float(-(p * np.log2(p)).sum())

def gini(labels):
    p = labels.value_counts(normalize=True)
    return float(1 - (p ** 2).sum())
survived = titanic["Survived"]
print(round(entropy(survived), 3), round(gini(survived), 3))
0.961 0.473
by_sex = titanic.groupby("Sex")["Survived"]
by_sex.agg([entropy, gini, "size"]).round(3)
entropy gini size
Sex
female 0.824 0.383 314
male 0.699 0.306 577
third_women = titanic[(titanic["Sex"] == "female")
                      & (titanic["Pclass"] == 3)]["Survived"]
print(third_women.value_counts().to_dict())
print(entropy(third_women), gini(third_women))
{1: 72, 0: 72}
1.0 0.5
def info_gain(data, col, target="Survived"):
    weights = data[col].value_counts(normalize=True)
    child = data.groupby(col)[target].apply(entropy)
    after = (weights * child).sum()
    return entropy(data[target]) - after
for col in ["Sex", "Pclass", "Embarked"]:
    print(col, round(info_gain(titanic, col), 3))
Sex 0.218
Pclass 0.084
Embarked 0.022

Your turn: work out the weighted Gini after splitting by Pclass.

w = titanic["Pclass"].value_counts(normalize=True)
g = titanic.groupby("Pclass")["Survived"].apply(gini)
print(g.round(3).to_dict())
print(round((w * g).sum(), 3))
{1: 0.466, 2: 0.499, 3: 0.367}
0.418

ID3, C4.5 and CART: why gain ratio exists#

Problem: plain information gain loves columns with many values.

print(titanic["PassengerId"].nunique())
print(round(info_gain(titanic, "PassengerId"), 3))
891
0.961
def gain_ratio(data, col):
    split_info = entropy(data[col])
    return info_gain(data, col) / split_info
rows = ["Sex", "Pclass", "Embarked", "Ticket", "PassengerId"]
table = pd.DataFrame({
    "values": [titanic[c].nunique() for c in rows],
    "gain": [info_gain(titanic, c) for c in rows],
    "ratio": [gain_ratio(titanic, c) for c in rows]},
    index=rows)
table.round(3).sort_values("ratio", ascending=False)
values gain ratio
Sex 2 0.218 0.232
PassengerId 891 0.961 0.098
Ticket 681 0.826 0.089
Pclass 3 0.084 0.058
Embarked 3 0.022 0.020
ID3 C4.5 CART
Split score information gain gain ratio Gini
Branches one per value one per value always two
Numbers, gaps, pruning no yes yes

Common mistake: asking scikit-learn for a C4.5 tree.

c45 = DecisionTreeClassifier(criterion="gain_ratio")
c45.fit(titanic[cols], titanic["Survived"])
---------------------------------------------------------------------------
InvalidParameterError                     Traceback (most recent call last)
Cell In[16], line 2
      1 c45 = DecisionTreeClassifier(criterion="gain_ratio")
----> 2 c45.fit(titanic[cols], titanic["Survived"])

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py:1393, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1388 partial_fit_and_fitted = (
   1389     fit_method.__name__ == "partial_fit" and _is_fitted(estimator)
   1390 )
   1392 if not global_skip_validation and not partial_fit_and_fitted:
-> 1393     estimator._validate_params()
   1395 with (
   1396     config_context(
   1397         skip_parameter_validation=(
   (...)
   1401     callback_management_context(estimator),
   1402 ):
   1403     return fit_method(estimator, *args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py:554, in BaseEstimator._validate_params(self)
    546 def _validate_params(self):
    547     """Validate types and values of constructor parameters
    548 
    549     The expected type and values must be defined in the `_parameter_constraints`
   (...)
    552     accepted constraints.
    553     """
--> 554     validate_parameter_constraints(
    555         self._parameter_constraints,
    556         self.get_params(deep=False),
    557         caller_name=self.__class__.__name__,
    558     )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_param_validation.py:98, in validate_parameter_constraints(parameter_constraints, params, caller_name)
     92 else:
     93     constraints_str = (
     94         f"{', '.join([str(c) for c in constraints[:-1]])} or"
     95         f" {constraints[-1]}"
     96     )
---> 98 raise InvalidParameterError(
     99     f"The {param_name!r} parameter of {caller_name} must be"
    100     f" {constraints_str}. Got {param_val!r} instead."
    101 )

InvalidParameterError: The 'criterion' parameter of DecisionTreeClassifier must be a str among {'gini', 'log_loss', 'entropy'}. Got 'gain_ratio' instead.
id3_like = DecisionTreeClassifier(criterion="entropy", max_depth=1,
                                  random_state=42)
id3_like.fit(titanic[cols], titanic["Survived"])
print(export_text(id3_like, feature_names=cols))
|--- IsFemale <= 0.50
|   |--- class: 0
|--- IsFemale >  0.50
|   |--- class: 1

Grow a CART tree on telecom churn#

Problem: predict which of 7,043 customers will leave.

telco = pd.read_csv("telco_churn.csv")
print(telco.shape)
print(telco["Churn"].value_counts(normalize=True).round(3))
(7043, 21)
Churn
No     0.735
Yes    0.265
Name: proportion, dtype: float64

Your turn: find the churn rate for each contract type.

is_churn = telco["Churn"] == "Yes"
is_churn.groupby(telco["Contract"]).mean().round(3)
Contract
Month-to-month    0.427
One year          0.113
Two year          0.028
Name: Churn, dtype: float64

Common mistake: converting a text column that hides blanks.

pd.to_numeric(telco["TotalCharges"])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File pandas/_libs/lib.pyx:2478, in pandas._libs.lib.maybe_convert_numeric()

ValueError: Unable to parse string " "

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
Cell In[20], line 1
----> 1 pd.to_numeric(telco["TotalCharges"])

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\tools\numeric.py:235, in to_numeric(arg, errors, downcast, dtype_backend)
    233     values = ensure_object(values)
    234     coerce_numeric = errors != "raise"
--> 235     values, new_mask = lib.maybe_convert_numeric(  # type: ignore[call-overload]
    236         values,
    237         set(),
    238         coerce_numeric=coerce_numeric,
    239         convert_to_masked_nullable=dtype_backend is not lib.no_default
    240         or (
    241             isinstance(values_dtype, StringDtype)
    242             and values_dtype.na_value is libmissing.NA
    243         ),
    244     )
    246 if new_mask is not None:
    247     # Remove unnecessary values, is expected later anyway and enables
    248     # downcasting
    249     values = values[~new_mask]

File pandas/_libs/lib.pyx:2521, in pandas._libs.lib.maybe_convert_numeric()

ValueError: Unable to parse string " " at position 488
total = pd.to_numeric(telco["TotalCharges"], errors="coerce")
print(total.isna().sum())
print(telco.loc[total.isna(), "tenure"].unique())
11
[0]
telco["TotalCharges"] = total.fillna(0)
y = (telco["Churn"] == "Yes").astype(int)
X_raw = telco.drop(columns=["customerID", "Churn"])
X = pd.get_dummies(X_raw, drop_first=True, dtype=int)
print(X.shape)
(7043, 30)
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42)
print(X_train.shape, X_test.shape)
(5282, 30) (1761, 30)

Quick check: a tree with no limits scores how well on its own training data?

full = DecisionTreeClassifier(random_state=42)
full.fit(X_train, y_train)
print(round(full.score(X_train, y_train), 3))
print(round(full.score(X_test, y_test), 3))
print(full.get_depth(), full.get_n_leaves())
0.998
0.739
24 1026

Prune the tree with depth and leaf size#

Problem: stop the tree memorising, and tune it without touching the test set.

from sklearn.model_selection import cross_val_score
depths = range(1, 13)
cv_acc = []
for d in depths:
    tree = DecisionTreeClassifier(max_depth=d, random_state=42)
    s = cross_val_score(tree, X_train, y_train, cv=5)
    cv_acc.append(s.mean())
cv = pd.Series(cv_acc, index=depths)
cv.round(3).head(8)
1    0.731
2    0.788
3    0.788
4    0.788
5    0.787
6    0.779
7    0.780
8    0.768
dtype: float64
ax = cv.plot(marker="o")
ax.set_title("Cross-validated accuracy by tree depth")
ax.set_xlabel("max_depth")
ax.set_ylabel("Mean accuracy (5 folds)")
plt.show()
No description has been provided for this image
best_depth = int(cv.idxmax())
pruned = DecisionTreeClassifier(max_depth=best_depth,
                                min_samples_leaf=20,
                                random_state=42)
pruned.fit(X_train, y_train)
print(best_depth, pruned.get_n_leaves())
print(round(pruned.score(X_test, y_test), 3))
3 8
0.777
for leaf in [1, 20, 100]:
    tree = DecisionTreeClassifier(min_samples_leaf=leaf,
                                  random_state=42)
    s = cross_val_score(tree, X_train, y_train, cv=5).mean()
    n = tree.fit(X_train, y_train).get_n_leaves()
    print(leaf, round(s, 3), n)
1 0.726 1026
20 0.778 170
100 0.789 37

Quick check: is 77.7% accuracy impressive if 73.5% of customers stay anyway?

from sklearn.metrics import classification_report
pred = pruned.predict(X_test)
print(classification_report(y_test, pred, digits=3))
              precision    recall  f1-score   support

           0      0.790     0.948     0.862      1294
           1      0.678     0.302     0.418       467

    accuracy                          0.777      1761
   macro avg      0.734     0.625     0.640      1761
weighted avg      0.760     0.777     0.744      1761

Read the tree and its feature importances#

Problem: explain the model in words a manager can use.

rules = export_text(pruned, feature_names=list(X.columns),
                    max_depth=2)
print(rules)
|--- tenure <= 10.50
|   |--- InternetService_Fiber optic <= 0.50
|   |   |--- OnlineBackup_No internet service <= 0.50
|   |   |   |--- class: 0
|   |   |--- OnlineBackup_No internet service >  0.50
|   |   |   |--- class: 0
|   |--- InternetService_Fiber optic >  0.50
|   |   |--- TotalCharges <= 120.00
|   |   |   |--- class: 1
|   |   |--- TotalCharges >  120.00
|   |   |   |--- class: 1
|--- tenure >  10.50
|   |--- InternetService_Fiber optic <= 0.50
|   |   |--- Contract_Two year <= 0.50
|   |   |   |--- class: 0
|   |   |--- Contract_Two year >  0.50
|   |   |   |--- class: 0
|   |--- InternetService_Fiber optic >  0.50
|   |   |--- tenure <= 43.50
|   |   |   |--- class: 0
|   |   |--- tenure >  43.50
|   |   |   |--- class: 0

imp = pd.Series(pruned.feature_importances_, index=X.columns)
top = imp.sort_values(ascending=False).head(6)
print(top.round(3))
tenure                              0.504
InternetService_Fiber optic         0.414
OnlineBackup_No internet service    0.047
Contract_Two year                   0.018
TotalCharges                        0.017
SeniorCitizen                       0.000
dtype: float64
ax = top.sort_values().plot(kind="barh", color="teal")
ax.set_title("Top features in the pruned churn tree")
ax.set_xlabel("Importance (share of impurity reduction)")
plt.show()
No description has been provided for this image
leaf_id = pd.Series(pruned.apply(X_train), index=X_train.index)
seg = y_train.groupby(leaf_id).agg(["mean", "size"])
seg.round(3).sort_values("mean", ascending=False)
mean size
6 0.857 182
7 0.670 448
13 0.436 810
3 0.424 510
14 0.189 883
4 0.177 339
10 0.113 1197
11 0.016 913
new = X_test.head(3)
print(pruned.predict_proba(new)[:, 1].round(3))
print(y_test.head(3).to_list())
[0.189 0.113 0.424]
[0, 0, 0]

Mini-project: who should the retention team call?#

Use the pruned tree to rank test customers and check the call list.

risk = pd.Series(pruned.predict_proba(X_test)[:, 1],
                 index=X_test.index)
cutoff = risk.quantile(0.8)
called = risk >= cutoff
print(called.sum(), round(cutoff, 3))
507 0.436
hit = y_test[called].mean()
base = y_test.mean()
caught = y_test[called].sum() / y_test.sum()
print(f"List {hit:.1%}, base {base:.1%}, caught {caught:.1%}")
List 55.0%, base 26.5%, caught 59.7%
charges = telco.loc[X_test.index, "MonthlyCharges"]
charges.groupby(called).median()
False    54.075
True     85.300
Name: MonthlyCharges, dtype: float64
listed = telco.loc[called[called].index]
listed["Contract"].value_counts(normalize=True).round(3)
Contract
Month-to-month    0.907
One year          0.077
Two year          0.016
Name: proportion, dtype: float64

Recap#

  • A tree asks the question that most reduces impurity, then repeats
  • Entropy and Gini both measure how mixed a group is
  • ID3 uses information gain; C4.5's gain ratio penalises ID-like columns
  • scikit-learn builds CART trees; criterion only changes the score
  • Prune with max_depth and min_samples_leaf, tuned by cross-validation

Challenges#

  1. Tune min_samples_leaf (5, 20, 50, 100) with cross-validation.
  2. Refit the churn tree with criterion="entropy". Do the top features change?
  3. Compute the gain ratio of Cabin on the Titanic. What does it say?

What's next#

Lesson 15: k-Nearest Neighbors (kNN) Algorithm Classify by looking at the most similar examples, and see why scaling matters.

Data Mining 15: k-Nearest Neighbors (kNN) Algorithm#

In this lesson you'll be able to:

  • Explain kNN as a vote among the most similar examples
  • Find nearest neighbours by hand with Euclidean distance
  • Show why kNN needs scaled features, and fix it with a pipeline
  • Choose k with cross-validation, not the test set
  • Read kNN's mistakes and vote shares Dataset: Palmer penguins, 344 real birds from Antarctica (data/pandas/penguins.csv)

Warm-up: a decision tree on the penguins#

Lesson 14 grew decision trees. Here is one on today's data.

import pandas as pd
import numpy as np
penguins = pd.read_csv("penguins.csv")
print(penguins.shape)
penguins.head(3)
(344, 7)
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g sex
0 Adelie Torgersen 39.1 18.7 181.0 3750.0 MALE
1 Adelie Torgersen 39.5 17.4 186.0 3800.0 FEMALE
2 Adelie Torgersen 40.3 18.0 195.0 3250.0 FEMALE
from sklearn.tree import DecisionTreeClassifier, export_text
num = ["bill_length_mm", "bill_depth_mm",
       "flipper_length_mm", "body_mass_g"]
known = penguins.dropna(subset=num)
tree = DecisionTreeClassifier(max_depth=2, random_state=42)
tree.fit(known[num], known["species"])
print(export_text(tree, feature_names=num))
|--- flipper_length_mm <= 206.50
|   |--- bill_length_mm <= 43.35
|   |   |--- class: Adelie
|   |--- bill_length_mm >  43.35
|   |   |--- class: Chinstrap
|--- flipper_length_mm >  206.50
|   |--- bill_depth_mm <= 17.65
|   |   |--- class: Gentoo
|   |--- bill_depth_mm >  17.65
|   |   |--- class: Chinstrap

Classify by asking the neighbours#

Problem: label a new bird using the birds most like it.

  • Measure the distance from the new bird to every known bird
  • Keep the k closest: its nearest neighbours
  • Let them vote; the majority species wins
import matplotlib.pyplot as plt
import seaborn as sns
ax = sns.scatterplot(data=known, x="bill_length_mm",
                     y="flipper_length_mm", hue="species")
ax.set_title("Penguins by bill and flipper length")
ax.set_xlabel("Bill length (mm)")
ax.set_ylabel("Flipper length (mm)")
plt.show()
No description has been provided for this image
from sklearn.model_selection import train_test_split
X = known[num]
y = known["species"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42)
print(X_train.shape, X_test.shape)
(256, 4) (86, 4)
query = X_test.iloc[[0]]
query
bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
53 42.0 19.5 200.0 4050.0
diff = X_train - query.to_numpy()
dist = np.sqrt((diff ** 2).sum(axis=1))
nearest = dist.nsmallest(5)
nearest.round(1)
209     7.9
63      8.2
167     8.6
205     9.2
165    10.1
dtype: float64

Quick check: how will these 5 neighbours vote?

y_train[nearest.index].value_counts()
species
Chinstrap    4
Adelie       1
Name: count, dtype: int64
print(y_test.iloc[0])
Adelie
manhattan = diff.abs().sum(axis=1)
y_train[manhattan.nsmallest(5).index].value_counts()
species
Chinstrap    4
Adelie       1
Name: count, dtype: int64

Fit kNN with scikit-learn#

Problem: repeat the vote for every test bird, safely.

Common mistake: fitting kNN on rows with missing measurements.

from sklearn.neighbors import KNeighborsClassifier
raw = KNeighborsClassifier(n_neighbors=5)
raw.fit(penguins[num], penguins["species"])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[11], line 3
      1 from sklearn.neighbors import KNeighborsClassifier
      2 raw = KNeighborsClassifier(n_neighbors=5)
----> 3 raw.fit(penguins[num], penguins["species"])

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py:1403, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1393     estimator._validate_params()
   1395 with (
   1396     config_context(
   1397         skip_parameter_validation=(
   (...)
   1401     callback_management_context(estimator),
   1402 ):
-> 1403     return fit_method(estimator, *args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\neighbors\_classification.py:243, in KNeighborsClassifier.fit(self, X, y)
    221 @_fit_context(
    222     # KNeighborsClassifier.metric is not validated yet
    223     prefer_skip_nested_validation=False
    224 )
    225 def fit(self, X, y):
    226     """Fit the k-nearest neighbors classifier from the training dataset.
    227 
    228     Parameters
   (...)
    241         The fitted k-nearest neighbors classifier.
    242     """
--> 243     return self._fit(X, y)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\neighbors\_base.py:471, in NeighborsBase._fit(self, X, y)
    469 if self.__sklearn_tags__().target_tags.required:
    470     if not isinstance(X, (KDTree, BallTree, NeighborsBase)):
--> 471         X, y = validate_data(
    472             self,
    473             X,
    474             y,
    475             accept_sparse="csr",
    476             multi_output=True,
    477             order="C",
    478             ensure_all_finite=ensure_all_finite,
    479         )
    481     if is_classifier(self):
    482         # Classification targets require a specific format
    483         if y.ndim == 1 or (y.ndim == 2 and y.shape[1] == 1):

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:3058, in validate_data(_estimator, X, y, reset, validate_separately, skip_check_array, **check_params)
   3056         y = check_array(y, input_name="y", **check_y_params)
   3057     else:
-> 3058         X, y = check_X_y(X, y, **check_params)
   3059     out = X, y
   3061 if not no_val_X and check_params.get("ensure_2d", True):

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:1330, in check_X_y(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, ensure_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, estimator)
   1325         estimator_name = _check_estimator_name(estimator)
   1326     raise ValueError(
   1327         f"{estimator_name} requires y to be passed, but the target y is None"
   1328     )
-> 1330 X = check_array(
   1331     X,
   1332     accept_sparse=accept_sparse,
   1333     accept_large_sparse=accept_large_sparse,
   1334     dtype=dtype,
   1335     order=order,
   1336     copy=copy,
   1337     force_writeable=force_writeable,
   1338     ensure_all_finite=ensure_all_finite,
   1339     ensure_2d=ensure_2d,
   1340     allow_nd=allow_nd,
   1341     ensure_min_samples=ensure_min_samples,
   1342     ensure_min_features=ensure_min_features,
   1343     estimator=estimator,
   1344     input_name="X",
   1345 )
   1347 y = _check_y(y, multi_output=multi_output, y_numeric=y_numeric, estimator=estimator)
   1349 check_consistent_length(X, y)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:1090, in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, ensure_all_finite, ensure_non_negative, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)
   1084     raise ValueError(
   1085         f"Found array with dim {array.ndim},"
   1086         f" while dim <= 2 is required{context}."
   1087     )
   1089 if ensure_all_finite:
-> 1090     _assert_all_finite(
   1091         array,
   1092         input_name=input_name,
   1093         estimator_name=estimator_name,
   1094         allow_nan=ensure_all_finite == "allow-nan",
   1095     )
   1097 if copy:
   1098     if _is_numpy_namespace(xp):
   1099         # only make a copy if `array` and `array_orig` may share memory`

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:137, in _assert_all_finite(X, allow_nan, msg_dtype, estimator_name, input_name)
    134 if first_pass_isfinite:
    135     return
--> 137 _assert_all_finite_element_wise(
    138     X,
    139     xp=xp,
    140     allow_nan=allow_nan,
    141     msg_dtype=msg_dtype,
    142     estimator_name=estimator_name,
    143     input_name=input_name,
    144 )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:186, in _assert_all_finite_element_wise(X, xp, allow_nan, msg_dtype, estimator_name, input_name)
    169 if estimator_name and input_name == "X" and has_nan_error:
    170     # Improve the error message on how to handle missing values in
    171     # scikit-learn.
    172     msg_err += (
    173         f"\n{estimator_name} does not accept missing values"
    174         " encoded as NaN natively. For supervised learning, you might want"
   (...)
    184         "#estimators-that-handle-nan-values"
    185     )
--> 186 raise ValueError(msg_err)

ValueError: Input X contains NaN.
KNeighborsClassifier does not accept missing values encoded as NaN natively. For supervised learning, you might want to consider sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept missing values encoded as NaNs natively. Alternatively, it is possible to preprocess the data, for instance by using an imputer transformer in a pipeline or drop samples with missing values. See https://scikit-learn.org/stable/modules/impute.html You can find a list of all estimators that handle NaN values at the following page: https://scikit-learn.org/stable/modules/impute.html#estimators-that-handle-nan-values
print(penguins[num].isna().any(axis=1).sum())
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train, y_train)
print(knn.predict(query))
print(round(knn.score(X_test, y_test), 3))
2
['Chinstrap']
0.791
d, idx = knn.kneighbors(query)
print(d.round(1))
[[ 7.9  8.2  8.6  9.2 10.1]]

Quick check: what did fit() actually learn?

print(knn.n_samples_fit_)
print(knn.classes_)
256
['Adelie' 'Chinstrap' 'Gentoo']
  • kNN is a lazy learner: fit stores the data, predict does the work
  • Fast to train, slower to predict on big tables
raw_pred = knn.predict(X_test)
pd.crosstab(y_test, raw_pred, rownames=["real"],
            colnames=["predicted"])
predicted Adelie Chinstrap Gentoo
real
Adelie 36 2 0
Chinstrap 13 4 0
Gentoo 3 0 28

Why scaling decides who your neighbours are#

Problem: one big-number column can drown out the others.

X_train.agg(["min", "max", "std"]).round(1)
bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
min 33.1 13.1 172.0 2700.0
max 59.6 21.5 231.0 6300.0
std 5.5 2.0 13.8 794.2
X_train.loc[nearest.index]
bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
209 49.3 19.9 203.0 4050.0
63 41.1 18.2 192.0 4050.0
167 50.5 19.6 201.0 4050.0
205 50.7 19.7 203.0 4050.0
165 52.0 18.1 201.0 4050.0
ax = sns.scatterplot(data=X_train.assign(species=y_train),
                     x="bill_length_mm", y="body_mass_g",
                     hue="species", alpha=0.4)
nb = X_train.loc[nearest.index]
ax.scatter(nb["bill_length_mm"], nb["body_mass_g"], color="black",
           marker="x", s=80, label="raw neighbours")
ax.set_title("Unscaled neighbours all share one body mass")
ax.set_xlabel("Bill length (mm)")
ax.set_ylabel("Body mass (g)")
ax.legend()
plt.show()
No description has been provided for this image
share = (diff ** 2).div((diff ** 2).sum(axis=1), axis=0)
share.mean().round(3)
bill_length_mm       0.015
bill_depth_mm        0.000
flipper_length_mm    0.007
body_mass_g          0.977
dtype: float64

Quick check: the neighbours were chosen almost entirely by weight. Is that fair?

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
scaled_knn = make_pipeline(StandardScaler(),
                           KNeighborsClassifier(n_neighbors=5))
scaled_knn.fit(X_train, y_train)
print(round(scaled_knn.score(X_test, y_test), 3))
1.0
scaler = scaled_knn.named_steps["standardscaler"]
model = scaled_knn.named_steps["kneighborsclassifier"]
d, idx = model.kneighbors(scaler.transform(query))
y_train.iloc[idx[0]].value_counts()
species
Adelie    5
Name: count, dtype: int64

Your turn: fit an unscaled kNN on bill length and bill depth only. Both are in millimetres.

bills = ["bill_length_mm", "bill_depth_mm"]
bill_knn = KNeighborsClassifier(n_neighbors=5)
bill_knn.fit(X_train[bills], y_train)
print(round(bill_knn.score(X_test[bills], y_test), 3))
0.953

Choose k with cross-validation#

Problem: too few neighbours is noisy, too many blurs the classes.

from sklearn.model_selection import cross_val_score
ks = range(1, 31, 2)
scores = []
for k in ks:
    pipe = make_pipeline(StandardScaler(),
                         KNeighborsClassifier(n_neighbors=k))
    scores.append(cross_val_score(pipe, X_train, y_train,
                                  cv=5).mean())
cv = pd.Series(scores, index=ks)
cv.round(3).head(8)
1     0.988
3     0.992
5     0.984
7     0.980
9     0.977
11    0.977
13    0.977
15    0.977
dtype: float64
ax = cv.plot(marker="o")
ax.set_title("Cross-validated accuracy by number of neighbours")
ax.set_xlabel("k (neighbours)")
ax.set_ylabel("Mean accuracy (5 folds)")
plt.show()
No description has been provided for this image
one = make_pipeline(StandardScaler(),
                    KNeighborsClassifier(n_neighbors=1))
one.fit(X_train, y_train)
print(one.score(X_train, y_train), round(cv[1], 3))
1.0 0.988
train_acc = []
for k in ks:
    pipe = make_pipeline(StandardScaler(),
                         KNeighborsClassifier(n_neighbors=k))
    train_acc.append(pipe.fit(X_train, y_train)
                     .score(X_train, y_train))
gap = pd.DataFrame({"train": train_acc, "cv": scores}, index=ks)
gap.round(3).head(5)
train cv
1 1.000 0.988
3 1.000 0.992
5 0.992 0.984
7 0.988 0.980
9 0.984 0.977
best_k = int(cv.idxmax())
final = make_pipeline(StandardScaler(),
                      KNeighborsClassifier(n_neighbors=best_k))
final.fit(X_train, y_train)
print(best_k, round(final.score(X_test, y_test), 3))
3 0.988

Your turn: try weights="distance" so closer neighbours count more. Score it with cross-validation.

weighted = make_pipeline(StandardScaler(), KNeighborsClassifier(
    n_neighbors=best_k, weights="distance"))
s = cross_val_score(weighted, X_train, y_train, cv=5).mean()
print(round(s, 3), round(cv[best_k], 3))
0.992 0.992

Read the mistakes and the vote shares#

Problem: find which species get confused, and how sure kNN was.

from sklearn.metrics import confusion_matrix
pred = final.predict(X_test)
labels = final.classes_
cm = confusion_matrix(y_test, pred, labels=labels)
pd.DataFrame(cm, index=labels, columns=labels)
Adelie Chinstrap Gentoo
Adelie 37 1 0
Chinstrap 0 17 0
Gentoo 0 0 31
from sklearn.metrics import classification_report
print(classification_report(y_test, pred, digits=3))
              precision    recall  f1-score   support

      Adelie      1.000     0.974     0.987        38
   Chinstrap      0.944     1.000     0.971        17
      Gentoo      1.000     1.000     1.000        31

    accuracy                          0.988        86
   macro avg      0.981     0.991     0.986        86
weighted avg      0.989     0.988     0.988        86

wrong = X_test[pred != y_test]
wrong.assign(real=y_test[wrong.index],
             predicted=pred[pred != y_test])
bill_length_mm bill_depth_mm flipper_length_mm body_mass_g real predicted
131 43.1 19.2 197.0 3500.0 Adelie Chinstrap
proba = pd.DataFrame(final.predict_proba(X_test),
                     columns=labels, index=X_test.index)
unsure = proba.max(axis=1) < 1
print(unsure.sum(), len(proba))
proba[unsure].round(2).head()
4 86
Adelie Chinstrap Gentoo
158 0.33 0.67 0.0
122 0.67 0.33 0.0
157 0.33 0.67 0.0
131 0.33 0.67 0.0

Common mistake: asking for k bigger than the data allows.

tiny = X_train.head(4)
small = KNeighborsClassifier(n_neighbors=5)
small.fit(tiny, y_train.head(4)).predict(query)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[33], line 3
      1 tiny = X_train.head(4)
      2 small = KNeighborsClassifier(n_neighbors=5)
----> 3 small.fit(tiny, y_train.head(4)).predict(query)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\neighbors\_classification.py:278, in KNeighborsClassifier.predict(self, X)
    275         return self.classes_[np.argmax(probabilities, axis=1)]
    276     # In that case, we do not need the distances to perform
    277     # the weighting so we do not compute them.
--> 278     neigh_ind = self.kneighbors(X, return_distance=False)
    279     neigh_dist = None
    280 else:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\neighbors\_base.py:847, in KNeighborsMixin.kneighbors(self, X, n_neighbors, return_distance)
    845     else:
    846         inequality_str = "n_neighbors <= n_samples_fit"
--> 847     raise ValueError(
    848         f"Expected {inequality_str}, but "
    849         f"n_neighbors = {n_neighbors}, n_samples_fit = {n_samples_fit}, "
    850         f"n_samples = {X.shape[0]}"  # include n_samples for common tests
    851     )
    853 n_jobs = effective_n_jobs(self.n_jobs)
    854 chunked_results = None

ValueError: Expected n_neighbors <= n_samples_fit, but n_neighbors = 5, n_samples_fit = 4, n_samples = 1
k = min(5, len(tiny))
small = KNeighborsClassifier(n_neighbors=k)
print(small.fit(tiny, y_train.head(4)).predict(query))
['Adelie']

Mini-project: can kNN tell a penguin's sex?#

Researchers often can't sex a bird in the field. Do the measurements give it away?

sexed = known.dropna(subset=["sex"])
Xs = pd.get_dummies(sexed[num + ["species"]], dtype=int)
ys = sexed["sex"]
print(Xs.shape)
ys.value_counts()
(333, 7)
sex
MALE      168
FEMALE    165
Name: count, dtype: int64
Xs_tr, Xs_te, ys_tr, ys_te = train_test_split(
    Xs, ys, test_size=0.25, stratify=ys, random_state=42)
sex_cv = []
for k in ks:
    pipe = make_pipeline(StandardScaler(),
                         KNeighborsClassifier(n_neighbors=k))
    sex_cv.append(cross_val_score(pipe, Xs_tr, ys_tr, cv=5).mean())
sex_k = ks[int(np.argmax(sex_cv))]
print(sex_k, round(max(sex_cv), 3))
9 0.928
ax = pd.Series(sex_cv, index=ks).plot(marker="o", color="purple")
ax.set_title("Sexing penguins: accuracy by number of neighbours")
ax.set_xlabel("k (neighbours)")
ax.set_ylabel("Mean accuracy (5 folds)")
plt.show()
No description has been provided for this image
sex_model = make_pipeline(StandardScaler(),
                          KNeighborsClassifier(n_neighbors=sex_k))
sex_model.fit(Xs_tr, ys_tr)
acc = sex_model.score(Xs_te, ys_te)
base = ys_te.value_counts(normalize=True).max()
print(f"kNN {acc:.1%} vs guessing {base:.1%}")
kNN 91.7% vs guessing 50.0%
three = Xs_te.head(3)
print(sex_model.predict_proba(three).round(2))
print(ys_te.head(3).to_list())
[[0.67 0.33]
 [0.67 0.33]
 [0.   1.  ]]
['FEMALE', 'MALE', 'MALE']
right = sex_model.predict(Xs_te) == ys_te
right.groupby(sexed.loc[Xs_te.index, "species"]).mean().round(3)
species
Adelie       0.906
Chinstrap    0.875
Gentoo       0.944
Name: sex, dtype: float64
sex_tree = DecisionTreeClassifier(max_depth=4, random_state=42)
t = cross_val_score(sex_tree, Xs_tr, ys_tr, cv=5).mean()
print(round(t, 3), round(max(sex_cv), 3))
0.891 0.928
print(f"Species: {final.score(X_test, y_test):.1%} with k={best_k}")
print(f"Sex: {acc:.1%} with k={sex_k}, guessing {base:.1%}")
Species: 98.8% with k=3
Sex: 91.7% with k=9, guessing 50.0%

Recap#

  • kNN stores the training data and votes among the k closest rows
  • Distance is dominated by big-number columns, so always scale
  • Put the scaler in a pipeline so it learns from training rows only
  • Choose k with cross-validation; odd k avoids two-way ties
  • predict_proba shows the vote shares behind each prediction

Challenges#

  1. Re-run the species model using only bill length and depth.
  2. Swap StandardScaler for MinMaxScaler. Does the best k change?
  3. Try metric="manhattan" in the sex model and compare scores.

What's next#

Lesson 16: Naive Bayes Classifier Classify with probabilities: fast, simple, and great with text.

Data Mining 16: Naive Bayes Classifier#

In this lesson you'll be able to:

  • Apply Bayes' rule to real counts by hand
  • Train Gaussian Naive Bayes on numeric measurements
  • Turn text into word counts and train Multinomial Naive Bayes
  • Fix the zero-frequency problem with smoothing
  • Choose a spam threshold that protects real messages Dataset: 5,572 real SMS messages labelled ham or spam (data/mining/sms_spam.csv)

Warm-up: kNN on text messages#

Lesson 15 classified with k-nearest neighbours. Same idea, today's data.

import pandas as pd
sms = pd.read_csv("sms_spam.csv")
sms["spam"] = (sms["label"] == "spam").astype(int)
print(sms.shape)
sms["label"].value_counts()
(5572, 3)
label
ham     4825
spam     747
Name: count, dtype: int64
sms["length"] = sms["message"].str.len()
sms["digits"] = sms["message"].str.count(r"\d")
sms["capitals"] = sms["message"].str.count(r"[A-Z]")
nums = ["length", "digits", "capitals"]
sms.groupby("label")[nums].mean().round(1)
length digits capitals
label
ham 71.5 0.3 4.1
spam 138.7 15.8 15.5
from sklearn.model_selection import train_test_split
train, test = train_test_split(
    sms, test_size=0.25, stratify=sms["spam"], random_state=42)
print(len(train), len(test))
4179 1393
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
knn = make_pipeline(StandardScaler(), KNeighborsClassifier(5))
knn.fit(train[nums], train["spam"])
print(round(knn.score(test[nums], test["spam"]), 3))
0.973

Bayes' rule with real counts#

Problem: turn 'how often spam says free' into 'how likely this message is spam'.

  • Prior: how common spam is before we read a word
  • Likelihood: how often spam (or ham) contains the clue
  • Posterior = prior x likelihood / how common the clue is
p_spam = sms["spam"].mean()
print(round(p_spam, 3))
0.134
has_free = sms["message"].str.contains(r"\bfree\b", case=False)
pd.crosstab(has_free, sms["label"])
label ham spam
message
False 4766 577
True 59 170
p_free_spam = has_free[sms["spam"] == 1].mean()
p_free = has_free.mean()
posterior = p_free_spam * p_spam / p_free
print(round(p_free_spam, 3), round(p_free, 3))
print(round(posterior, 3))
0.228 0.041
0.742

Quick check: Bayes said 0.742. What share of messages that contain 'free' really are spam?

print(round(sms.loc[has_free, "spam"].mean(), 3))
0.742

Your turn: use Bayes' rule to find the spam chance for messages that say 'call'.

has_call = sms["message"].str.contains(r"\bcall\b", case=False)
p_call_spam = has_call[sms["spam"] == 1].mean()
print(round(p_call_spam * p_spam / has_call.mean(), 3))
0.596

Why it is called naive#

Naive Bayes multiplies the evidence from each clue as if the clues were independent.

def lift(flag):
    return flag[sms["spam"] == 1].mean() / flag[sms["spam"] == 0].mean()

has_txt = sms["message"].str.contains(r"\btxt\b", case=False)
odds = p_spam / (1 - p_spam) * lift(has_free) * lift(has_txt)
print(round(lift(has_free), 1), round(lift(has_txt), 1))
print(round(odds / (1 + odds), 3))
both = has_free & has_txt
print(both.sum(), round(sms.loc[both, "spam"].mean(), 3))
18.6 75.5
0.995
54 1.0

Gaussian Naive Bayes on message measurements#

Problem: our measurements are numbers like length, not yes-or-no clues.

from sklearn.naive_bayes import GaussianNB
gnb = GaussianNB()
gnb.fit(train[nums], train["spam"])
print(round(gnb.score(test[nums], test["spam"]), 3))
0.961
means = pd.DataFrame(gnb.theta_, index=["ham", "spam"], columns=nums)
means.round(1)
length digits capitals
ham 71.2 0.3 4.0
spam 139.3 15.6 15.6
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(7, 4))
for name, grp in train.groupby("label"):
    ax.hist(grp["digits"], bins=30, alpha=0.6, label=name)
ax.set_title("Digits per message, ham vs spam")
ax.set_xlabel("Digits in the message")
ax.set_ylabel("Number of messages")
ax.legend()
plt.show()
No description has been provided for this image
probs = gnb.predict_proba(test[nums].head(3))
print(probs.round(3))
print(test["label"].head(3).to_list())
[[1. 0.]
 [0. 1.]
 [1. 0.]]
['ham', 'spam', 'ham']

Your turn: train GaussianNB on the digits column alone. How close does it get to 0.961?

solo = GaussianNB().fit(train[["digits"]], train["spam"])
print(round(solo.score(test[["digits"]], test["spam"]), 3))
0.97

Common mistake: giving MultinomialNB scaled features

from sklearn.naive_bayes import MultinomialNB
scaled = StandardScaler().fit_transform(train[nums])
MultinomialNB().fit(scaled, train["spam"])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[17], line 3
      1 from sklearn.naive_bayes import MultinomialNB
      2 scaled = StandardScaler().fit_transform(train[nums])
----> 3 MultinomialNB().fit(scaled, train["spam"])

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py:1403, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1393     estimator._validate_params()
   1395 with (
   1396     config_context(
   1397         skip_parameter_validation=(
   (...)
   1401     callback_management_context(estimator),
   1402 ):
-> 1403     return fit_method(estimator, *args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\naive_bayes.py:794, in _BaseDiscreteNB.fit(self, X, y, sample_weight)
    792 n_classes = Y.shape[1]
    793 self._init_counters(n_classes, n_features)
--> 794 self._count(X, Y)
    795 alpha = self._check_alpha()
    796 self._update_feature_log_prob(alpha)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\naive_bayes.py:921, in MultinomialNB._count(self, X, Y)
    919 def _count(self, X, Y):
    920     """Count and smooth feature occurrences."""
--> 921     check_non_negative(X, "MultinomialNB (input X)")
    922     self.feature_count_ += safe_sparse_dot(Y.T, X)
    923     self.class_count_ += Y.sum(axis=0)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:1791, in check_non_negative(X, whom)
   1788     X_min = xp.min(X)
   1790 if X_min < 0:
-> 1791     raise ValueError(f"Negative values in data passed to {whom}.")

ValueError: Negative values in data passed to MultinomialNB (input X).
mnb_nums = MultinomialNB().fit(train[nums], train["spam"])
print(round(mnb_nums.score(test[nums], test["spam"]), 3))
0.947

From words to counts#

Problem: a model needs numbers, but the real evidence is in the words.

from sklearn.feature_extraction.text import CountVectorizer
demo = CountVectorizer()
counts = demo.fit_transform(sms["message"].head(2))
words = demo.get_feature_names_out()
print(counts.shape)
pd.DataFrame(counts.toarray(), columns=words).iloc[:, :8]
(2, 23)
amore available buffet bugis cine crazy go got
0 1 1 1 1 1 1 1 1
1 0 0 0 0 0 0 0 0
vec = CountVectorizer()
Xw_train = vec.fit_transform(train["message"])
Xw_test = vec.transform(test["message"])
print(Xw_train.shape, Xw_test.shape)
print(round(Xw_train.nnz / (Xw_train.shape[0] * Xw_train.shape[1]), 4))
(4179, 7387) (1393, 7387)
0.0018

Train a word-based spam filter#

Problem: score every word in a message at once, and decide.

mnb = MultinomialNB()
mnb.fit(Xw_train, train["spam"])
print(round(mnb.score(Xw_test, test["spam"]), 3))
0.986
from sklearn.metrics import confusion_matrix, classification_report
pred = mnb.predict(Xw_test)
print(confusion_matrix(test["spam"], pred))
[[1204    2]
 [  17  170]]
print(classification_report(test["spam"], pred,
                            target_names=["ham", "spam"], digits=3))
              precision    recall  f1-score   support

         ham      0.986     0.998     0.992      1206
        spam      0.988     0.909     0.947       187

    accuracy                          0.986      1393
   macro avg      0.987     0.954     0.970      1393
weighted avg      0.986     0.986     0.986      1393

names = vec.get_feature_names_out()
log_p = mnb.feature_log_prob_
spam_lift = pd.Series(log_p[1] - log_p[0], index=names)
spam_lift.nlargest(8).round(2)
claim         5.43
prize         5.13
150p          4.92
tone          4.71
18            4.58
500           4.58
cs            4.58
guaranteed    4.58
dtype: float64
top = spam_lift.nlargest(10).sort_values()
ax = top.plot(kind="barh", figsize=(7, 4))
ax.set_title("Words most typical of spam")
ax.set_xlabel("Log probability lift, spam vs ham")
ax.set_ylabel("Word")
plt.show()
No description has been provided for this image

Your turn: list the 8 words that point most strongly to ham.

spam_lift.nsmallest(8).round(2)
gt      -4.50
lt      -4.49
he      -4.17
lor     -3.87
da      -3.80
she     -3.74
later   -3.66
amp     -3.34
dtype: float64

Common mistake: fitting a new vectorizer on the test messages

Xw_bad = CountVectorizer().fit_transform(test["message"])
mnb.predict(Xw_bad)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[27], line 2
      1 Xw_bad = CountVectorizer().fit_transform(test["message"])
----> 2 mnb.predict(Xw_bad)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\naive_bayes.py:112, in _BaseNB.predict(self, X)
    110 check_is_fitted(self)
    111 xp, _ = get_namespace(X)
--> 112 X = self._check_X(X)
    113 jll = self._joint_log_likelihood(X)
    114 pred_indices = xp.argmax(jll, axis=1)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\naive_bayes.py:609, in _BaseDiscreteNB._check_X(self, X)
    607 def _check_X(self, X):
    608     """Validate X, used only in predict* methods."""
--> 609     return validate_data(self, X, accept_sparse="csr", reset=False)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:3062, in validate_data(_estimator, X, y, reset, validate_separately, skip_check_array, **check_params)
   3059     out = X, y
   3061 if not no_val_X and check_params.get("ensure_2d", True):
-> 3062     _check_n_features(_estimator, X, reset=reset)
   3064 return out

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:2926, in _check_n_features(estimator, X, reset)
   2923     return
   2925 if n_features != estimator.n_features_in_:
-> 2926     raise ValueError(
   2927         f"X has {n_features} features, but {estimator.__class__.__name__} "
   2928         f"is expecting {estimator.n_features_in_} features as input."
   2929     )

ValueError: X has 4084 features, but MultinomialNB is expecting 7387 features as input.
new_msgs = test["message"].head(3)
print(mnb.predict(vec.transform(new_msgs)))
print(test["spam"].head(3).to_list())
[0 0 0]
[0, 1, 0]

Smoothing: the zero-frequency problem#

Problem: one word never seen in ham would make the ham chance exactly zero.

wc = mnb.feature_count_
only_spam = names[(wc[0] == 0) & (wc[1] >= 15)]
print(len(only_spam))
print(only_spam[:8])
28
['000' '1000' '10p' '150p' '150ppm' '18' '500' '5000']
import numpy as np
i = vec.vocabulary_[only_spam[0]]
print(wc[0, i] / wc[0].sum())
print(f"{np.exp(log_p[0, i]):.7f}")
0.0
0.0000184
from sklearn.model_selection import cross_val_score
for a in [0.01, 0.1, 0.5, 1.0]:
    model = MultinomialNB(alpha=a)
    f1 = cross_val_score(model, Xw_train, train["spam"], cv=5,
                         scoring="f1").mean()
    print(a, round(f1, 3))
0.01 0.935
0.1 0.925
0.5 0.923
1.0 0.929

Quick check: which alpha would you keep, and why not pick it on the test set?

best = MultinomialNB(alpha=0.01)
best.fit(Xw_train, train["spam"])
print(round(best.score(Xw_test, test["spam"]), 3))
0.988

Mini-project: a filter that doesn't lose real messages#

Question: how strict must the filter be so almost no genuine message is blocked?

from sklearn.model_selection import cross_val_predict
pipe = make_pipeline(CountVectorizer(), MultinomialNB(alpha=0.01))
p_cv = cross_val_predict(pipe, train["message"], train["spam"],
                         cv=5, method="predict_proba")[:, 1]
is_ham = (train["spam"] == 0).to_numpy()
for t in [0.5, 0.9, 0.99]:
    blocked = ((p_cv >= t) & is_ham).sum()
    caught = (p_cv[~is_ham] >= t).mean()
    print(t, blocked, round(caught, 3))
0.5 32 0.929
0.9 13 0.916
0.99 2 0.909
pipe.fit(train["message"], train["spam"])
p_test = pipe.predict_proba(test["message"])[:, 1]
flag = (p_test >= 0.99).astype(int)
print(confusion_matrix(test["spam"], flag))
[[1206    0]
 [  19  168]]
sp = test.assign(flag=flag).query("spam == 1")
sp.groupby("flag")[["length", "digits"]].median()
length digits
flag
0 129.0 4.0
1 148.0 18.0
blocked = ((flag == 1) & (test["spam"] == 0)).sum()
caught = flag[test["spam"] == 1].mean()
print(f"Blocked ham: {blocked}, spam caught: {caught:.1%}")
Blocked ham: 0, spam caught: 89.8%

Recap#

  • Bayes' rule: posterior = prior x likelihood / evidence
  • Naive means clues are treated as independent and multiplied
  • GaussianNB for measurements; MultinomialNB for counts, never negatives
  • Fit the vectorizer on training text only; transform the rest
  • Tune alpha and the threshold with cross-validation, not the test set

Challenges#

  1. Add the three measurements as extra columns. Does accuracy improve?
  2. Try CountVectorizer(ngram_range=(1, 2)) so word pairs count too.
  3. Find the threshold where blocked ham first rises above zero.

What's next#

Lesson 17: Logistic Regression for Classification Learn weights for every feature and read them as odds.

Data Mining 17: Logistic Regression for Classification#

In this lesson you'll be able to:

  • Explain how the sigmoid turns a score into a probability
  • Clean and encode real churn data for a linear model
  • Train and evaluate LogisticRegression on a held-out test set
  • Read coefficients as odds ratios a manager understands
  • Build a retention call list and measure its lift Dataset: IBM Telco customer churn, 7,043 customers (data/mining/telco_churn.csv)

Warm-up: Naive Bayes on churn#

Lesson 16 built Naive Bayes classifiers. Here's one on today's data.

import pandas as pd
df = pd.read_csv("telco_churn.csv")
df["churn"] = (df["Churn"] == "Yes").astype(int)
print(df.shape)
print(round(df["churn"].mean(), 3))
(7043, 22)
0.265
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB
train, test = train_test_split(
    df, test_size=0.25, stratify=df["churn"], random_state=42)
two = ["tenure", "MonthlyCharges"]
nb = GaussianNB().fit(train[two], train["churn"])
print(round(nb.score(test[two], test["churn"]), 3))
0.781

From a score to a probability#

Problem: a straight line can predict -0.3 or 1.4, but a probability must stay between 0 and 1.

  • Add up the features, each times a weight, into a score z
  • Squash z through the S-shaped sigmoid: p = 1 / (1 + e^-z)
  • Predict churn when p is 0.5 or more
bands = pd.cut(df["tenure"], bins=[-1, 12, 24, 48, 72],
              labels=["0-12", "13-24", "25-48", "49-72"])
df.groupby(bands, observed=True)["churn"].mean().round(3)
tenure
0-12     0.474
13-24    0.287
25-48    0.204
49-72    0.095
Name: churn, dtype: float64
from sklearn.linear_model import LogisticRegression
one = LogisticRegression()
one.fit(train[["tenure"]], train["churn"])
print(one.coef_.round(4), one.intercept_.round(4))
[[-0.0374]] [-0.0047]
import matplotlib.pyplot as plt
rate = train.groupby("tenure")["churn"].mean()
grid = rate.index.to_frame(index=False)
curve = one.predict_proba(grid)[:, 1]
fig, ax = plt.subplots(figsize=(7, 4))
ax.scatter(rate.index, rate, s=12, label="Actual churn rate")
ax.plot(grid["tenure"], curve, color="red", label="Logistic curve")
ax.set_title("Churn probability falls with tenure")
ax.set_xlabel("Tenure (months)")
ax.set_ylabel("Share who churned")
ax.legend()
No description has been provided for this image
import numpy as np
z = one.intercept_[0] + one.coef_[0, 0] * 12
print(round(1 / (1 + np.exp(-z)), 3))
row = test.loc[test["tenure"] == 12, ["tenure"]].head(1)
print(one.predict_proba(row)[:, 1].round(3))
0.389
[0.389]

Quick check: what churn chance does the curve give a loyal customer at 72 months?

row72 = test.loc[test["tenure"] == 72, ["tenure"]].head(1)
print(one.predict_proba(row72)[:, 1].round(3))
[0.063]

Your turn: fit a one-feature model on MonthlyCharges. Do higher bills raise churn?

bill = LogisticRegression()
bill.fit(train[["MonthlyCharges"]], train["churn"])
print(bill.coef_.round(4))
[[0.0157]]

Prepare real data for the model#

Problem: the model needs complete numbers, but this table holds text and blanks.

print(df["TotalCharges"].dtype)
print(df["TotalCharges"].head(3).to_list())
str
['29.85', '1889.5', '108.15']

Common mistake: converting a text column that hides blanks

pd.to_numeric(df["TotalCharges"])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File pandas/_libs/lib.pyx:2478, in pandas._libs.lib.maybe_convert_numeric()

ValueError: Unable to parse string " "

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
Cell In[11], line 1
----> 1 pd.to_numeric(df["TotalCharges"])

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\tools\numeric.py:235, in to_numeric(arg, errors, downcast, dtype_backend)
    233     values = ensure_object(values)
    234     coerce_numeric = errors != "raise"
--> 235     values, new_mask = lib.maybe_convert_numeric(  # type: ignore[call-overload]
    236         values,
    237         set(),
    238         coerce_numeric=coerce_numeric,
    239         convert_to_masked_nullable=dtype_backend is not lib.no_default
    240         or (
    241             isinstance(values_dtype, StringDtype)
    242             and values_dtype.na_value is libmissing.NA
    243         ),
    244     )
    246 if new_mask is not None:
    247     # Remove unnecessary values, is expected later anyway and enables
    248     # downcasting
    249     values = values[~new_mask]

File pandas/_libs/lib.pyx:2521, in pandas._libs.lib.maybe_convert_numeric()

ValueError: Unable to parse string " " at position 488
tc = pd.to_numeric(df["TotalCharges"], errors="coerce")
blank = tc.isna()
print(blank.sum())
print(df.loc[blank, "tenure"].unique())
11
[0]
df["TotalCharges"] = tc.fillna(0)
print(df["TotalCharges"].isna().sum())
0

Quick check: Contract has three levels. How many 0 or 1 columns does it really need?

print(pd.get_dummies(df["Contract"]).shape)
print(pd.get_dummies(df["Contract"], drop_first=True).shape)
(7043, 3)
(7043, 2)
nums = ["SeniorCitizen", "tenure", "MonthlyCharges", "TotalCharges"]
cats = ["Contract", "InternetService", "PaymentMethod",
        "PaperlessBilling", "Partner", "Dependents"]
X = pd.get_dummies(df[nums + cats], drop_first=True, dtype=int)
print(X.shape)
print(list(X.columns[4:7]))
(7043, 14)
['Contract_One year', 'Contract_Two year', 'InternetService_Fiber optic']
X_train, X_test = X.loc[train.index], X.loc[test.index]
y_train, y_test = train["churn"], test["churn"]
print(X_train.shape, X_test.shape)
(5282, 14) (1761, 14)

Your turn: which Contract level did drop_first remove? Check the levels.

print(sorted(df["Contract"].unique()))
['Month-to-month', 'One year', 'Two year']

Train and evaluate the full model#

Problem: does the full model beat a simple guess on customers it hasn't seen?

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
model = make_pipeline(StandardScaler(),
                      LogisticRegression(max_iter=1000))
model.fit(X_train, y_train)
print(round(model.score(X_test, y_test), 3))
print(round(1 - y_test.mean(), 3))
0.794
0.735
from sklearn.metrics import confusion_matrix, classification_report
pred = model.predict(X_test)
print(confusion_matrix(y_test, pred))
print(classification_report(y_test, pred, digits=3))
[[1156  138]
 [ 225  242]]
              precision    recall  f1-score   support

           0      0.837     0.893     0.864      1294
           1      0.637     0.518     0.571       467

    accuracy                          0.794      1761
   macro avg      0.737     0.706     0.718      1761
weighted avg      0.784     0.794     0.787      1761

probs = model.predict_proba(X_test.head(5))[:, 1]
print(probs.round(3))
print(y_test.head(5).to_list())
[0.159 0.04  0.057 0.737 0.013]
[0, 0, 0, 0, 0]
p_all = model.predict_proba(X_test)[:, 1]
fig, ax = plt.subplots(figsize=(7, 4))
ax.hist(p_all[y_test == 0], bins=20, alpha=0.6, label="Stayed")
ax.hist(p_all[y_test == 1], bins=20, alpha=0.6, label="Left")
ax.set_title("Predicted churn probability by outcome")
ax.set_xlabel("Predicted probability of leaving")
ax.set_ylabel("Number of customers")
ax.legend()
No description has been provided for this image
print(round(np.median(p_all[y_test == 0]), 3))
print(round(np.median(p_all[y_test == 1]), 3))
0.105
0.522

Common mistake: scoring a raw customer row that was never encoded

model.predict_proba(test.head(1))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[23], line 1
----> 1 model.predict_proba(test.head(1))

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\pipeline.py:925, in Pipeline.predict_proba(self, X, **params)
    923 if not _routing_enabled():
    924     for _, name, transform in self._iter(with_final=False):
--> 925         Xt = transform.transform(Xt)
    926     return self.steps[-1][1].predict_proba(Xt, **params)
    928 # metadata routing enabled

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_set_output.py:327, in _wrap_method_output.<locals>.wrapped(self, X, *args, **kwargs)
    325 @wraps(f)
    326 def wrapped(self, X, *args, **kwargs):
--> 327     data_to_wrap = f(self, X, *args, **kwargs)
    328     if isinstance(data_to_wrap, tuple):
    329         # only wrap the first output for cross decomposition
    330         return_tuple = (
    331             _wrap_data_with_container(method, data_to_wrap[0], X, self),
    332             *data_to_wrap[1:],
    333         )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\preprocessing\_data.py:1111, in StandardScaler.transform(self, X, copy)
   1108 check_is_fitted(self)
   1110 copy = copy if copy is not None else self.copy
-> 1111 X = validate_data(
   1112     self,
   1113     X,
   1114     reset=False,
   1115     accept_sparse="csr",
   1116     copy=copy,
   1117     dtype=supported_float_dtypes(xp, X_device),
   1118     force_writeable=True,
   1119     ensure_all_finite="allow-nan",
   1120 )
   1122 if sparse.issparse(X):
   1123     if self.with_mean:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:3016, in validate_data(_estimator, X, y, reset, validate_separately, skip_check_array, **check_params)
   2932 def validate_data(
   2933     _estimator,
   2934     /,
   (...)
   2940     **check_params,
   2941 ):
   2942     """Validate input data and set or check feature names and counts of the input.
   2943 
   2944     This helper function should be used in an estimator that requires input
   (...)
   3014         validated.
   3015     """
-> 3016     _check_feature_names(_estimator, X, reset=reset)
   3017     tags = get_tags(_estimator)
   3018     if y is None and tags.target_tags.required:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:2868, in _check_feature_names(estimator, X, reset)
   2865 if not missing_names and not unexpected_names:
   2866     message += "Feature names must be in the same order as they were in fit.\n"
-> 2868 raise ValueError(message)

ValueError: The feature names should match those that were passed during fit.
Feature names unseen at fit time:
- Churn
- Contract
- Dependents
- DeviceProtection
- InternetService
- ...
Feature names seen at fit time, yet now missing:
- Contract_One year
- Contract_Two year
- Dependents_Yes
- InternetService_Fiber optic
- InternetService_No
- ...
model.predict_proba(X_test.head(1)).round(3)
array([[0.841, 0.159]])

Read the coefficients as odds#

Problem: a manager asks why customers leave, not just who.

coefs = pd.Series(model[-1].coef_[0], index=X.columns)
coefs.sort_values().round(2)
tenure                                  -1.26
Contract_Two year                       -0.63
InternetService_No                      -0.34
Contract_One year                       -0.31
Dependents_Yes                          -0.11
MonthlyCharges                          -0.07
PaymentMethod_Credit card (automatic)   -0.01
Partner_Yes                              0.00
PaymentMethod_Mailed check               0.02
SeniorCitizen                            0.09
PaymentMethod_Electronic check           0.21
PaperlessBilling_Yes                     0.23
InternetService_Fiber optic              0.45
TotalCharges                             0.57
dtype: float64

Quick check: tenure lowers churn, yet TotalCharges raises it. How can both be true?

df[["tenure", "MonthlyCharges", "TotalCharges"]].corr().round(2)
tenure MonthlyCharges TotalCharges
tenure 1.00 0.25 0.83
MonthlyCharges 0.25 1.00 0.65
TotalCharges 0.83 0.65 1.00
X_train2 = X_train.drop(columns="TotalCharges")
X_test2 = X_test.drop(columns="TotalCharges")
model2 = make_pipeline(StandardScaler(),
                       LogisticRegression(max_iter=1000))
model2.fit(X_train2, y_train)
print(round(model2.score(X_test2, y_test), 3))
coefs2 = pd.Series(model2[-1].coef_[0], index=X_train2.columns)
print(round(coefs2["tenure"], 2))
0.792
-0.76
  • Odds = p / (1 - p): a 0.2 chance is odds of 1 to 4, or 0.25
  • Each weight multiplies the odds by exp(weight)
p = model2.predict_proba(X_test2.head(1))[0, 1]
print(round(p, 3), round(p / (1 - p), 3))
0.165 0.197
per_unit = model2[-1].coef_[0] / model2[0].scale_
odds = pd.Series(np.exp(per_unit), index=X_train2.columns)
odds.sort_values().round(3)
Contract_Two year                        0.236
Contract_One year                        0.466
InternetService_No                       0.494
Dependents_Yes                           0.775
PaymentMethod_Credit card (automatic)    0.964
tenure                                   0.970
MonthlyCharges                           1.004
Partner_Yes                              1.007
PaymentMethod_Mailed check               1.091
SeniorCitizen                            1.285
PaymentMethod_Electronic check           1.567
PaperlessBilling_Yes                     1.608
InternetService_Fiber optic              2.359
dtype: float64
ax = odds.sort_values().plot(kind="barh", figsize=(7, 5))
ax.axvline(1, color="grey", linestyle="--")
ax.set_title("What moves the odds of churn")
ax.set_xlabel("Odds ratio (below 1 protects, above 1 risks)")
ax.set_ylabel("Feature")
plt.show()
No description has been provided for this image

Your turn: tenure's odds ratio is per month. What is it for a whole year?

print(round(odds["tenure"] ** 12, 3))
0.691

Regularisation with C#

Problem: a model can lean too hard on noisy features. C sets how far it may.

from sklearn.model_selection import cross_val_score
for c in [0.01, 0.1, 1, 10]:
    lr = LogisticRegression(C=c, max_iter=1000)
    pipe = make_pipeline(StandardScaler(), lr)
    f1 = cross_val_score(pipe, X_train2, y_train, cv=5,
                         scoring="f1").mean()
    print(c, round(f1, 3))
0.01 0.548
0.1 0.574
1 0.576
10 0.576

Quick check: is it worth moving away from the default C of 1?

strict = make_pipeline(StandardScaler(),
                       LogisticRegression(C=0.01, max_iter=1000))
strict.fit(X_train2, y_train)
print(round(np.abs(strict[-1].coef_).sum(), 2))
print(round(np.abs(model2[-1].coef_).sum(), 2))
2.87
3.24

Mini-project: who gets a retention call?#

Question: the team can call 1 test customer in 5. How many leavers does the list reach?

p_test = model2.predict_proba(X_test2)[:, 1]
cut = np.quantile(p_test, 0.8)
call = p_test >= cut
print(call.sum(), round(cut, 3))
353 0.523
hit = y_test[call].mean()
base = y_test.mean()
print(round(hit, 3), round(base, 3), round(hit / base, 2))
0.649 0.265 2.45
print(y_test[call].sum(), y_test.sum())
229 467
test.loc[call, "Contract"].value_counts(normalize=True).round(3)
Contract
Month-to-month    0.994
One year          0.006
Name: proportion, dtype: float64
groups = pd.qcut(p_test, 5, labels=["1 low", "2", "3", "4", "5 high"])
by_group = y_test.groupby(groups, observed=True).mean()
ax = by_group.plot(kind="bar", rot=0, figsize=(7, 4))
ax.set_title("Real churn rate by predicted risk group")
ax.set_xlabel("Risk group (fifths of customers)")
ax.set_ylabel("Share who actually left")
print(by_group.round(3))
1 low     0.008
2         0.071
3         0.224
4         0.375
5 high    0.648
Name: churn, dtype: float64
No description has been provided for this image
reached = y_test[call].sum() / y_test.sum()
print(f"Calling 20% of customers reaches {reached:.0%} of leavers,")
print(f"{hit / base:.1f} times better than calling at random.")
Calling 20% of customers reaches 49% of leavers,
2.4 times better than calling at random.

Recap#

  • The sigmoid squashes a weighted score into a probability
  • Coerce messy text to numbers, then fill gaps with an honest value
  • Scale inside a pipeline; score rows as tables, double brackets
  • exp(weight) is an odds ratio; drop copycat columns to read them
  • Tune C with cross-validation; rank by probability to target action

Challenges#

  1. Add TechSupport and OnlineSecurity. Do they protect against churn?
  2. Try class_weight='balanced'. What happens to recall for leavers?
  3. Build a call list for the riskiest 10%. What is its lift?

What's next#

Lesson 18: Support Vector Machines (SVM) Find the widest boundary between classes, and bend it with kernels.

Data Mining 18: Support Vector Machines (SVM)#

In this lesson you'll be able to:

  • Explain margins and support vectors with a real picture
  • Handle hidden missing values and scale features for an SVM
  • Compare linear and RBF kernels
  • Tune C and the kernel with cross-validation, then test once
  • Turn SVM scores into a screening rule for a clinic Dataset: Pima Indians diabetes, 768 women (data/mining/pima_diabetes.csv)

Warm-up: logistic regression on diabetes#

Lesson 17 built logistic regression. Here's one on today's data.

import pandas as pd
df = pd.read_csv("pima_diabetes.csv")
print(df.shape)
print(round(df["outcome"].mean(), 3))
(768, 9)
0.349
from sklearn.model_selection import train_test_split
train, test = train_test_split(
    df, test_size=0.25, stratify=df["outcome"], random_state=42)
feats = list(df.columns[:-1])
print(len(train), len(test))
576 192
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
lr = make_pipeline(StandardScaler(), LogisticRegression())
lr.fit(train[feats], train["outcome"])
print(round(lr.score(test[feats], test["outcome"]), 3))
0.734

Find the missing values hiding as zeros#

Problem: a blood pressure of 0 isn't a measurement. It's a missing one.

hidden = ["glucose", "blood_pressure", "skin_thickness", "insulin",
          "bmi"]
(df[hidden] == 0).sum()
glucose             5
blood_pressure     35
skin_thickness    227
insulin           374
bmi                11
dtype: int64
import numpy as np
X = df[feats].copy()
X[hidden] = X[hidden].replace(0, np.nan)
X_train, X_test = X.loc[train.index], X.loc[test.index]
y_train, y_test = train["outcome"], test["outcome"]
print(X_train.isna().sum().sum())
475
print(round(df["insulin"].mean(), 1))
print(round(X["insulin"].mean(), 1))
79.8
155.5

Common mistake: fitting an SVM on data that still has gaps

from sklearn.svm import SVC
SVC().fit(X_train, y_train)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[8], line 2
      1 from sklearn.svm import SVC
----> 2 SVC().fit(X_train, y_train)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py:1403, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1393     estimator._validate_params()
   1395 with (
   1396     config_context(
   1397         skip_parameter_validation=(
   (...)
   1401     callback_management_context(estimator),
   1402 ):
-> 1403     return fit_method(estimator, *args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\svm\_base.py:211, in BaseLibSVM.fit(self, X, y, sample_weight)
    209     check_consistent_length(X, y)
    210 else:
--> 211     X, y = validate_data(
    212         self,
    213         X,
    214         y,
    215         dtype=np.float64,
    216         order="C",
    217         accept_sparse="csr",
    218         accept_large_sparse=False,
    219     )
    221 y = self._validate_targets(y)
    223 sample_weight = np.asarray(
    224     [] if sample_weight is None else sample_weight, dtype=np.float64
    225 )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:3058, in validate_data(_estimator, X, y, reset, validate_separately, skip_check_array, **check_params)
   3056         y = check_array(y, input_name="y", **check_y_params)
   3057     else:
-> 3058         X, y = check_X_y(X, y, **check_params)
   3059     out = X, y
   3061 if not no_val_X and check_params.get("ensure_2d", True):

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:1330, in check_X_y(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, ensure_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, estimator)
   1325         estimator_name = _check_estimator_name(estimator)
   1326     raise ValueError(
   1327         f"{estimator_name} requires y to be passed, but the target y is None"
   1328     )
-> 1330 X = check_array(
   1331     X,
   1332     accept_sparse=accept_sparse,
   1333     accept_large_sparse=accept_large_sparse,
   1334     dtype=dtype,
   1335     order=order,
   1336     copy=copy,
   1337     force_writeable=force_writeable,
   1338     ensure_all_finite=ensure_all_finite,
   1339     ensure_2d=ensure_2d,
   1340     allow_nd=allow_nd,
   1341     ensure_min_samples=ensure_min_samples,
   1342     ensure_min_features=ensure_min_features,
   1343     estimator=estimator,
   1344     input_name="X",
   1345 )
   1347 y = _check_y(y, multi_output=multi_output, y_numeric=y_numeric, estimator=estimator)
   1349 check_consistent_length(X, y)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:1090, in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, ensure_all_finite, ensure_non_negative, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)
   1084     raise ValueError(
   1085         f"Found array with dim {array.ndim},"
   1086         f" while dim <= 2 is required{context}."
   1087     )
   1089 if ensure_all_finite:
-> 1090     _assert_all_finite(
   1091         array,
   1092         input_name=input_name,
   1093         estimator_name=estimator_name,
   1094         allow_nan=ensure_all_finite == "allow-nan",
   1095     )
   1097 if copy:
   1098     if _is_numpy_namespace(xp):
   1099         # only make a copy if `array` and `array_orig` may share memory`

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:137, in _assert_all_finite(X, allow_nan, msg_dtype, estimator_name, input_name)
    134 if first_pass_isfinite:
    135     return
--> 137 _assert_all_finite_element_wise(
    138     X,
    139     xp=xp,
    140     allow_nan=allow_nan,
    141     msg_dtype=msg_dtype,
    142     estimator_name=estimator_name,
    143     input_name=input_name,
    144 )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:186, in _assert_all_finite_element_wise(X, xp, allow_nan, msg_dtype, estimator_name, input_name)
    169 if estimator_name and input_name == "X" and has_nan_error:
    170     # Improve the error message on how to handle missing values in
    171     # scikit-learn.
    172     msg_err += (
    173         f"\n{estimator_name} does not accept missing values"
    174         " encoded as NaN natively. For supervised learning, you might want"
   (...)
    184         "#estimators-that-handle-nan-values"
    185     )
--> 186 raise ValueError(msg_err)

ValueError: Input X contains NaN.
SVC does not accept missing values encoded as NaN natively. For supervised learning, you might want to consider sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept missing values encoded as NaNs natively. Alternatively, it is possible to preprocess the data, for instance by using an imputer transformer in a pipeline or drop samples with missing values. See https://scikit-learn.org/stable/modules/impute.html You can find a list of all estimators that handle NaN values at the following page: https://scikit-learn.org/stable/modules/impute.html#estimators-that-handle-nan-values
from sklearn.impute import SimpleImputer
from sklearn.model_selection import cross_val_score
svm = make_pipeline(SimpleImputer(strategy="median"),
                    StandardScaler(), SVC())
f1 = cross_val_score(svm, X_train, y_train, cv=5, scoring="f1")
print(round(f1.mean(), 3))
0.631

Quick check: why fill medians inside the pipeline, not on the whole table first?

print(X_train["insulin"].median())
print(X_test["insulin"].median())
125.0
130.0

The widest street: margins and support vectors#

Problem: many lines can split two classes. Which one should we trust?

  • The SVM picks the boundary with the widest margin, a street between classes
  • Patients on the edge of the street are the support vectors
  • C sets how many patients may stand inside the street
two = ["glucose", "bmi"]
d2 = X_train[two].dropna()
y2 = y_train.loc[d2.index]
lin2 = make_pipeline(StandardScaler(), SVC(kernel="linear"))
lin2.fit(d2, y2)
print(len(d2), lin2[-1].n_support_)
563 [148 148]
print(lin2.decision_function(d2.head(3)).round(2))
print(y2.head(3).to_list())
[-0.32 -1.58 -0.94]
[0, 0, 0]
import matplotlib.pyplot as plt
from sklearn.inspection import DecisionBoundaryDisplay
fig, ax = plt.subplots(figsize=(7, 5))
DecisionBoundaryDisplay.from_estimator(
    lin2, d2, response_method="decision_function", ax=ax,
    plot_method="contour", levels=[-1, 0, 1], colors="k",
    linestyles=["--", "-", "--"])
ax.scatter(d2["glucose"], d2["bmi"], c=y2, cmap="coolwarm", s=10)
ax.set_title("Linear SVM: the street between the classes")
ax.set_xlabel("Glucose")
ax.set_ylabel("BMI")
No description has been provided for this image

Quick check: if C gets smaller, does the street get wider or narrower?

for c in [0.01, 1, 100]:
    lin = SVC(kernel="linear", C=c)
    m = make_pipeline(StandardScaler(), lin).fit(d2, y2)
    print(c, m[-1].n_support_.sum())
0.01 361
1 296
100 295

Your turn: fit an RBF SVM on glucose and BMI. How many support vectors does it keep?

rbf2 = make_pipeline(StandardScaler(), SVC(kernel="rbf"))
rbf2.fit(d2, y2)
print(rbf2[-1].n_support_, rbf2[-1].n_support_.sum())
[145 142] 287

Why scaling matters for SVMs#

Problem: the SVM measures distance, so big-number features drown out small ones.

X_train[["insulin", "diabetes_pedigree"]].describe().round(2)
insulin diabetes_pedigree
count 303.00 576.00
mean 148.09 0.48
std 107.42 0.33
min 16.00 0.08
25% 76.00 0.25
50% 125.00 0.38
75% 181.00 0.65
max 744.00 2.33
raw = make_pipeline(SimpleImputer(strategy="median"), SVC())
for name, m in [("unscaled", raw), ("scaled", svm)]:
    s = cross_val_score(m, X_train, y_train, cv=5, scoring="f1")
    print(name, round(s.mean(), 3))
unscaled 0.603
scaled 0.631

Your turn: which raw feature has the biggest spread? Sort the standard deviations.

X_train.std().round(1).sort_values()
diabetes_pedigree      0.3
pregnancies            3.3
bmi                    6.8
skin_thickness        10.5
age                   11.9
blood_pressure        12.5
glucose               30.2
insulin              107.4
dtype: float64

Kernels: bending the boundary#

Problem: sometimes no straight street can separate the classes.

  • Linear kernel: a straight boundary
  • RBF kernel: similarity to nearby patients, so the boundary can curve
  • gamma sets how far each patient's influence reaches
fig, axes = plt.subplots(1, 2, figsize=(10, 4))
for ax, k in zip(axes, ["linear", "rbf"]):
    m = make_pipeline(StandardScaler(), SVC(kernel=k)).fit(d2, y2)
    DecisionBoundaryDisplay.from_estimator(m, d2, ax=ax, alpha=0.3,
        response_method="predict", cmap="coolwarm")
    ax.scatter(d2["glucose"], d2["bmi"], c=y2, cmap="coolwarm", s=8)
    ax.set_title(f"{k} kernel")
    ax.set_xlabel("Glucose")
    ax.set_ylabel("BMI")
plt.tight_layout()
No description has been provided for this image
for k in ["linear", "rbf", "poly"]:
    m = make_pipeline(SimpleImputer(strategy="median"),
                      StandardScaler(), SVC(kernel=k))
    s = cross_val_score(m, X_train, y_train, cv=5, scoring="f1")
    print(k, round(s.mean(), 3))
linear 0.647
rbf 0.631
poly 0.506

Your turn: set gamma=1 on the RBF SVM. Compare training accuracy with cross-validated accuracy.

wiggly = make_pipeline(SimpleImputer(strategy="median"),
                       StandardScaler(), SVC(gamma=1))
wiggly.fit(X_train, y_train)
print(round(wiggly.score(X_train, y_train), 3))
print(round(cross_val_score(wiggly, X_train, y_train, cv=5).mean(), 3))
0.969
0.688
gammas = [0.01, 0.1, 1, 10]
fit_acc, cv_acc = [], []
for g in gammas:
    m = make_pipeline(SimpleImputer(strategy="median"),
                      StandardScaler(), SVC(gamma=g))
    fit_acc.append(m.fit(X_train, y_train).score(X_train, y_train))
    cv_acc.append(cross_val_score(m, X_train, y_train, cv=5).mean())
ax = pd.DataFrame({"training": fit_acc, "cross-validated": cv_acc},
                  index=gammas).plot(marker="o", logx=True)
ax.set_title("Larger gamma memorises the training patients")
ax.set_xlabel("gamma (log scale)")
ax.set_ylabel("Accuracy")
No description has been provided for this image

Tune C and the kernel, then test once#

Problem: choose the settings without ever peeking at the test set.

from sklearn.model_selection import GridSearchCV
options = dict(svc__kernel=["linear", "rbf"], svc__C=[0.1, 1, 10])
grid = GridSearchCV(svm, options, cv=5, scoring="f1")
grid.fit(X_train, y_train)
print(grid.best_params_, round(grid.best_score_, 3))
{'svc__C': 1, 'svc__kernel': 'linear'} 0.647
res = pd.DataFrame(grid.cv_results_)
cols = ["param_svc__kernel", "param_svc__C", "mean_test_score"]
res.sort_values("rank_test_score")[cols].head(4).round(3)
param_svc__kernel param_svc__C mean_test_score
2 linear 1.0 0.647
4 linear 10.0 0.645
0 linear 0.1 0.643
3 rbf 1.0 0.631

Quick check: the top three differ by 0.004. Is that difference real?

top = res.sort_values("rank_test_score").head(3)
top[["mean_test_score", "std_test_score"]].round(3)
mean_test_score std_test_score
2 0.647 0.031
4 0.645 0.030
0 0.643 0.016
best = grid.best_estimator_
print(round(best.score(X_test, y_test), 3))
print(round(lr.score(test[feats], test["outcome"]), 3))
0.714
0.734
from sklearn.metrics import confusion_matrix, classification_report
pred = best.predict(X_test)
print(confusion_matrix(y_test, pred))
print(classification_report(y_test, pred, digits=3))
[[105  20]
 [ 35  32]]
              precision    recall  f1-score   support

           0      0.750     0.840     0.792       125
           1      0.615     0.478     0.538        67

    accuracy                          0.714       192
   macro avg      0.683     0.659     0.665       192
weighted avg      0.703     0.714     0.704       192

Common mistake: asking an SVM for probabilities

best.predict_proba(X_test.head(3))
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_available_if.py:32, in _AvailableIfDescriptor._check(self, obj, owner)
     31 try:
---> 32     check_result = self.check(obj)
     33 except Exception as e:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\svm\_base.py:866, in BaseSVC._check_proba(self)
    865 if self.probability == "deprecated" or not self.probability:
--> 866     raise AttributeError(
    867         "predict_proba is not available when probability=False"
    868     )
    869 if self._impl not in ("c_svc", "nu_svc"):

AttributeError: predict_proba is not available when probability=False

The above exception was the direct cause of the following exception:

AttributeError                            Traceback (most recent call last)
File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_available_if.py:32, in _AvailableIfDescriptor._check(self, obj, owner)
     31 try:
---> 32     check_result = self.check(obj)
     33 except Exception as e:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\pipeline.py:47, in _final_estimator_has.<locals>.check(self)
     45 def check(self):
     46     # raise original `AttributeError` if `attr` does not exist
---> 47     getattr(self._final_estimator, attr)
     48     return True

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_available_if.py:43, in _AvailableIfDescriptor.__get__(self, obj, owner)
     40 if obj is not None:
     41     # delegate only on instances, not the classes.
     42     # this is to allow access to the docstrings.
---> 43     self._check(obj, owner=owner)
     44     out = MethodType(self.fn, obj)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_available_if.py:34, in _AvailableIfDescriptor._check(self, obj, owner)
     33 except Exception as e:
---> 34     raise AttributeError(attr_err_msg) from e
     36 if not check_result:

AttributeError: This 'SVC' has no attribute 'predict_proba'

The above exception was the direct cause of the following exception:

AttributeError                            Traceback (most recent call last)
Cell In[28], line 1
----> 1 best.predict_proba(X_test.head(3))

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_available_if.py:43, in _AvailableIfDescriptor.__get__(self, obj, owner)
     39 def __get__(self, obj, owner=None):
     40     if obj is not None:
     41         # delegate only on instances, not the classes.
     42         # this is to allow access to the docstrings.
---> 43         self._check(obj, owner=owner)
     44         out = MethodType(self.fn, obj)
     46     else:
     47         # This makes it possible to use the decorated method as an unbound method,
     48         # for instance when monkeypatching.

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_available_if.py:34, in _AvailableIfDescriptor._check(self, obj, owner)
     32     check_result = self.check(obj)
     33 except Exception as e:
---> 34     raise AttributeError(attr_err_msg) from e
     36 if not check_result:
     37     raise AttributeError(attr_err_msg)

AttributeError: This 'Pipeline' has no attribute 'predict_proba'
print(best.decision_function(X_test.head(3)).round(2))
print(y_test.head(3).to_list())
[-0.35 -0.21 -1.73]
[1, 0, 0]

Mini-project: a screening rule for a clinic#

Question: catch at least 80% of diabetic patients. How many must be sent for a test?

from sklearn.model_selection import cross_val_predict
s_cv = cross_val_predict(best, X_train, y_train, cv=5,
                         method="decision_function")
cut = np.quantile(s_cv[(y_train == 1).to_numpy()], 0.2)
print(round(cut, 3))
-0.77
fig, ax = plt.subplots(figsize=(7, 4))
ax.hist(s_cv[(y_train == 0).to_numpy()], bins=30, alpha=0.6,
        label="No diabetes")
ax.hist(s_cv[(y_train == 1).to_numpy()], bins=30, alpha=0.6,
        label="Diabetes")
ax.axvline(0, color="grey", linestyle="--", label="Default")
ax.axvline(cut, color="black", label="Screening cut-off")
ax.set_title("Moving the cut-off left catches more diabetes")
ax.set_xlabel("SVM decision score")
ax.set_ylabel("Number of patients")
ax.legend()
No description has been provided for this image
s_test = best.decision_function(X_test)
has = (y_test == 1).to_numpy()
for name, line in [("default", 0), ("screening", cut)]:
    flag = s_test >= line
    print(name, round(flag[has].mean(), 3), round(flag.mean(), 3))
default 0.478 0.271
screening 0.791 0.474
flag = s_test >= cut
print(confusion_matrix(y_test, flag.astype(int)))
[[87 38]
 [14 53]]
print(X_test.loc[flag, "glucose"].median())
print(X_test.loc[~flag, "glucose"].median())
141.0
100.0
tests = flag.sum()
found = flag[has].sum()
print(tests, found, round(tests / found, 2))
91 53 1.72
caught = flag[has].mean()
sent = flag.mean()
print(f"Screening catches {caught:.0%} of diabetic patients")
print(f"by sending {sent:.0%} of patients for a test.")
Screening catches 79% of diabetic patients
by sending 47% of patients for a test.

Recap#

  • An SVM picks the boundary with the widest margin; support vectors hold it
  • Turn impossible zeros into NaN and impute inside the pipeline
  • Always scale: SVMs measure distance
  • Kernels bend the boundary; C and gamma control how tightly it fits
  • Tune with GridSearchCV, test once, and move the cut-off to match the cost

Challenges#

  1. Add gamma values [0.01, 0.1, 1] to the grid for the RBF kernel.
  2. Find the cut-off that catches 90% of diabetic training patients.
  3. Try SVC(class_weight='balanced'). How does recall change?

What's next#

Lesson 19: Introduction to Neural Networks for Data Mining Stack simple units into layers that learn their own features.

Data Mining 19: Introduction to Neural Networks for Data Mining#

In this lesson you'll be able to:

  • Explain a neuron, a layer and training in plain words
  • Train a neural network with scikit-learn's MLPClassifier
  • Look inside a trained network and redo one prediction by hand
  • Spot overfitting and tame it with regularisation
  • Score new patients and compare the network with simpler models Dataset: Pima Indians diabetes, 768 women (data/mining/pima_diabetes.csv)

Warm-up: the SVM from lesson 18#

Lesson 18 drew a boundary with a support vector machine. Let's set the score to beat.

import numpy as np
import pandas as pd
df = pd.read_csv("pima_diabetes.csv")
print(df.shape)
df.head()
(768, 9)
pregnancies glucose blood_pressure skin_thickness insulin bmi diabetes_pedigree age outcome
0 6 148 72 35 0 33.6 0.627 50 1
1 1 85 66 29 0 26.6 0.351 31 0
2 8 183 64 0 0 23.3 0.672 32 1
3 1 89 66 23 94 28.1 0.167 21 0
4 0 137 40 35 168 43.1 2.288 33 1
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score
quick_svm = make_pipeline(StandardScaler(), SVC())
raw_X = df.drop(columns="outcome")
cv = cross_val_score(quick_svm, raw_X, df["outcome"], cv=5)
print(cv.round(3), round(cv.mean(), 3))
[0.766 0.753 0.747 0.817 0.778] 0.772

Prepare data a network can learn from#

Problem: zeros that really mean 'not measured', and features on very different scales.

print(df["outcome"].value_counts())
print((df == 0).sum())
outcome
0    500
1    268
Name: count, dtype: int64
pregnancies          111
glucose                5
blood_pressure        35
skin_thickness       227
insulin              374
bmi                   11
diabetes_pedigree      0
age                    0
outcome              500
dtype: int64
fake_zero = ["glucose", "blood_pressure", "skin_thickness",
             "insulin", "bmi"]
df[fake_zero] = df[fake_zero].replace(0, np.nan)
df[fake_zero].isna().sum()
glucose             5
blood_pressure     35
skin_thickness    227
insulin           374
bmi                11
dtype: int64
new_patients = df.tail(5)
df = df.iloc[:-5]
X = df.drop(columns="outcome")
y = df["outcome"]
print(X.shape, y.shape)
(763, 8) (763,)
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)
print(len(X_train), len(X_test))
print(round(y_train.mean(), 3), round(y_test.mean(), 3))
572 191
0.35 0.351

Common mistake: feeding missing values straight into a network

from sklearn.neural_network import MLPClassifier
net = MLPClassifier(random_state=42)
net.fit(X_train, y_train)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[8], line 3
      1 from sklearn.neural_network import MLPClassifier
      2 net = MLPClassifier(random_state=42)
----> 3 net.fit(X_train, y_train)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py:1403, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1393     estimator._validate_params()
   1395 with (
   1396     config_context(
   1397         skip_parameter_validation=(
   (...)
   1401     callback_management_context(estimator),
   1402 ):
-> 1403     return fit_method(estimator, *args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\neural_network\_multilayer_perceptron.py:853, in BaseMultilayerPerceptron.fit(self, X, y, sample_weight)
    830 @_fit_context(prefer_skip_nested_validation=True)
    831 def fit(self, X, y, sample_weight=None):
    832     """Fit the model to data matrix X and target(s) y.
    833 
    834     Parameters
   (...)
    851         Returns a trained MLP model.
    852     """
--> 853     return self._fit(X, y, sample_weight=sample_weight, incremental=False)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\neural_network\_multilayer_perceptron.py:477, in BaseMultilayerPerceptron._fit(self, X, y, sample_weight, incremental)
    470     raise ValueError(
    471         "hidden_layer_sizes must be > 0, got %s." % hidden_layer_sizes
    472     )
    473 first_pass = not hasattr(self, "coefs_") or (
    474     not self.warm_start and not incremental
    475 )
--> 477 X, y = self._validate_input(X, y, incremental, reset=first_pass)
    478 n_samples, n_features = X.shape
    479 if sample_weight is not None:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\neural_network\_multilayer_perceptron.py:1213, in MLPClassifier._validate_input(self, X, y, incremental, reset)
   1212 def _validate_input(self, X, y, incremental, reset):
-> 1213     X, y = validate_data(
   1214         self,
   1215         X,
   1216         y,
   1217         accept_sparse=["csr", "csc"],
   1218         multi_output=True,
   1219         dtype=(np.float64, np.float32),
   1220         reset=reset,
   1221     )
   1222     if y.ndim == 2 and y.shape[1] == 1:
   1223         y = column_or_1d(y, warn=True)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:3058, in validate_data(_estimator, X, y, reset, validate_separately, skip_check_array, **check_params)
   3056         y = check_array(y, input_name="y", **check_y_params)
   3057     else:
-> 3058         X, y = check_X_y(X, y, **check_params)
   3059     out = X, y
   3061 if not no_val_X and check_params.get("ensure_2d", True):

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:1330, in check_X_y(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, ensure_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, estimator)
   1325         estimator_name = _check_estimator_name(estimator)
   1326     raise ValueError(
   1327         f"{estimator_name} requires y to be passed, but the target y is None"
   1328     )
-> 1330 X = check_array(
   1331     X,
   1332     accept_sparse=accept_sparse,
   1333     accept_large_sparse=accept_large_sparse,
   1334     dtype=dtype,
   1335     order=order,
   1336     copy=copy,
   1337     force_writeable=force_writeable,
   1338     ensure_all_finite=ensure_all_finite,
   1339     ensure_2d=ensure_2d,
   1340     allow_nd=allow_nd,
   1341     ensure_min_samples=ensure_min_samples,
   1342     ensure_min_features=ensure_min_features,
   1343     estimator=estimator,
   1344     input_name="X",
   1345 )
   1347 y = _check_y(y, multi_output=multi_output, y_numeric=y_numeric, estimator=estimator)
   1349 check_consistent_length(X, y)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:1090, in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, ensure_all_finite, ensure_non_negative, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)
   1084     raise ValueError(
   1085         f"Found array with dim {array.ndim},"
   1086         f" while dim <= 2 is required{context}."
   1087     )
   1089 if ensure_all_finite:
-> 1090     _assert_all_finite(
   1091         array,
   1092         input_name=input_name,
   1093         estimator_name=estimator_name,
   1094         allow_nan=ensure_all_finite == "allow-nan",
   1095     )
   1097 if copy:
   1098     if _is_numpy_namespace(xp):
   1099         # only make a copy if `array` and `array_orig` may share memory`

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:137, in _assert_all_finite(X, allow_nan, msg_dtype, estimator_name, input_name)
    134 if first_pass_isfinite:
    135     return
--> 137 _assert_all_finite_element_wise(
    138     X,
    139     xp=xp,
    140     allow_nan=allow_nan,
    141     msg_dtype=msg_dtype,
    142     estimator_name=estimator_name,
    143     input_name=input_name,
    144 )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:186, in _assert_all_finite_element_wise(X, xp, allow_nan, msg_dtype, estimator_name, input_name)
    169 if estimator_name and input_name == "X" and has_nan_error:
    170     # Improve the error message on how to handle missing values in
    171     # scikit-learn.
    172     msg_err += (
    173         f"\n{estimator_name} does not accept missing values"
    174         " encoded as NaN natively. For supervised learning, you might want"
   (...)
    184         "#estimators-that-handle-nan-values"
    185     )
--> 186 raise ValueError(msg_err)

ValueError: Input X contains NaN.
MLPClassifier does not accept missing values encoded as NaN natively. For supervised learning, you might want to consider sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept missing values encoded as NaNs natively. Alternatively, it is possible to preprocess the data, for instance by using an imputer transformer in a pipeline or drop samples with missing values. See https://scikit-learn.org/stable/modules/impute.html You can find a list of all estimators that handle NaN values at the following page: https://scikit-learn.org/stable/modules/impute.html#estimators-that-handle-nan-values
from sklearn.impute import SimpleImputer
prep = make_pipeline(SimpleImputer(strategy="median"),
                     StandardScaler())
X_train_ready = prep.fit_transform(X_train)
print(np.isnan(X_train_ready).sum())
print(X_train_ready.mean(axis=0).round(2) + 0)
print(X_train_ready.std(axis=0).round(2))
0
[0. 0. 0. 0. 0. 0. 0. 0.]
[1. 1. 1. 1. 1. 1. 1. 1.]

Quick check: why does scaling matter so much for a network?

X_train[["insulin", "diabetes_pedigree"]].describe().round(2)
insulin diabetes_pedigree
count 304.00 572.00
mean 147.82 0.48
std 108.87 0.33
min 15.00 0.08
25% 75.75 0.25
50% 123.50 0.39
75% 180.00 0.64
max 744.00 2.33

Train your first neural network#

Problem: let the model invent its own useful combinations of features.

import matplotlib.pyplot as plt
colours = y_train.map({0: "tab:blue", 1: "tab:red"})
plt.figure(figsize=(7, 4))
plt.scatter(X_train["glucose"], X_train["bmi"], c=colours, s=12)
plt.title("Glucose vs BMI: no single clean dividing line")
plt.xlabel("Glucose")
plt.ylabel("BMI")
plt.show()
No description has been provided for this image
  • Neuron: weighted sum of inputs, then an on-off style switch
  • Layer: several neurons side by side, each finding a pattern
  • Training: nudge every weight to shrink the prediction error
mlp = make_pipeline(
    SimpleImputer(strategy="median"), StandardScaler(),
    MLPClassifier(hidden_layer_sizes=(8,), max_iter=2000,
                  random_state=42))
mlp.fit(X_train, y_train)
print("train:", round(mlp.score(X_train, y_train), 3))
print("test: ", round(mlp.score(X_test, y_test), 3))
train: 0.797
test:  0.723
svm = make_pipeline(
    SimpleImputer(strategy="median"), StandardScaler(),
    SVC(random_state=42))
svm.fit(X_train, y_train)
print("SVM test:", round(svm.score(X_test, y_test), 3))
SVM test: 0.723
probs = mlp.predict_proba(X_test.head())[:, 1]
print(probs.round(3))
print(mlp.predict(X_test.head()))
print(y_test.head().to_numpy())
[0.325 0.315 0.008 0.858 0.552]
[0 0 0 1 1]
[0 0 0 1 0]
net = mlp[-1]
plt.figure(figsize=(7, 4))
plt.plot(net.loss_curve_)
plt.title("How the network learned: training loss per round")
plt.xlabel("Training round (epoch)")
plt.ylabel("Loss (lower is better)")
plt.show()
No description has been provided for this image

Quick check: the network scored higher on training than on test patients. Is that a warning sign?

gap = mlp.score(X_train, y_train) - mlp.score(X_test, y_test)
print("gap:", round(gap, 3))
gap: 0.075

Your turn#

Train a network with just 4 hidden neurons. Report its train and test accuracy.

mlp4 = make_pipeline(
    SimpleImputer(strategy="median"), StandardScaler(),
    MLPClassifier(hidden_layer_sizes=(4,), max_iter=2000,
                  random_state=42))
mlp4.fit(X_train, y_train)
print("train:", round(mlp4.score(X_train, y_train), 3))
print("test: ", round(mlp4.score(X_test, y_test), 3))
train: 0.787
test:  0.723

Look inside the network#

Problem: a network feels like a black box. It is really just weights and sums.

for i, w in enumerate(net.coefs_):
    print("weights", i, w.shape)
for i, b in enumerate(net.intercepts_):
    print("biases ", i, b.shape)
weights 0 (8, 8)
weights 1 (8, 1)
biases  0 (8,)
biases  1 (1,)

Quick check: how many numbers did training adjust in total?

n_weights = sum(w.size for w in net.coefs_)
n_biases = sum(b.size for b in net.intercepts_)
print(n_weights, n_biases, n_weights + n_biases)
72 9 81
w_in = pd.DataFrame(net.coefs_[0], index=X.columns)
w_in.columns = [f"n{i}" for i in range(8)]
w_in.round(2)
n0 n1 n2 n3 n4 n5 n6 n7
pregnancies -0.40 0.57 0.61 0.39 -0.71 -0.33 -0.34 0.42
glucose -0.21 0.46 -0.83 0.75 0.52 -0.66 -0.76 -0.52
blood_pressure -0.13 -0.23 -0.25 -0.38 -0.15 -0.01 -0.02 -0.33
skin_thickness 0.27 0.30 -0.30 -0.03 -0.06 -0.20 0.38 -0.58
insulin -0.68 0.32 0.20 0.07 -0.50 -0.12 0.55 0.19
bmi -0.68 0.15 -0.26 0.53 -0.21 0.04 -0.54 -0.36
diabetes_pedigree -0.33 -0.24 0.70 0.36 0.30 0.11 -0.23 0.29
age -0.64 -0.14 -0.29 -0.10 -0.32 -0.68 0.54 0.11
strength = w_in.abs().sum(axis=1)
strength.sort_values(ascending=False).round(2)
glucose              4.70
pregnancies          3.78
age                  2.82
bmi                  2.77
insulin              2.63
diabetes_pedigree    2.56
skin_thickness       2.12
blood_pressure       1.48
dtype: float64

A forward pass, step by step:

  1. Hidden layer: weighted sum + bias, then ReLU (negatives become 0)
  2. Output: weighted sum of hidden values + bias, then the logistic squash
x = prep.transform(X_test.head(1))
hidden = np.maximum(0, x @ net.coefs_[0] + net.intercepts_[0])
print(hidden.round(2))
z = hidden @ net.coefs_[1] + net.intercepts_[1]
p_hand = 1 / (1 + np.exp(-z))
print("by hand:", p_hand.round(4))
print("sklearn:", mlp.predict_proba(X_test.head(1))[:, 1].round(4))
[[0.   0.   0.   0.4  0.   1.28 0.56 0.  ]]
by hand: [[0.3249]]
sklearn: [0.3249]

Size, overfitting and regularisation#

Problem: bigger networks can memorise the training patients instead of learning.

X_fit, X_val, y_fit, y_val = train_test_split(
    X_train, y_train, test_size=0.25, random_state=42,
    stratify=y_train)
print(len(X_fit), len(X_val), len(X_test))
429 143 191
def build(sizes, alpha=0.0001):
    return make_pipeline(
        SimpleImputer(strategy="median"), StandardScaler(),
        MLPClassifier(hidden_layer_sizes=sizes, alpha=alpha,
                      max_iter=2000, random_state=42))
rows = []
for sizes in [(2,), (8,), (64, 64)]:
    model = build(sizes).fit(X_fit, y_fit)
    rows.append({"layers": str(sizes),
                 "fit": model.score(X_fit, y_fit),
                 "validation": model.score(X_val, y_val)})
sizes_df = pd.DataFrame(rows).round(3)
sizes_df
layers fit validation
0 (2,) 0.762 0.720
1 (8,) 0.795 0.797
2 (64, 64) 1.000 0.769
ax = sizes_df.plot(x="layers", kind="bar", rot=0, figsize=(7, 4))
ax.set_title("Bigger networks memorise: fit vs validation accuracy")
ax.set_xlabel("Hidden layers")
ax.set_ylabel("Accuracy")
ax.set_ylim(0.5, 1.0)
plt.show()
No description has been provided for this image
  • alpha adds a penalty for large weights (L2 regularisation)
  • early_stopping stops training when a validation slice stops improving
for a in [0.0001, 1, 3, 10]:
    model = build((64, 64), alpha=a).fit(X_fit, y_fit)
    tr = model.score(X_fit, y_fit)
    va = model.score(X_val, y_val)
    print(f"alpha={a}: fit {tr:.3f}, validation {va:.3f}")
alpha=0.0001: fit 1.000, validation 0.769
alpha=1: fit 0.814, validation 0.825
alpha=3: fit 0.762, validation 0.797
alpha=10: fit 0.718, validation 0.706

Quick check: which alpha would you pick, and why not decide using the test set?

final = build((64, 64), alpha=1).fit(X_train, y_train)
print("test:", round(final.score(X_test, y_test), 3))
test: 0.738

Your turn#

Train the (64, 64) network with early_stopping=True on X_fit. Compare fit and validation accuracy.

early = make_pipeline(
    SimpleImputer(strategy="median"), StandardScaler(),
    MLPClassifier(hidden_layer_sizes=(64, 64), early_stopping=True,
                  max_iter=2000, random_state=42))
early.fit(X_fit, y_fit)
print("fit:       ", round(early.score(X_fit, y_fit), 3))
print("validation:", round(early.score(X_val, y_val), 3))
fit:        0.767
validation: 0.769

Score new patients#

Problem: use the trained network on people it has never seen.

Common mistake: passing the new rows with the answer column still attached

final.predict(new_patients)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[30], line 1
----> 1 final.predict(new_patients)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\pipeline.py:794, in Pipeline.predict(self, X, **params)
    792 if not _routing_enabled():
    793     for _, name, transform in self._iter(with_final=False):
--> 794         Xt = transform.transform(Xt)
    795     return self.steps[-1][1].predict(Xt, **params)
    797 # metadata routing enabled

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_set_output.py:327, in _wrap_method_output.<locals>.wrapped(self, X, *args, **kwargs)
    325 @wraps(f)
    326 def wrapped(self, X, *args, **kwargs):
--> 327     data_to_wrap = f(self, X, *args, **kwargs)
    328     if isinstance(data_to_wrap, tuple):
    329         # only wrap the first output for cross decomposition
    330         return_tuple = (
    331             _wrap_data_with_container(method, data_to_wrap[0], X, self),
    332             *data_to_wrap[1:],
    333         )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\impute\_base.py:617, in SimpleImputer.transform(self, X)
    602 """Impute all missing values in `X`.
    603 
    604 Parameters
   (...)
    613     `X` with imputed values.
    614 """
    615 check_is_fitted(self)
--> 617 X = self._validate_input(X, in_fit=False)
    618 statistics = self.statistics_
    620 if X.shape[1] != statistics.shape[0]:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\impute\_base.py:375, in SimpleImputer._validate_input(self, X, in_fit)
    373         raise new_ve from None
    374     else:
--> 375         raise ve
    377 if in_fit:
    378     # Use the dtype seen in `fit` for non-`fit` conversion
    379     self._fit_dtype = X.dtype

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\impute\_base.py:356, in SimpleImputer._validate_input(self, X, in_fit)
    353     ensure_all_finite = True
    355 try:
--> 356     X = validate_data(
    357         self,
    358         X,
    359         reset=in_fit,
    360         accept_sparse="csc",
    361         dtype=dtype,
    362         force_writeable=True if not in_fit else None,
    363         ensure_all_finite=ensure_all_finite,
    364         copy=self.copy,
    365     )
    366 except ValueError as ve:
    367     if "could not convert" in str(ve):

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:3016, in validate_data(_estimator, X, y, reset, validate_separately, skip_check_array, **check_params)
   2932 def validate_data(
   2933     _estimator,
   2934     /,
   (...)
   2940     **check_params,
   2941 ):
   2942     """Validate input data and set or check feature names and counts of the input.
   2943 
   2944     This helper function should be used in an estimator that requires input
   (...)
   3014         validated.
   3015     """
-> 3016     _check_feature_names(_estimator, X, reset=reset)
   3017     tags = get_tags(_estimator)
   3018     if y is None and tags.target_tags.required:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:2868, in _check_feature_names(estimator, X, reset)
   2865 if not missing_names and not unexpected_names:
   2866     message += "Feature names must be in the same order as they were in fit.\n"
-> 2868 raise ValueError(message)

ValueError: The feature names should match those that were passed during fit.
Feature names unseen at fit time:
- outcome
new_X = new_patients.drop(columns="outcome")
risk = final.predict_proba(new_X)[:, 1]
report = new_X[["glucose", "bmi", "age"]].copy()
report["risk"] = risk.round(3)
report["actual"] = new_patients["outcome"]
report
glucose bmi age risk actual
763 101.0 32.9 63 0.086 0
764 122.0 36.8 27 0.305 0
765 121.0 26.2 30 0.273 0
766 126.0 30.1 47 0.223 1
767 93.0 30.4 23 0.006 0

Quick check: which patient would you call in first, and would the default 0.5 cut-off flag her?

top = report["risk"].idxmax()
print(report.loc[top])
print("flagged at 0.5:", report.loc[top, "risk"] >= 0.5)
glucose    122.000
bmi         36.800
age         27.000
risk         0.305
actual       0.000
Name: 764, dtype: float64
flagged at 0.5: False
test_pred = final.predict(X_test)
pd.crosstab(y_test, test_pred, rownames=["actual"],
            colnames=["predicted"])
predicted 0 1
actual
0 101 23
1 27 40

Mini-project: is the network worth it here?#

Compare logistic regression, SVM and a regularised network with 5-fold cross-validation on the training set.

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_validate
models = {
    "logistic": make_pipeline(SimpleImputer(strategy="median"),
                              StandardScaler(),
                              LogisticRegression(max_iter=1000)),
    "svm": svm,
    "network": build((8,), alpha=1),
}
scores = {}
for name, model in models.items():
    cv = cross_validate(model, X_train, y_train, cv=5,
                        scoring=["accuracy", "recall"])
    scores[name] = {"accuracy": cv["test_accuracy"].mean(),
                    "recall": cv["test_recall"].mean()}
results = pd.DataFrame(scores).T.round(3)
results
accuracy recall
logistic 0.778 0.57
svm 0.775 0.58
network 0.767 0.60
ax = results.plot(kind="bar", rot=0, figsize=(7, 4))
ax.set_title("Cross-validated scores on the Pima training set")
ax.set_xlabel("Model")
ax.set_ylabel("Average score over 5 folds")
ax.set_ylim(0.4, 0.9)
plt.show()
No description has been provided for this image
best = results["accuracy"].idxmax()
edge = results.loc["network"] - results.loc["logistic"]
print("most accurate:", best)
print(edge.round(3))
most accurate: logistic
accuracy   -0.011
recall      0.030
dtype: float64

Recap#

  • A neuron is a weighted sum plus a switch; layers stack neurons
  • Networks need complete, scaled inputs: use a pipeline
  • A trained network is just weights you can inspect and recompute
  • Bigger is not better: watch the train-test gap, use alpha
  • On small tables, simple models often match a network

Challenges#

  1. Try hidden_layer_sizes=(16, 8) with alpha=1. Does cross-validated recall improve?
  2. Swap activation to "tanh" and compare the loss curves.
  3. Rebuild the hand calculation for the second test patient.

What's next#

Lesson 20: Evaluating Classifiers - Accuracy, Precision, Recall, F1-Score Accuracy alone hides costly mistakes. Learn the metrics that don't.

Data Mining 20: Evaluating Classifiers- Accuracy, Precision, Recall, F1-Score#

In this lesson you'll be able to:

  • Explain why accuracy alone can hide a useless model
  • Read a confusion matrix and name its four cells
  • Calculate precision, recall and F1 by hand and with scikit-learn
  • Read a classification report, including macro and weighted averages
  • Move the decision threshold to hit a recall target, honestly Dataset: Pima Indians diabetes, 768 women (data/mining/pima_diabetes.csv)

Warm-up: the network from lesson 19#

Rebuild lesson 19's prepared data and its regularised network.

import numpy as np
import pandas as pd
df = pd.read_csv("pima_diabetes.csv")
fake_zero = ["glucose", "blood_pressure", "skin_thickness",
             "insulin", "bmi"]
df[fake_zero] = df[fake_zero].replace(0, np.nan)
X = df.drop(columns="outcome")
y = df["outcome"]
print(X.shape, round(y.mean(), 3))
(768, 8) 0.349
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)
print(len(X_train), len(X_test), y_test.sum())
576 192 67
from sklearn.pipeline import make_pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.neural_network import MLPClassifier
net = make_pipeline(SimpleImputer(strategy="median"),
                    StandardScaler(),
                    MLPClassifier(hidden_layer_sizes=(8,), alpha=1,
                                  max_iter=2000, random_state=42))
net.fit(X_train, y_train)
print("network accuracy:", round(net.score(X_test, y_test), 3))
network accuracy: 0.703

Why accuracy alone can mislead#

Problem: a high accuracy can come from a model that never finds a single case.

from sklearn.linear_model import LogisticRegression
model = make_pipeline(SimpleImputer(strategy="median"),
                      StandardScaler(),
                      LogisticRegression(max_iter=1000))
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
print(y_pred[:10])
print(y_test.head(10).to_numpy())
[0 0 0 0 1 0 0 0 1 0]
[1 0 0 1 1 0 0 0 0 0]
  • Accuracy = correct predictions / all predictions
from sklearn.metrics import accuracy_score
acc = accuracy_score(y_test, y_pred)
by_hand = (y_pred == y_test).mean()
print(round(acc, 3), round(by_hand, 3))
0.724 0.724

Quick check: what accuracy would a 'model' get by telling every woman she does NOT have diabetes?

from sklearn.dummy import DummyClassifier
lazy = DummyClassifier(strategy="most_frequent")
lazy.fit(X_train, y_train)
lazy_pred = lazy.predict(X_test)
print("lazy accuracy:", round(accuracy_score(y_test, lazy_pred), 3))
print("cases found:", lazy_pred.sum())
lazy accuracy: 0.651
cases found: 0
print(y_test.value_counts(normalize=True).round(3))
outcome
0    0.651
1    0.349
Name: proportion, dtype: float64

Count the four outcomes with a confusion matrix#

Problem: see exactly which kinds of mistakes the model makes.

  • True positive (TP): has diabetes, flagged
  • False negative (FN): has diabetes, missed
  • False positive (FP): healthy, flagged (false alarm)
  • True negative (TN): healthy, not flagged
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_test, y_pred)
print(cm)
[[105  20]
 [ 33  34]]
labels = pd.DataFrame(cm, index=["actual no", "actual yes"],
                      columns=["predicted no", "predicted yes"])
labels
predicted no predicted yes
actual no 105 20
actual yes 33 34

Common mistake: unpacking the matrix into four names without flattening it

tn, fp, fn, tp = confusion_matrix(y_test, y_pred)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[11], line 1
----> 1 tn, fp, fn, tp = confusion_matrix(y_test, y_pred)

ValueError: not enough values to unpack (expected 4, got 2)
tn, fp, fn, tp = confusion_matrix(y_test, y_pred).ravel()
print("TN", tn, "FP", fp, "FN", fn, "TP", tp)
print(round((tp + tn) / (tp + tn + fp + fn), 3))
TN 105 FP 20 FN 33 TP 34
0.724
import matplotlib.pyplot as plt
from sklearn.metrics import ConfusionMatrixDisplay
disp = ConfusionMatrixDisplay(cm, display_labels=["no", "yes"])
disp.plot(cmap="Blues")
plt.title("Logistic regression on 192 test patients")
plt.xlabel("Predicted diabetes")
plt.ylabel("Actual diabetes")
plt.show()
No description has been provided for this image

Your turn#

Build the confusion matrix for the lazy model's predictions, lazy_pred. Which cell holds every diabetic woman?

lazy_cm = confusion_matrix(y_test, lazy_pred)
pd.DataFrame(lazy_cm, index=labels.index, columns=labels.columns)
predicted no predicted yes
actual no 125 0
actual yes 67 0

Precision and recall: two different questions#

Problem: measure false alarms and missed cases separately.

  • Precision = TP / (TP + FP): when we say yes, how often are we right?
  • Recall = TP / (TP + FN): of the real cases, how many did we catch?
from sklearn.metrics import precision_score, recall_score
print(round(tp / (tp + fp), 3), round(precision_score(y_test, y_pred), 3))
print(round(tp / (tp + fn), 3), round(recall_score(y_test, y_pred), 3))
0.63 0.63
0.507 0.507

Quick check: for a diabetes screening test, which mistake is worse, a false alarm or a missed case?

print("missed cases:", fn, "of", tp + fn)
print("false alarms:", fp, "of", tp + fp, "flagged")
missed cases: 33 of 67
false alarms: 20 of 54 flagged
print(round(recall_score(y_test, y_pred, pos_label=0), 3))
print(round(tn / (tn + fp), 3))
0.84
0.84

Common mistake: passing probabilities where labels are expected

probs = model.predict_proba(X_test)[:, 1]
recall_score(y_test, probs)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[18], line 2
      1 probs = model.predict_proba(X_test)[:, 1]
----> 2 recall_score(y_test, probs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_param_validation.py:218, in validate_params.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    212 try:
    213     with config_context(
    214         skip_parameter_validation=(
    215             prefer_skip_nested_validation or global_skip_validation
    216         )
    217     ):
--> 218         return func(*args, **kwargs)
    219 except InvalidParameterError as e:
    220     # When the function is just a wrapper around an estimator, we allow
    221     # the function to delegate validation to the estimator, but we replace
    222     # the name of the estimator by the name of the function in the error
    223     # message to avoid confusion.
    224     msg = re.sub(
    225         r"parameter of \w+ must be",
    226         f"parameter of {func.__qualname__} must be",
    227         str(e),
    228     )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\metrics\_classification.py:2833, in recall_score(y_true, y_pred, labels, pos_label, average, sample_weight, zero_division)
   2662 @validate_params(
   2663     {
   2664         "y_true": ["array-like", "sparse matrix"],
   (...)
   2689     zero_division="warn",
   2690 ):
   2691     """Compute the recall.
   2692 
   2693     The recall is the ratio ``tp / (tp + fn)`` where ``tp`` is the number of
   (...)
   2831     array([1. , 1. , 0.5])
   2832     """
-> 2833     _, r, _, _ = precision_recall_fscore_support(
   2834         y_true,
   2835         y_pred,
   2836         labels=labels,
   2837         pos_label=pos_label,
   2838         average=average,
   2839         warn_for=("recall",),
   2840         sample_weight=sample_weight,
   2841         zero_division=zero_division,
   2842     )
   2843     return r

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_param_validation.py:191, in validate_params.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    189 global_skip_validation = get_config()["skip_parameter_validation"]
    190 if global_skip_validation:
--> 191     return func(*args, **kwargs)
    193 func_sig = signature(func)
    195 # Map *args/**kwargs to the function signature

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\metrics\_classification.py:2145, in precision_recall_fscore_support(y_true, y_pred, beta, labels, pos_label, average, warn_for, sample_weight, zero_division)
   2143 xp, _, device_ = get_namespace_and_device(y_pred)
   2144 y_true, sample_weight = move_to(y_true, sample_weight, xp=xp, device=device_)
-> 2145 labels = _check_set_wise_labels(y_true, y_pred, average, labels, pos_label)
   2147 # Calculate tp_sum, pred_sum, true_sum ###
   2148 samplewise = average == "samples"

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\metrics\_classification.py:1910, in _check_set_wise_labels(y_true, y_pred, average, labels, pos_label)
   1907     raise ValueError("average has to be one of " + str(average_options))
   1909 y_true, y_pred = attach_unique(y_true, y_pred)
-> 1910 y_type, present_labels, y_true, y_pred, _ = _check_targets(y_true, y_pred)
   1911 if average == "binary":
   1912     if y_type == "binary":

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\metrics\_classification.py:134, in _check_targets(y_true, y_pred, sample_weight)
    131     y_type = {"multiclass"}
    133 if len(y_type) > 1:
--> 134     raise ValueError(
    135         "Classification metrics can't handle a mix of {0} and {1} targets".format(
    136             type_true, type_pred
    137         )
    138     )
    140 # We can't have more than one value in y_type => The set is no more needed
    141 y_type = y_type.pop()

ValueError: Classification metrics can't handle a mix of binary and continuous targets
labels_05 = (probs >= 0.5).astype(int)
print(round(recall_score(y_test, labels_05), 3))
print((labels_05 == y_pred).all())
0.507
True

One number for both: F1 and the classification report#

Problem: compare models with one score that punishes a weak precision or recall.

  • F1 = 2 x precision x recall / (precision + recall)
  • A harmonic mean is dragged down by the weaker of the two
from sklearn.metrics import f1_score
p = precision_score(y_test, y_pred)
r = recall_score(y_test, y_pred)
print("plain average:", round((p + r) / 2, 3))
print("F1 by hand:   ", round(2 * p * r / (p + r), 3))
print("f1_score:     ", round(f1_score(y_test, y_pred), 3))
plain average: 0.569
F1 by hand:    0.562
f1_score:      0.562
print("lazy F1:", f1_score(y_test, lazy_pred, zero_division=0))
lazy F1: 0.0
from sklearn.metrics import classification_report
print(classification_report(y_test, y_pred,
                            target_names=["no", "yes"]))
              precision    recall  f1-score   support

          no       0.76      0.84      0.80       125
         yes       0.63      0.51      0.56        67

    accuracy                           0.72       192
   macro avg       0.70      0.67      0.68       192
weighted avg       0.72      0.72      0.72       192

  • support: how many real cases of each class
  • macro avg: plain average of the two classes
  • weighted avg: average weighted by support

Your turn#

Print a classification report for the warm-up network net. Does it beat logistic regression on recall?

net_pred = net.predict(X_test)
print(classification_report(y_test, net_pred,
                            target_names=["no", "yes"]))
              precision    recall  f1-score   support

          no       0.75      0.81      0.78       125
         yes       0.59      0.51      0.54        67

    accuracy                           0.70       192
   macro avg       0.67      0.66      0.66       192
weighted avg       0.70      0.70      0.70       192

Check the scores are stable with cross-validation#

Problem: one test set of 192 women is a small sample. How much could the scores wobble?

from sklearn.model_selection import cross_validate
metrics = ["accuracy", "precision", "recall", "f1"]
cv = cross_validate(model, X_train, y_train, cv=5,
                    scoring=metrics)
folds = pd.DataFrame({m: cv[f"test_{m}"] for m in metrics})
folds.round(3)
accuracy precision recall f1
0 0.750 0.658 0.610 0.633
1 0.757 0.700 0.525 0.600
2 0.791 0.722 0.650 0.684
3 0.800 0.815 0.550 0.657
4 0.800 0.774 0.600 0.676
folds.agg(["mean", "std"]).round(3)
accuracy precision recall f1
mean 0.780 0.734 0.587 0.650
std 0.024 0.062 0.050 0.034

Quick check: which metric wobbles most from fold to fold, and why might that be?

print(folds.std().idxmax(), round(folds.std().max(), 3))
print(int(y_train.sum() / 5))
precision 0.062
40

Move the threshold to trade precision for recall#

Problem: 0.5 is just a default. Pick the cut-off that suits the real cost of mistakes.

from sklearn.model_selection import cross_val_predict
train_probs = cross_val_predict(model, X_train, y_train, cv=5,
                                method="predict_proba")[:, 1]
print(train_probs[:5].round(3))
[0.357 0.285 0.168 0.103 0.251]
rows = []
for t in [0.2, 0.3, 0.4, 0.5, 0.6, 0.7]:
    flag = (train_probs >= t).astype(int)
    rows.append({"threshold": t,
                 "precision": precision_score(y_train, flag),
                 "recall": recall_score(y_train, flag),
                 "flagged": flag.sum()})
trade = pd.DataFrame(rows).set_index("threshold").round(3)
trade
precision recall flagged
threshold
0.2 0.516 0.876 341
0.3 0.598 0.776 261
0.4 0.670 0.667 200
0.5 0.728 0.587 162
0.6 0.786 0.512 131
0.7 0.830 0.438 106
ax = trade[["precision", "recall"]].plot(marker="o", figsize=(7, 4))
ax.set_title("Lower threshold: more recall, less precision")
ax.set_xlabel("Decision threshold")
ax.set_ylabel("Score on training folds")
plt.show()
No description has been provided for this image

Quick check: at what threshold do precision and recall roughly meet?

gap = (trade["precision"] - trade["recall"]).abs()
print(gap.idxmin(), gap.min().round(3))
0.4 0.003

Your turn#

Add an f1 column to trade and find the threshold with the best F1.

p, r = trade["precision"], trade["recall"]
trade["f1"] = (2 * p * r / (p + r)).round(3)
print(trade["f1"])
print("best F1 at", trade["f1"].idxmax())
threshold
0.2    0.649
0.3    0.675
0.4    0.668
0.5    0.650
0.6    0.620
0.7    0.573
Name: f1, dtype: float64
best F1 at 0.3

Mini-project: a screening rule that catches 80% of cases#

A clinic wants to catch at least 80% of diabetic women. What threshold, and what does it cost in follow-ups?

cutoffs = np.arange(0.05, 0.95, 0.01).round(2)
ok = [t for t in cutoffs
      if recall_score(y_train, train_probs >= t) >= 0.8]
best_t = max(ok)
print("chosen threshold:", best_t)
chosen threshold: 0.27
screen = (probs >= best_t).astype(int)
tn2, fp2, fn2, tp2 = confusion_matrix(y_test, screen).ravel()
print("caught", tp2, "of", tp2 + fn2, "- missed", fn2)
print("follow-ups:", tp2 + fp2, "- false alarms:", fp2)
print("recall:", round(recall_score(y_test, screen), 3))
print("precision:", round(precision_score(y_test, screen), 3))
caught 54 of 67 - missed 13
follow-ups: 94 - false alarms: 40
recall: 0.806
precision: 0.574
summary = pd.DataFrame({
    "default 0.5": [tp, fn, tp + fp],
    f"screening {best_t}": [tp2, fn2, tp2 + fp2]},
    index=["caught", "missed", "follow-ups"])
summary
default 0.5 screening 0.27
caught 34 54
missed 33 13
follow-ups 54 94
extra_caught = tp2 - tp
extra_tests = (tp2 + fp2) - (tp + fp)
print(f"{extra_caught} more cases caught for {extra_tests} more "
      f"follow-ups: {extra_tests / extra_caught:.1f} tests each")
20 more cases caught for 40 more follow-ups: 2.0 tests each

Recap#

  • Always compare accuracy with a lazy baseline
  • The confusion matrix splits mistakes into false alarms and misses
  • Precision punishes false alarms; recall punishes missed cases
  • F1 is high only when both are high
  • Choose thresholds on training folds, then test once

Challenges#

  1. Repeat the 80% screening rule with the network net. Which model needs fewer follow-ups?
  2. Use fbeta_score with beta=2 to weight recall twice as much as precision.
  3. Write a function that prints accuracy, precision, recall and F1 for any model.

What's next#

Lesson 21: ROC Curves, AUC, and Confusion Matrix Analysis See every threshold at once, and compare models with one number.

Data Mining 21: ROC Curves, AUC, and Confusion Matrix Analysis#

In this lesson you'll be able to:

  • Turn a confusion matrix into true and false positive rates
  • Draw an ROC curve and read any threshold off it
  • Explain AUC in plain words, and check it by hand
  • Compare models fairly with ROC curves and cross-validated AUC
  • Break a model's performance down by customer segment Dataset: IBM Telco customer churn, 7,043 customers (data/mining/telco_churn.csv)

Warm-up: precision and recall from lesson 20#

A fast churn model on three numeric columns, judged the lesson 20 way.

import numpy as np
import pandas as pd
df = pd.read_csv("telco_churn.csv")
print(df.shape)
print(df["Churn"].value_counts(normalize=True).round(3))
(7043, 21)
Churn
No     0.735
Yes    0.265
Name: proportion, dtype: float64
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score
few = df[["tenure", "MonthlyCharges", "SeniorCitizen"]]
churned = (df["Churn"] == "Yes").astype(int)
a, b, c, d = train_test_split(few, churned, random_state=42)
pred = make_pipeline(StandardScaler(),
                     LogisticRegression()).fit(a, c).predict(b)
print(round(precision_score(d, pred), 3),
      round(recall_score(d, pred), 3))
0.696 0.455

Clean the churn data before modelling#

Problem: one numeric column is secretly text, and dropna can't see the gaps.

print(df.isna().sum().sum())
print(df["TotalCharges"].dtype)
print(df["TotalCharges"].head(3).to_list())
0
str
['29.85', '1889.5', '108.15']

Common mistake: forcing text into numbers with astype(float)

df["TotalCharges"].astype(float)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[5], line 1
----> 1 df["TotalCharges"].astype(float)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\generic.py:6529, in NDFrame.astype(self, dtype, copy, errors)
   6525     results = [ser.astype(dtype, errors=errors) for _, ser in self.items()]
   6527 else:
   6528     # else, only a single dtype is given
-> 6529     new_data = self._mgr.astype(dtype=dtype, errors=errors)
   6530     res = self._constructor_from_mgr(new_data, axes=new_data.axes)
   6531     return res.__finalize__(self, method="astype")

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\managers.py:627, in BaseBlockManager.astype(self, dtype, errors)
    626 def astype(self, dtype, errors: str = "raise") -> Self:
--> 627     return self.apply("astype", dtype=dtype, errors=errors)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\managers.py:445, in BaseBlockManager.apply(self, f, align_keys, **kwargs)
    443         applied = b.apply(f, **kwargs)
    444     else:
--> 445         applied = getattr(b, f)(**kwargs)
    446     result_blocks = extend_blocks(applied, result_blocks)
    448 out = type(self).from_blocks(result_blocks, [ax.view() for ax in self.axes])

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\blocks.py:607, in Block.astype(self, dtype, errors, squeeze)
    604         raise ValueError("Can not squeeze with more than one column.")
    605     values = values[0, :]  # type: ignore[call-overload]
--> 607 new_values = astype_array_safe(values, dtype, errors=errors)
    609 new_values = maybe_coerce_values(new_values)
    611 refs = None

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\dtypes\astype.py:240, in astype_array_safe(values, dtype, copy, errors)
    237     dtype = dtype.numpy_dtype
    239 try:
--> 240     new_values = astype_array(values, dtype, copy=copy)
    241 except (ValueError, TypeError):
    242     # e.g. _astype_nansafe can fail on object-dtype of strings
    243     #  trying to convert to float
    244     if errors == "ignore":

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\dtypes\astype.py:182, in astype_array(values, dtype, copy)
    178     return values
    180 if not isinstance(values, np.ndarray):
    181     # i.e. ExtensionArray
--> 182     values = values.astype(dtype, copy=copy)
    184 else:
    185     values = _astype_nansafe(values, dtype, copy=copy)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\string_arrow.py:336, in ArrowStringArray.astype(self, dtype, copy)
    334     return dtype.__from_arrow__(data)
    335 elif isinstance(dtype, np.dtype) and np.issubdtype(dtype, np.floating):
--> 336     return self.to_numpy(dtype=dtype, na_value=np.nan)
    338 return super().astype(dtype, copy=copy)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\arrow\array.py:1803, in ArrowExtensionArray.to_numpy(self, dtype, copy, na_value)
   1801 result = data._pa_array.to_numpy()
   1802 if dtype is not None:
-> 1803     result = result.astype(dtype, copy=False)
   1804 if copy:
   1805     result = result.copy()

ValueError: could not convert string to float: ' '
total = pd.to_numeric(df["TotalCharges"], errors="coerce")
print(total.isna().sum())
print(df.loc[total.isna(), "tenure"].unique())
11
[0]
df["TotalCharges"] = total
df = df.dropna(subset=["TotalCharges"])
print(df.shape)
(7032, 21)
y = (df["Churn"] == "Yes").astype(int)
raw = df.drop(columns=["customerID", "Churn"])
X = pd.get_dummies(raw, drop_first=True, dtype=int)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)
print(X.shape, len(X_test), y_test.sum())
(7032, 30) 1758 467
model = make_pipeline(StandardScaler(),
                      LogisticRegression(max_iter=1000))
model.fit(X_train, y_train)
probs = model.predict_proba(X_test)[:, 1]
print(probs[:6].round(3))
[0.603 0.009 0.508 0.789 0.11  0.2  ]

Confusion matrix analysis: rates, not just counts#

Problem: compare thresholds fairly when churners and stayers come in very different numbers.

  • True positive rate (TPR) = TP / (TP + FN), the same as recall
  • False positive rate (FPR) = FP / (FP + TN)
from sklearn.metrics import confusion_matrix
tn, fp, fn, tp = confusion_matrix(y_test, probs >= 0.5).ravel()
print("TN", tn, "FP", fp, "FN", fn, "TP", tp)
print("TPR", round(tp / (tp + fn), 3))
print("FPR", round(fp / (fp + tn), 3))
TN 1150 FP 141 FN 197 TP 270
TPR 0.578
FPR 0.109
import matplotlib.pyplot as plt
from sklearn.metrics import ConfusionMatrixDisplay
ConfusionMatrixDisplay.from_predictions(
    y_test, probs >= 0.5, normalize="true",
    display_labels=["stayed", "churned"], cmap="Blues")
plt.title("Each row sums to 1: rates at threshold 0.5")
plt.xlabel("Predicted")
plt.ylabel("Actual")
plt.show()
No description has been provided for this image
def rates(t):
    tn, fp, fn, tp = confusion_matrix(y_test, probs >= t).ravel()
    return {"threshold": t, "TP": tp, "FP": fp,
            "TPR": tp / (tp + fn), "FPR": fp / (fp + tn)}

grid = pd.DataFrame([rates(t) for t in [0.2, 0.3, 0.5, 0.7]])
grid.round(3)
threshold TP FP TPR FPR
0 0.2 399 472 0.854 0.366
1 0.3 359 331 0.769 0.256
2 0.5 270 141 0.578 0.109
3 0.7 85 33 0.182 0.026

Quick check: as the threshold drops from 0.7 to 0.2, what happens to both rates?

print(grid["TPR"].is_monotonic_decreasing)
print(grid["FPR"].is_monotonic_decreasing)
True
True

Your turn#

Add a precision column to grid using its TP and FP columns. Which cut-off is most precise?

grid["precision"] = grid["TP"] / (grid["TP"] + grid["FP"])
grid[["threshold", "TPR", "FPR", "precision"]].round(3)
threshold TPR FPR precision
0 0.2 0.854 0.366 0.458
1 0.3 0.769 0.256 0.520
2 0.5 0.578 0.109 0.657
3 0.7 0.182 0.026 0.720

Draw the ROC curve#

Problem: see every possible threshold at once, not four of them.

from sklearn.metrics import roc_curve
fpr, tpr, cuts = roc_curve(y_test, probs)
roc = pd.DataFrame({"cut": cuts, "FPR": fpr, "TPR": tpr})
print(len(roc) > 100)
roc.head().round(3)
True
cut FPR TPR
0 inf 0.000 0.000
1 0.854 0.000 0.002
2 0.852 0.000 0.004
3 0.847 0.001 0.004
4 0.832 0.001 0.006
plt.figure(figsize=(6, 5))
plt.plot(fpr, tpr, label="logistic regression")
plt.plot([0, 1], [0, 1], "k--", label="random guessing")
plt.scatter(fp / (fp + tn), tp / (tp + fn), color="red",
            zorder=3, label="threshold 0.5")
plt.title("ROC curve: churn model on 1,758 test customers")
plt.xlabel("False positive rate (loyal customers flagged)")
plt.ylabel("True positive rate (churners caught)")
plt.legend()
plt.show()
No description has been provided for this image
  • Bottom-left corner: threshold so high nobody is flagged
  • Top-right corner: threshold so low everybody is flagged
  • The closer the curve hugs the top-left, the better

Common mistake: passing the original Yes/No text labels

churn_text = df.loc[y_test.index, "Churn"]
roc_curve(churn_text, probs)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[17], line 2
      1 churn_text = df.loc[y_test.index, "Churn"]
----> 2 roc_curve(churn_text, probs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_param_validation.py:218, in validate_params.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    212 try:
    213     with config_context(
    214         skip_parameter_validation=(
    215             prefer_skip_nested_validation or global_skip_validation
    216         )
    217     ):
--> 218         return func(*args, **kwargs)
    219 except InvalidParameterError as e:
    220     # When the function is just a wrapper around an estimator, we allow
    221     # the function to delegate validation to the estimator, but we replace
    222     # the name of the estimator by the name of the function in the error
    223     # message to avoid confusion.
    224     msg = re.sub(
    225         r"parameter of \w+ must be",
    226         f"parameter of {func.__qualname__} must be",
    227         str(e),
    228     )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\metrics\_ranking.py:1319, in roc_curve(y_true, y_score, pos_label, sample_weight, drop_intermediate)
   1221 """Compute Receiver operating characteristic (ROC).
   1222 
   1223 Note: Support beyond :term:`binary` classification tasks, via one-vs-rest or
   (...)
   1315 array([ inf, 0.8 , 0.4 , 0.35, 0.1 ])
   1316 """
   1317 xp, _, device = get_namespace_and_device(y_true, y_score)
-> 1319 _, fps, _, tps, thresholds = confusion_matrix_at_thresholds(
   1320     y_true, y_score, pos_label=pos_label, sample_weight=sample_weight
   1321 )
   1323 # Attempt to drop thresholds corresponding to points in between and
   1324 # collinear with other points. These are always suboptimal and do not
   1325 # appear on a plotted ROC curve (and thus do not affect the AUC).
   (...)
   1330 # kept, but does not drop more complicated cases like fps = [1, 3, 7],
   1331 # tps = [1, 2, 4]; there is no harm in keeping too many thresholds.
   1332 if drop_intermediate and fps.shape[0] > 2:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:80, in _deprecate_positional_args.<locals>._inner_deprecate_positional_args.<locals>.inner_f(*args, **kwargs)
     78 extra_args = len(args) - len(all_args)
     79 if extra_args <= 0:
---> 80     return f(*args, **kwargs)
     82 # extra_args > 0
     83 args_msg = [
     84     f"{name}={arg}"
     85     for name, arg in zip(kwonly_args[:extra_args], args[-extra_args:])
     86 ]

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_param_validation.py:191, in validate_params.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    189 global_skip_validation = get_config()["skip_parameter_validation"]
    190 if global_skip_validation:
--> 191     return func(*args, **kwargs)
    193 func_sig = signature(func)
    195 # Map *args/**kwargs to the function signature

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\metrics\_ranking.py:1012, in confusion_matrix_at_thresholds(y_true, y_score, pos_label, sample_weight)
   1009     raise ValueError("{0} format is not supported".format(y_type))
   1011 xp, _, device = get_namespace_and_device(y_score)
-> 1012 pos_label = _check_pos_label_consistency(pos_label, y_true)
   1013 xp_y_true, _ = get_namespace(y_true)
   1014 # Make `y_true` a boolean vector. Use `asarray` as `y_true` could be a list

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:2726, in _check_pos_label_consistency(pos_label, y_true)
   2724         classes = move_to(classes, xp=np, device="cpu")
   2725         classes_repr = ", ".join([repr(c) for c in classes.tolist()])
-> 2726         raise ValueError(
   2727             f"y_true takes value in {{{classes_repr}}} and pos_label is not "
   2728             "specified: either make y_true take value in {0, 1} or "
   2729             "{-1, 1} or pass pos_label explicitly."
   2730         )
   2731     pos_label = 1
   2733 return pos_label

ValueError: y_true takes value in {'No', 'Yes'} and pos_label is not specified: either make y_true take value in {0, 1} or {-1, 1} or pass pos_label explicitly.
fpr2, tpr2, _ = roc_curve(churn_text, probs, pos_label="Yes")
print(np.allclose(fpr, fpr2), np.allclose(tpr, tpr2))
True True

Your turn#

Find the first point on the curve that catches at least 80% of churners. What false positive rate does it cost?

hit = roc[roc["TPR"] >= 0.8].iloc[0]
hit.round(3)
cut    0.262
FPR    0.289
TPR    0.801
Name: 329, dtype: float64

AUC: one number for ranking quality#

Problem: summarise the whole curve in one number that doesn't depend on any threshold.

from sklearn.metrics import roc_auc_score, auc
print(round(roc_auc_score(y_test, probs), 3))
print(round(auc(fpr, tpr), 3))
0.84
0.84

AUC in plain words: pick one churner and one loyal customer at random. AUC is the chance the churner gets the higher score.

plt.figure(figsize=(7, 4))
plt.hist(probs[y_test == 0], bins=30, alpha=0.6, label="stayed")
plt.hist(probs[y_test == 1], bins=30, alpha=0.6, label="churned")
plt.title("Churners mostly score higher, but the groups overlap")
plt.xlabel("Predicted churn probability")
plt.ylabel("Number of customers")
plt.legend()
plt.show()
No description has been provided for this image
churn_scores = probs[y_test.to_numpy() == 1]
stay_scores = probs[y_test.to_numpy() == 0]
wins = churn_scores[:, None] > stay_scores[None, :]
print(wins.size)
print(round(wins.mean(), 3))
602897
0.84
for col in ["MonthlyCharges", "tenure"]:
    print(col, round(roc_auc_score(y_test, X_test[col]), 3))
MonthlyCharges 0.613
tenure 0.246

Quick check: tenure scores below 0.5. Is it useless, or something else?

print(round(roc_auc_score(y_test, -X_test["tenure"]), 3))
0.754

Common mistake: computing AUC from yes/no predictions

hard = model.predict(X_test)
print("AUC from labels:", round(roc_auc_score(y_test, hard), 3))
print("AUC from scores:", round(roc_auc_score(y_test, probs), 3))
AUC from labels: 0.734
AUC from scores: 0.84

Compare models with ROC curves#

Problem: decide which of two models ranks customers better, without cheating on the test set.

from sklearn.tree import DecisionTreeClassifier
tree = DecisionTreeClassifier(max_depth=5, random_state=42)
tree.fit(X_train, y_train)
tree_probs = tree.predict_proba(X_test)[:, 1]
print(len(np.unique(tree_probs)))
26
t_fpr, t_tpr, _ = roc_curve(y_test, tree_probs)
plt.figure(figsize=(6, 5))
plt.plot(fpr, tpr, label="logistic regression")
plt.plot(t_fpr, t_tpr, label="decision tree")
plt.plot([0, 1], [0, 1], "k--", label="random guessing")
plt.title("Two churn models, one ROC chart")
plt.xlabel("False positive rate")
plt.ylabel("True positive rate")
plt.legend()
plt.show()
No description has been provided for this image
from sklearn.model_selection import cross_val_score
for name, m in [("logistic", model), ("tree", tree)]:
    s = cross_val_score(m, X_train, y_train, cv=5, scoring="roc_auc")
    print(name, s.round(3), "mean", round(s.mean(), 3))
logistic [0.851 0.855 0.854 0.825 0.845] mean 0.846
tree [0.829 0.838 0.844 0.809 0.823] mean 0.828

Quick check: does logistic regression win in every fold, or only on average?

lr_cv = cross_val_score(model, X_train, y_train, cv=5,
                        scoring="roc_auc")
tr_cv = cross_val_score(tree, X_train, y_train, cv=5,
                        scoring="roc_auc")
print((lr_cv > tr_cv).sum(), "of 5 folds")
5 of 5 folds
print("logistic test AUC:", round(roc_auc_score(y_test, probs), 3))
print("tree test AUC:    ", round(roc_auc_score(y_test, tree_probs), 3))
logistic test AUC: 0.84
tree test AUC:     0.83

Your turn#

The retention team accepts a 10% false positive rate. Which model catches more churners there? Hint: np.interp.

lr_at_10 = np.interp(0.10, fpr, tpr)
tree_at_10 = np.interp(0.10, t_fpr, t_tpr)
print(round(lr_at_10, 3), round(tree_at_10, 3))
0.522 0.477

Mini-project: where does the churn model fail?#

Analyse AUC and the confusion matrix separately for each contract type.

contract = df.loc[y_test.index, "Contract"]
rows = []
for kind in contract.unique():
    m = (contract == kind).to_numpy()
    tn, fp, fn, tp = confusion_matrix(y_test[m], probs[m] >= 0.5,
                                      labels=[0, 1]).ravel()
    rows.append({"contract": kind, "customers": m.sum(),
                 "churn_rate": y_test[m].mean(), "caught": tp,
                 "missed": fn,
                 "auc": roc_auc_score(y_test[m], probs[m])})
seg = pd.DataFrame(rows).set_index("contract").round(3)
seg
customers churn_rate caught missed auc
contract
Month-to-month 978 0.423 270 144 0.747
Two year 408 0.025 0 10 0.735
One year 372 0.116 0 43 0.685
ax = seg["auc"].plot(kind="bar", rot=0, figsize=(7, 4))
ax.axhline(roc_auc_score(y_test, probs), color="red", ls="--")
ax.set_title("AUC inside each contract type vs overall")
ax.set_xlabel("Contract type")
ax.set_ylabel("AUC")
ax.set_ylim(0.5, 0.9)
plt.show()
No description has been provided for this image
scores = pd.DataFrame({"contract": contract, "churned": y_test,
                       "score": probs})
avg = scores.groupby(["contract", "churned"])["score"].mean()
avg.unstack().round(3)
churned 0 1
contract
Month-to-month 0.357 0.539
One year 0.111 0.177
Two year 0.025 0.052
long = seg.drop(index="Month-to-month")
print("long-contract churners:", long["caught"].sum()
      + long["missed"].sum())
print("caught at 0.5:", long["caught"].sum())
long-contract churners: 53
caught at 0.5: 0

Recap#

  • TPR is recall; FPR is the share of negatives wrongly flagged
  • The ROC curve plots both rates for every threshold
  • AUC = chance a random positive outranks a random negative
  • AUC needs scores, not labels; below 0.5 means flipped
  • Compare models with cross-validated AUC, then check segments

Challenges#

  1. Build a Pipeline of StandardScaler and LogisticRegression on only tenure and MonthlyCharges. Compare its test AUC.
  2. Draw ROC curves for the tree at max_depth 3 and 8. Which bends closer to the top-left?
  3. Use RocCurveDisplay.from_estimator to redraw the curve in one line.

What's next#

Lesson 22: Hands-on Classification Project- Predicting Customer Churn Put every classifier and metric together on one business problem.

Data Mining 22: Hands-on Classification Project- Predicting Customer Churn#

In this lesson you'll be able to:

  • Clean and encode a real customer table for modelling
  • Train five classic classifiers on one honest train/test split
  • Compare them on precision, recall, F1 and AUC, not just accuracy
  • Turn the winner into a ranked call list for a retention team Dataset: IBM Telco customer churn, 7,043 customers (data/mining/telco_churn.csv)

Warm-up: the confusion matrix from lesson 21#

Lesson 21 judged models with ROC curves, AUC and the confusion matrix.

import pandas as pd
df = pd.read_csv("telco_churn.csv")
print(df.shape)
df["Churn"].value_counts(normalize=True).round(3)
(7043, 21)
Churn
No     0.735
Yes    0.265
Name: proportion, dtype: float64
from sklearn.metrics import confusion_matrix
stay = pd.Series("No", index=df.index)
print((stay == df["Churn"]).mean().round(3))
confusion_matrix(df["Churn"], stay)
0.735
array([[5174,    0],
       [1869,    0]])

Get to know the churners#

Problem: find who leaves, and fix the messy columns, before any model.

print(df.dtypes.value_counts())
df[["tenure", "MonthlyCharges", "TotalCharges", "Contract"]].head(3)
str        18
int64       2
float64     1
Name: count, dtype: int64
tenure MonthlyCharges TotalCharges Contract
0 1 29.85 29.85 Month-to-month
1 34 56.95 1889.5 One year
2 2 53.85 108.15 Month-to-month

Common mistake: doing maths on a column that only looks numeric

df["TotalCharges"].mean()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[5], line 1
----> 1 df["TotalCharges"].mean()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\util\_decorators.py:336, in deprecate_nonkeyword_arguments.<locals>.decorate.<locals>.wrapper(*args, **kwargs)
    330 if len(args) > num_allow_args:
    331     warnings.warn(
    332         msg.format(arguments=_format_argument_list(allow_args)),
    333         klass,
    334         stacklevel=find_stack_level(),
    335     )
--> 336 return func(*args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\series.py:8113, in Series.mean(self, axis, skipna, numeric_only, **kwargs)
   8063 @deprecate_nonkeyword_arguments(Pandas4Warning, allowed_args=["self"], name="mean")
   8064 def mean(
   8065     self,
   (...)
   8069     **kwargs,
   8070 ) -> Any:
   8071     """
   8072     Return the mean of the values over the requested axis.
   8073 
   (...)
   8111     2.0
   8112     """
-> 8113     return NDFrame.mean(
   8114         self, axis=axis, skipna=skipna, numeric_only=numeric_only, **kwargs
   8115     )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\generic.py:11819, in NDFrame.mean(self, axis, skipna, numeric_only, **kwargs)
  11811 def mean(
  11812     self,
  11813     *,
   (...)
  11817     **kwargs,
  11818 ) -> Series | float:
> 11819     return self._stat_function(
  11820         "mean", nanops.nanmean, axis, skipna, numeric_only, **kwargs
  11821     )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\generic.py:11773, in NDFrame._stat_function(self, name, func, axis, skipna, numeric_only, **kwargs)
  11769 nv.validate_func(name, (), kwargs)
  11771 validate_bool_kwarg(skipna, "skipna", none_allowed=False)
> 11773 return self._reduce(
  11774     func, name=name, axis=axis, skipna=skipna, numeric_only=numeric_only
  11775 )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\series.py:7480, in Series._reduce(self, op, name, axis, skipna, numeric_only, filter_type, **kwds)
   7476     self._get_axis_number(axis)
   7478 if isinstance(delegate, ExtensionArray):
   7479     # dispatch to ExtensionArray interface
-> 7480     result = delegate._reduce(name, skipna=skipna, **kwds)
   7482 else:
   7483     # dispatch to numpy arrays
   7484     if numeric_only and self.dtype.kind not in "iufcb":
   7485         # i.e. not is_numeric_dtype(self.dtype)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\string_arrow.py:564, in ArrowStringArray._reduce(self, name, skipna, keepdims, **kwargs)
    562     result = self._reduce_calc(name, skipna=skipna, keepdims=keepdims, **kwargs)
    563 else:
--> 564     raise TypeError(f"Cannot perform reduction '{name}' with string dtype")
    566 if name in ("argmin", "argmax") and isinstance(result, pa.Array):
    567     return self._convert_int_result(result)

TypeError: Cannot perform reduction 'mean' with string dtype
total = pd.to_numeric(df["TotalCharges"], errors="coerce")
print(total.isna().sum())
11
blank = total.isna()
df.loc[blank, ["tenure", "MonthlyCharges", "TotalCharges"]].head()
tenure MonthlyCharges TotalCharges
488 0 52.55
753 0 20.25
936 0 80.85
1082 0 25.75
1340 0 56.05
df["TotalCharges"] = total.fillna(0)
print(round(df["TotalCharges"].mean(), 2))
2279.73
churned = df["Churn"].eq("Yes")
churned.groupby(df["Contract"]).mean().round(3)
Contract
Month-to-month    0.427
One year          0.113
Two year          0.028
Name: Churn, dtype: float64

Quick check: do brand-new customers churn more or less than loyal ones?

edges = [0, 12, 24, 48, 72]
labels = ["0-12", "13-24", "25-48", "49-72"]
bands = pd.cut(df["tenure"], bins=edges, labels=labels,
              include_lowest=True)
churned.groupby(bands, observed=True).mean().round(3)
tenure
0-12     0.474
13-24    0.287
25-48    0.204
49-72    0.095
Name: Churn, dtype: float64
import matplotlib.pyplot as plt
rate = churned.groupby(bands, observed=True).mean()
ax = rate.plot(kind="bar", rot=0, figsize=(7, 4), color="tab:red")
ax.set_title("Churn rate falls the longer customers stay")
ax.set_xlabel("Tenure (months)")
ax.set_ylabel("Share who churned")
plt.show()
No description has been provided for this image

Turn the table into model-ready numbers#

Problem: models need numbers only, and a test set they never see.

y = churned.astype(int)
X = df.drop(columns=["customerID", "Churn"])
print(X.shape)
(7043, 19)

Common mistake: handing text columns straight to a model

from sklearn.tree import DecisionTreeClassifier
DecisionTreeClassifier(random_state=42).fit(X, y)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_18956\871816586.py in ?()
      1 from sklearn.tree import DecisionTreeClassifier
----> 2 DecisionTreeClassifier(random_state=42).fit(X, y)

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py in ?(estimator, *args, **kwargs)
   1399                     )
   1400                 ),
   1401                 callback_management_context(estimator),
   1402             ):
-> 1403                 return fit_method(estimator, *args, **kwargs)

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\tree\_classes.py in ?(self, X, y, sample_weight, check_input)
   1011         self : DecisionTreeClassifier
   1012             Fitted estimator.
   1013         """
   1014 
-> 1015         super()._fit(
   1016             X,
   1017             y,
   1018             sample_weight=sample_weight,

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\tree\_classes.py in ?(self, X, y, sample_weight, check_input, missing_values_in_feature_mask)
    243             check_X_params = dict(
    244                 dtype=np.float32, accept_sparse="csc", ensure_all_finite=False
    245             )
    246             check_y_params = dict(ensure_2d=False, dtype=None)
--> 247             X, y = validate_data(
    248                 self, X, y, validate_separately=(check_X_params, check_y_params)
    249             )
    250 

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py in ?(_estimator, X, y, reset, validate_separately, skip_check_array, **check_params)
   3049             # :(
   3050             check_X_params, check_y_params = validate_separately
   3051             if "estimator" not in check_X_params:
   3052                 check_X_params = {**default_check_params, **check_X_params}
-> 3053             X = check_array(X, input_name="X", **check_X_params)
   3054             if "estimator" not in check_y_params:
   3055                 check_y_params = {**default_check_params, **check_y_params}
   3056             y = check_array(y, input_name="y", **check_y_params)

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py in ?(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, ensure_all_finite, ensure_non_negative, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)
   1035                         )
   1036                     array = xp.astype(array, dtype, copy=False)
   1037                 else:
   1038                     array = _asarray_with_order(array, order=order, dtype=dtype, xp=xp)
-> 1039             except ComplexWarning as complex_warning:
   1040                 raise ValueError(
   1041                     "Complex data not supported\n{}\n".format(array)
   1042                 ) from complex_warning

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_array_api.py in ?(array, dtype, order, copy, xp, device)
    979         # Use NumPy API to support order
    980         if copy is True:
    981             array = numpy.array(array, order=order, dtype=dtype)
    982         else:
--> 983             array = numpy.asarray(array, order=order, dtype=dtype)
    984 
    985         # At this point array is a NumPy ndarray. We convert it to an array
    986         # container that is consistent with the input's namespace.

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\generic.py in ?(self, dtype, copy)
   2023             )
   2024         values = self._values
   2025         if copy is None:
   2026             # Note: branch avoids `copy=None` for NumPy 1.x support
-> 2027             arr = np.asarray(values, dtype=dtype)
   2028         else:
   2029             arr = np.array(values, dtype=dtype, copy=copy)
   2030 

ValueError: could not convert string to float: 'Female'
X = pd.get_dummies(X, drop_first=True, dtype=int)
print(X.shape)
X.columns[-6:].tolist()
(7043, 30)
['Contract_One year',
 'Contract_Two year',
 'PaperlessBilling_Yes',
 'PaymentMethod_Credit card (automatic)',
 'PaymentMethod_Electronic check',
 'PaymentMethod_Mailed check']
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42)
print(len(X_train), len(X_test))
print(round(y_train.mean(), 3), round(y_test.mean(), 3))
5282 1761
0.265 0.265

Quick check: why fit the scaler on the training rows only?

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler().fit(X_train)
X_train_s = scaler.transform(X_train)
X_test_s = scaler.transform(X_test)
print(X_train_s.std().round(3), X_test_s.std().round(3))
1.0 0.994

Line up five classic classifiers#

Problem: which family of model suits this data? Train them all on one split.

  • Logistic regression: a weighted score turned into a probability
  • Decision tree: a flowchart of yes or no questions
  • kNN: ask the most similar past customers
  • Naive Bayes: combine each column's evidence as if independent
  • SVM: draw the widest possible boundary between the groups
from sklearn.dummy import DummyClassifier
base = DummyClassifier(strategy="most_frequent")
base.fit(X_train_s, y_train)
print(round(base.score(X_test_s, y_test), 3))
0.735
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC
models = {
    "Logistic regression": LogisticRegression(max_iter=1000),
    "Decision tree": DecisionTreeClassifier(max_depth=5,
                                            random_state=42),
    "kNN": KNeighborsClassifier(n_neighbors=25),
    "Naive Bayes": GaussianNB(),
    "SVM": SVC(random_state=42),
}
for name, model in models.items():
    model.fit(X_train_s, y_train)
    acc = model.score(X_test_s, y_test)
    print(f"{name:20} {acc:.3f}")
Logistic regression  0.806
Decision tree        0.792
kNN                  0.786
Naive Bayes          0.661
SVM                  0.793

Your turn#

Fit a decision tree with no depth limit. Compare its training and test accuracy.

deep = DecisionTreeClassifier(random_state=42)
deep.fit(X_train_s, y_train)
print(round(deep.score(X_train_s, y_train), 3))
print(round(deep.score(X_test_s, y_test), 3))
0.998
0.738

Compare beyond accuracy#

Problem: accuracy hides how many churners each model actually catches.

from sklearn.metrics import accuracy_score, precision_score
from sklearn.metrics import recall_score, f1_score, roc_auc_score
def churn_scores(model):
    pred = model.predict(X_test_s)
    if hasattr(model, "predict_proba"):
        risk = model.predict_proba(X_test_s)[:, 1]
    else:
        risk = model.decision_function(X_test_s)
    return {"accuracy": accuracy_score(y_test, pred),
            "precision": precision_score(y_test, pred),
            "recall": recall_score(y_test, pred),
            "f1": f1_score(y_test, pred),
            "auc": roc_auc_score(y_test, risk)}
scores = {name: churn_scores(m) for name, m in models.items()}
table = pd.DataFrame(scores).T
table.sort_values("auc", ascending=False).round(3)
accuracy precision recall f1 auc
Logistic regression 0.806 0.661 0.555 0.603 0.846
kNN 0.786 0.599 0.589 0.594 0.823
Decision tree 0.792 0.651 0.467 0.544 0.822
Naive Bayes 0.661 0.431 0.869 0.576 0.813
SVM 0.793 0.643 0.497 0.560 0.804

Quick check: the retention team wants to reach as many leavers as possible. Which column matters most?

table[["recall", "precision"]].sort_values("recall").round(3)
recall precision
Decision tree 0.467 0.651
SVM 0.497 0.643
Logistic regression 0.555 0.661
kNN 0.589 0.599
Naive Bayes 0.869 0.431
from sklearn.metrics import RocCurveDisplay
fig, ax = plt.subplots(figsize=(7, 5))
for name, model in models.items():
    RocCurveDisplay.from_estimator(model, X_test_s, y_test,
                                   name=name, ax=ax)
ax.plot([0, 1], [0, 1], "k--", label="Random guess")
ax.set_title("ROC curves for five churn classifiers")
ax.set_xlabel("False positive rate (stayers flagged)")
ax.set_ylabel("True positive rate (leavers caught)")
ax.legend()
plt.show()
No description has been provided for this image

Catch more churners with class weights#

Problem: leavers are only a quarter of customers, so models play safe.

Setting class_weight="balanced" makes each mistake on a rare leaver cost more. We'll apply it to logistic regression, the best ranker.

logreg = models["Logistic regression"]
confusion_matrix(y_test, logreg.predict(X_test_s))
array([[1161,  133],
       [ 208,  259]])
balanced = LogisticRegression(max_iter=1000,
                              class_weight="balanced")
balanced.fit(X_train_s, y_train)
confusion_matrix(y_test, balanced.predict(X_test_s))
array([[949, 345],
       [ 95, 372]])
both = pd.DataFrame({
    "plain": churn_scores(logreg),
    "balanced": churn_scores(balanced)})
both.round(3)
plain balanced
accuracy 0.806 0.750
precision 0.661 0.519
recall 0.555 0.797
f1 0.603 0.628
auc 0.846 0.846

Your turn#

Give the depth-5 decision tree balanced class weights. What happens to its recall?

tree_bal = DecisionTreeClassifier(max_depth=5, random_state=42,
                                  class_weight="balanced")
tree_bal.fit(X_train_s, y_train)
print(round(churn_scores(models["Decision tree"])["recall"], 3))
print(round(churn_scores(tree_bal)["recall"], 3))
0.467
0.739

Explain what drives churn#

Problem: a score is not enough; the business needs reasons to act on.

weights = pd.Series(balanced.coef_[0], index=X.columns)
weights.nlargest(5).round(2)
InternetService_Fiber optic    0.78
TotalCharges                   0.52
StreamingMovies_Yes            0.29
StreamingTV_Yes                0.25
MultipleLines_Yes              0.19
dtype: float64

Quick check: will a two-year contract raise or lower the churn risk?

weights.nsmallest(5).round(2)
tenure               -1.16
MonthlyCharges       -1.02
Contract_Two year    -0.59
Contract_One year    -0.30
OnlineSecurity_Yes   -0.12
dtype: float64
top = weights.abs().nlargest(10).index
ax = weights[top].sort_values().plot(kind="barh", figsize=(7, 5))
ax.set_title("What pushes customers to leave or stay")
ax.set_xlabel("Weight (positive = more likely to churn)")
ax.set_ylabel("Feature")
plt.show()
No description has been provided for this image

Weights describe this model's pattern, not proof of cause.

Mini-project: build the retention call list#

The team can call 10% of customers. How many leavers would our list reach?

calls = df.loc[X_test.index, ["customerID", "Contract"]].copy()
calls["risk"] = balanced.predict_proba(X_test_s)[:, 1]
calls["churned"] = y_test
calls = calls.sort_values("risk", ascending=False)
calls.head().round(3)
customerID Contract risk churned
3380 5178-LMXOP Month-to-month 0.936 1
3346 2545-EBUPK Month-to-month 0.925 0
2631 6861-XWTWQ Month-to-month 0.924 1
4585 1069-XAIEM Month-to-month 0.924 1
6866 0295-PPHDO Month-to-month 0.923 1
n_calls = len(calls) // 10
called = calls.head(n_calls)
hit = called["churned"].mean()
base_rate = calls["churned"].mean()
print(n_calls, round(hit, 3), round(base_rate, 3))
176 0.744 0.265
calls["decile"] = pd.qcut(calls["risk"], 10, labels=False) + 1
by_decile = calls.groupby("decile")["churned"].mean()
ax = by_decile.plot(kind="bar", rot=0, figsize=(7, 4))
ax.axhline(base_rate, color="k", ls="--", label="Average")
ax.set_title("Real churn rate by predicted risk group")
ax.set_xlabel("Risk group (10 = highest predicted risk)")
ax.set_ylabel("Share who churned")
ax.legend()
plt.show()
No description has been provided for this image
caught = called["churned"].sum()
all_leavers = calls["churned"].sum()
lift = hit / base_rate
print(f"Calls reach {caught} of {all_leavers} leavers, "
      f"lift {lift:.1f}x")
Calls reach 131 of 467 leavers, lift 2.8x

Recap#

  • Fix hidden text values (TotalCharges) before any maths or model
  • Encode text with get_dummies; split with stratify; fit the scaler on train only
  • Always compare models against a do-nothing baseline
  • Choose the model by recall, precision and AUC, not accuracy alone
  • class_weight='balanced' trades precision for many more caught churners

Challenges#

  1. Change kNN's n_neighbors to 5 and 75. How do recall and AUC change?
  2. Drop TotalCharges and refit the balanced logistic regression. Does AUC drop?
  3. Build the call list for 20% of customers. What share are real leavers?

What's next#

Lesson 23: Introduction to Ensemble Learning Combine many models so their mistakes cancel out.

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.