Mathew K Analytics

Lesson 5 · Data Mining Full Course

Data Mining Full Course Part 5 (3 Hours): Market Basket Analysis, Outliers & Fraud Detection

All 8 lesson notebooks from Part 5 of the Data Mining Full Course in Python, in teaching order. Each lesson starts with a heading; the timestamps below jump…

📓 Full notebook

Download .ipynb

Data Mining Full Course, Part 5: Market Basket Analysis, Outliers and Fraud Detection#

All 8 lesson notebooks from Part 5 of the Data Mining Full Course in Python, in teaching order. Each lesson starts with a heading; the timestamps below jump to it in the video.

  • 0:00 Lesson 33: Association Rule Mining with Apriori Algorithm
  • 0:23:18 Lesson 34: FP-Growth and Market Basket Analysis
  • 0:47:48 Lesson 35: Sequential Pattern Mining
  • 1:12:07 Lesson 36: Introduction to Anomaly and Outlier Detection
  • 1:36:25 Lesson 37: Statistical Methods for Outlier Detection
  • 2:00:42 Lesson 38: Distance and Density-Based Outlier Detection
  • 2:23:58 Lesson 39: Applications in Fraud Detection and Cybersecurity
  • 2:46:46 Lesson 40: Hands-on Project- Credit Card Fraud Detection

Data files: save the datasets from the lesson page in the same folder as this notebook. Files the lessons create are written to that folder too.

Data Mining 33: Association Rule Mining with Apriori Algorithm#

In this lesson you'll be able to:

  • Turn raw shopping baskets into a True/False item table
  • Calculate support, confidence and lift by hand
  • Find frequent itemsets with Apriori and choose min_support
  • Generate, filter and read association rules
  • Turn rules into a store recommendation Dataset: 9,835 real grocery baskets from one month at a store (data/mining/groceries_baskets.csv)

Warm-up: from grouping rows to grouping items#

Lesson 32 grouped countries that looked alike. Today we look for items that appear together.

with open("groceries_baskets.csv") as f:
    baskets = [line.strip().split(",") for line in f]
print(len(baskets))
print(baskets[:3])
9835
[['citrus fruit', 'semi-finished bread', 'margarine', 'ready soups'], ['tropical fruit', 'yogurt', 'coffee'], ['whole milk']]
import pandas as pd
sizes = pd.Series([len(b) for b in baskets])
sizes.describe().round(2)
count    9835.00
mean        4.41
std         3.59
min         1.00
25%         2.00
50%         3.00
75%         6.00
max        32.00
dtype: float64

Quick check: what share of baskets hold just one item?

print((sizes == 1).sum(), round((sizes == 1).mean(), 3))
2159 0.22

From baskets to a True/False item table#

Problem: Apriori needs one row per basket and one True/False column per item.

items = pd.Series([i for b in baskets for i in b])
print(items.nunique())
items.value_counts().head(8)
171
whole milk          2513
other vegetables    1903
rolls/buns          1809
soda                1715
yogurt              1372
bottled water       1087
root vegetables     1072
tropical fruit      1032
Name: count, dtype: int64
import matplotlib.pyplot as plt
items.value_counts().head(15).sort_values().plot(kind="barh")
plt.title("Top 15 products by number of baskets")
plt.xlabel("Number of baskets")
plt.ylabel("Product")
plt.show()
No description has been provided for this image

Common mistake: handing the raw list of baskets to apriori

from mlxtend.frequent_patterns import apriori
apriori(baskets, min_support=0.02)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[7], line 2
      1 from mlxtend.frequent_patterns import apriori
----> 2 apriori(baskets, min_support=0.02)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\mlxtend\frequent_patterns\apriori.py:267, in apriori(df, min_support, use_colnames, max_len, verbose, low_memory, n_jobs)
    261 if min_support <= 0.0 or min_support > 1.0:
    262     raise ValueError(
    263         "`min_support` must be a positive "
    264         "number within the interval `(0, 1]`. "
    265         "Got %s." % min_support
    266     )
--> 267 fpc.valid_input_check(df)
    269 if hasattr(df, "sparse"):
    270     # DataFrame with SparseArray (pandas >= 0.24)
    272     if df.size == 0:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\mlxtend\frequent_patterns\fpcommon.py:152, in valid_input_check(df, null_values)
    141     msg = (
    142         "SparseDataFrame support has been deprecated in pandas 1.0,"
    143         " and is no longer supported in mlxtend. "
   (...)
    148         " for supporting sparse data in DataFrames."
    149     )
    150     raise TypeError(msg)
--> 152 if df.size == 0:
    153     return
    154 if hasattr(df, "sparse"):

AttributeError: 'list' object has no attribute 'size'
from mlxtend.preprocessing import TransactionEncoder
te = TransactionEncoder()
onehot = pd.DataFrame(te.fit_transform(baskets), columns=te.columns_)
print(onehot.shape)
(9835, 171)
onehot[["whole milk", "yogurt", "rolls/buns", "coffee"]].head()
whole milk yogurt rolls/buns coffee
0 False False False False
1 False True False True
2 True False False False
3 False True False False
4 True False False False

Support: how often does an itemset appear?#

Problem: we need a number for how common an item, or a group of items, is.

  • Itemset: one or more items, such as {whole milk, yogurt}
  • Support: the share of all baskets that contain the whole itemset

Common mistake: guessing a product's name

onehot["milk"].mean()
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\indexes\base.py:3641, in Index.get_loc(self, key)
   3640 try:
-> 3641     return self._engine.get_loc(casted_key)
   3642 except KeyError as err:

File pandas/_libs/index.pyx:168, in pandas._libs.index.IndexEngine.get_loc()

File pandas/_libs/index.pyx:197, in pandas._libs.index.IndexEngine.get_loc()

File pandas/_libs/hashtable_class_helper.pxi:7668, in pandas._libs.hashtable.PyObjectHashTable.get_item()

File pandas/_libs/hashtable_class_helper.pxi:7676, in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'milk'

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
Cell In[10], line 1
----> 1 onehot["milk"].mean()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\frame.py:4378, in DataFrame.__getitem__(self, key)
   4376 if self.columns.nlevels > 1:
   4377     return self._getitem_multilevel(key)
-> 4378 indexer = self.columns.get_loc(key)
   4379 if is_integer(indexer):
   4380     indexer = [indexer]

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\indexes\base.py:3648, in Index.get_loc(self, key)
   3643     if isinstance(casted_key, slice) or (
   3644         isinstance(casted_key, abc.Iterable)
   3645         and any(isinstance(x, slice) for x in casted_key)
   3646     ):
   3647         raise InvalidIndexError(key) from err
-> 3648     raise KeyError(key) from err
   3649 except TypeError:
   3650     # If we have a listlike key, _check_indexing_error will raise
   3651     #  InvalidIndexError. Otherwise we fall through and re-raise
   3652     #  the TypeError.
   3653     self._check_indexing_error(key)

KeyError: 'milk'
[c for c in onehot.columns if "milk" in c]
['UHT-milk', 'butter milk', 'condensed milk', 'whole milk']
milk = onehot["whole milk"]
yog = onehot["yogurt"]
print(round(milk.mean(), 3), round(yog.mean(), 3))
0.256 0.14

Quick check: can {whole milk, yogurt} have higher support than yogurt alone?

both = milk & yog
print(both.sum(), yog.sum())
print(round(both.mean(), 3))
551 1372
0.056

Your turn: what is the support of {beef, root vegetables}?

beef_root = onehot["beef"] & onehot["root vegetables"]
print(beef_root.sum(), round(beef_root.mean(), 4))
171 0.0174
frequent = apriori(onehot, min_support=0.02, use_colnames=True)
frequent["length"] = frequent["itemsets"].apply(len)
frequent["length"].value_counts()
length
2    61
1    59
3     2
Name: count, dtype: int64
for _, row in frequent[frequent["length"] == 3].iterrows():
    print(sorted(row["itemsets"]), round(row["support"], 3))
['other vegetables', 'root vegetables', 'whole milk'] 0.023
['other vegetables', 'whole milk', 'yogurt'] 0.022
pairs = frequent[frequent["length"] == 2]
pairs.nlargest(5, "support").round(3)
support itemsets length
91 0.075 frozenset({whole milk, other vegetables}) 2
103 0.057 frozenset({whole milk, rolls/buns}) 2
119 0.056 frozenset({whole milk, yogurt}) 2
106 0.049 frozenset({whole milk, root vegetables}) 2
85 0.047 frozenset({other vegetables, root vegetables}) 2
top10 = pairs.nlargest(10, "support")
names = top10["itemsets"].apply(lambda s: " + ".join(sorted(s)))
top10.set_index(names)["support"].sort_values().plot(kind="barh")
plt.title("Ten most common product pairs")
plt.xlabel("Support (share of baskets)")
plt.ylabel("Pair")
plt.show()
No description has been provided for this image

The Apriori trick and choosing min_support#

Problem: 171 products make far too many combinations to count them all.

  • If an itemset is rare, every bigger set containing it is rarer still
  • So Apriori grows sets one item at a time, dropping rare ones early
n = onehot.shape[1]
print(n * (n - 1) // 2)
rare = [c for c in onehot.columns if onehot[c].mean() < 0.02]
print(len(rare))
14535
112
a = onehot["whole milk"]
ab = a & onehot["other vegetables"]
abc = ab & onehot["rolls/buns"]
print(round(a.mean(), 3), round(ab.mean(), 3), round(abc.mean(), 3))
0.256 0.075 0.018
counts = {}
for s in [0.05, 0.03, 0.02, 0.01, 0.005]:
    counts[s] = len(apriori(onehot, min_support=s, use_colnames=True))
counts = pd.Series(counts)
counts
0.050      31
0.030      63
0.020     122
0.010     332
0.005    1000
dtype: int64
counts.plot(marker="o", logy=True)
plt.title("Frequent itemsets by minimum support")
plt.xlabel("Minimum support")
plt.ylabel("Number of itemsets (log scale)")
plt.show()
No description has been provided for this image

Common mistake: giving min_support as a number of baskets

apriori(onehot, min_support=50, use_colnames=True)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[23], line 1
----> 1 apriori(onehot, min_support=50, use_colnames=True)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\mlxtend\frequent_patterns\apriori.py:262, in apriori(df, min_support, use_colnames, max_len, verbose, low_memory, n_jobs)
    259     return np.array(out).reshape(-1)
    261 if min_support <= 0.0 or min_support > 1.0:
--> 262     raise ValueError(
    263         "`min_support` must be a positive "
    264         "number within the interval `(0, 1]`. "
    265         "Got %s." % min_support
    266     )
    267 fpc.valid_input_check(df)
    269 if hasattr(df, "sparse"):
    270     # DataFrame with SparseArray (pandas >= 0.24)

ValueError: `min_support` must be a positive number within the interval `(0, 1]`. Got 50.
share = 50 / len(onehot)
few = apriori(onehot, min_support=share, use_colnames=True)
print(round(share, 4), len(few))
0.0051 1000

Rules: confidence and lift#

Problem: knowing two items are common together isn't a recommendation yet.

  • Confidence(A -> B): of the baskets with A, the share that also have B
  • Lift: confidence divided by B's own support. Above 1 means A really boosts B
conf = both.sum() / yog.sum()
lift = conf / milk.mean()
print(round(conf, 3), round(lift, 3))
0.402 1.572

Quick check: is whole milk -> yogurt as strong as yogurt -> whole milk?

conf_back = both.sum() / milk.sum()
lift_back = conf_back / yog.mean()
print(round(conf_back, 3), round(lift_back, 3))
0.219 1.572
from mlxtend.frequent_patterns import association_rules
freq = apriori(onehot, min_support=0.01, use_colnames=True)
rules = association_rules(freq, num_itemsets=len(onehot),
                          metric="confidence", min_threshold=0.3)
print(len(rules))
125
rules["if"] = rules["antecedents"].apply(lambda s: ", ".join(sorted(s)))
rules["then"] = rules["consequents"].apply(lambda s: ", ".join(sorted(s)))
rules = rules.sort_values(["if", "then"], ignore_index=True)
cols = ["if", "then", "confidence", "lift"]
by_conf = rules.sort_values("confidence", ascending=False,
                            kind="stable")
by_conf[cols].head().round(3)
if then confidence lift
23 citrus fruit, root vegetables other vegetables 0.586 3.030
94 root vegetables, tropical fruit other vegetables 0.585 3.021
35 curd, yogurt whole milk 0.582 2.279
14 butter, other vegetables whole milk 0.574 2.245
95 root vegetables, tropical fruit whole milk 0.570 2.231

Quick check: most top rules end in whole milk. Is milk really special?

print(round((rules["then"] == "whole milk").mean(), 3))
print(round(milk.mean(), 3))
0.528
0.256
by_lift = rules.sort_values("lift", ascending=False, kind="stable")
by_lift[cols].head().round(3)
if then confidence lift
21 citrus fruit, other vegetables root vegetables 0.359 3.295
70 other vegetables, tropical fruit root vegetables 0.343 3.145
1 beef root vegetables 0.331 3.040
23 citrus fruit, root vegetables other vegetables 0.586 3.030
94 root vegetables, tropical fruit other vegetables 0.585 3.021

Quick check: can lift drop below 1? What would that mean?

every = association_rules(freq, num_itemsets=len(onehot),
                          metric="lift", min_threshold=0)
print(len(every), (every["lift"] < 1).sum())
every.nsmallest(2, "lift")[["antecedents", "consequents", "lift"]].round(3)
618 20
antecedents consequents lift
86 frozenset({canned beer}) frozenset({rolls/buns}) 0.79
87 frozenset({rolls/buns}) frozenset({canned beer}) 0.79
rules.plot.scatter(x="confidence", y="lift", alpha=0.6)
plt.axhline(1, color="grey", linestyle="--")
plt.title(f"Confidence against lift for {len(rules)} rules")
plt.xlabel("Confidence")
plt.ylabel("Lift")
plt.show()
No description has been provided for this image

Your turn: list the rules whose if side contains root vegetables, best lift first.

has_root = by_lift["antecedents"].apply(
    lambda s: "root vegetables" in s)
root = by_lift[has_root]
print(len(root))
root[cols].head().round(3)
11
if then confidence lift
23 citrus fruit, root vegetables other vegetables 0.586 3.030
94 root vegetables, tropical fruit other vegetables 0.585 3.021
86 rolls/buns, root vegetables other vegetables 0.502 2.595
97 root vegetables, yogurt other vegetables 0.500 2.584
96 root vegetables, whole milk other vegetables 0.474 2.450

Mini-project: a cross-promotion shortlist for the store#

Find strong, common pair rules a store manager could act on this week.

pair = rules["antecedents"].apply(len) == 1
strong = rules[pair & (rules["lift"] > 1.5)
                & (rules["support"] >= 0.015)]
strong = strong.sort_values("lift", ascending=False, kind="stable")
print(len(strong))
strong[cols + ["support"]].head(8).round(3)
36
if then confidence lift support
1 beef root vegetables 0.331 3.040 0.017
33 curd yogurt 0.324 2.326 0.017
92 root vegetables other vegetables 0.435 2.247 0.047
16 chicken other vegetables 0.417 2.155 0.018
114 whipped/sour cream other vegetables 0.403 2.082 0.029
11 butter whole milk 0.497 1.946 0.028
0 beef other vegetables 0.376 1.943 0.020
82 pork other vegetables 0.376 1.941 0.022
strong[["if", "then"]].value_counts("then")
then
whole milk          18
other vegetables    14
rolls/buns           2
root vegetables      1
yogurt               1
Name: count, dtype: int64

Your turn: find the best shortlisted rule whose then side is not a staple.

staples = ["whole milk", "other vegetables"]
fresh = strong[~strong["then"].isin(staples)]
fresh[cols].round(3)
if then confidence lift
1 beef root vegetables 0.331 3.040
33 curd yogurt 0.324 2.326
42 frankfurter rolls/buns 0.326 1.772
99 sausage rolls/buns 0.326 1.771
top = strong.iloc[0]
n = round(top["support"] * len(onehot))
print(top["if"], "->", top["then"], n,
      round(top["confidence"], 3), round(top["lift"], 3))
beef -> root vegetables 171 0.331 3.04
import seaborn as sns
grid = strong.pivot(index="if", columns="then", values="lift")
plt.figure(figsize=(6, 7))
sns.heatmap(grid.round(1), annot=True, cmap="Greens")
plt.title("Lift of the shortlisted rules")
plt.show()
No description has been provided for this image

Honesty check: a real rule should hold in both halves of the baskets.

half = len(onehot) // 2
for part in [onehot.iloc[:half], onehot.iloc[half:]]:
    b, r = part["beef"], part["root vegetables"]
    conf_h = (b & r).sum() / b.sum()
    print(len(part), round(conf_h, 3), round(conf_h / r.mean(), 3))
4917 0.336 2.919
4918 0.327 3.18
import os
strong[cols + ["support"]].round(3).to_csv(
    "m33_rules.csv",
    index=False)
print(pd.read_csv("m33_rules.csv").shape)
(36, 5)

Finding: beef -> root vegetables is the standout: 171 baskets, lift 3.04. 32 of the 36 shortlisted rules end in whole milk or other vegetables, the store's staples.

Recap#

  • Encode baskets as a True/False table with TransactionEncoder
  • Support: share of baskets containing the whole itemset
  • Apriori prunes any set whose subsets are already rare
  • min_support is a share between 0 and 1, not a count
  • Confidence can flatter popular items; lift above 1 is the real signal

Challenges#

  1. Rerun the rules with min_support 0.005. How many rules have lift above 2?
  2. Find every rule whose then side is not whole milk or other vegetables.
  3. Keep only baskets with 5 or more items and repeat the mini-project.

What's next#

Lesson 34: FP-Growth and Market Basket Analysis A faster way to find the same itemsets, and deeper basket analysis.

Data Mining 34: FP-Growth and Market Basket Analysis#

In this lesson you'll be able to:

  • Explain why Apriori slows down, and how an FP-tree avoids it
  • Mine frequent itemsets with FP-Growth in mlxtend
  • Read support, confidence and lift on association rules
  • Filter hundreds of rules down to the few worth acting on
  • Test a rule-based recommender on baskets it has never seen Dataset: 9,835 real grocery baskets (data/mining/groceries_baskets.csv)

Warm-up: Apriori from lesson 33#

Load the baskets, one-hot encode them, and find the common items.

import pandas as pd
from mlxtend.preprocessing import TransactionEncoder
from mlxtend.frequent_patterns import apriori
with open("groceries_baskets.csv") as f:
    baskets = [line.rstrip("\n").split(",") for line in f]
print(len(baskets))
print(baskets[:3])
9835
[['citrus fruit', 'semi-finished bread', 'margarine', 'ready soups'], ['tropical fruit', 'yogurt', 'coffee'], ['whole milk']]
te = TransactionEncoder()
X = pd.DataFrame(te.fit_transform(baskets), columns=te.columns_)
print(X.shape)
freq_ap = apriori(X, min_support=0.05, use_colnames=True)
freq_ap.sort_values("support", ascending=False).head()
(9835, 169)
support itemsets
26 0.255516 frozenset({whole milk})
15 0.193493 frozenset({other vegetables})
19 0.183935 frozenset({rolls/buns})
23 0.174377 frozenset({soda})
27 0.139502 frozenset({yogurt})

Why Apriori slows down#

Problem: lower the support threshold and the work explodes.

sizes = {}
for s in [0.05, 0.02, 0.01, 0.005, 0.003]:
    sizes[s] = len(apriori(X, min_support=s, use_colnames=True))
print(sizes)
{0.05: 31, 0.02: 122, 0.01: 333, 0.005: 1001, 0.003: 2226}
import matplotlib.pyplot as plt
pd.Series(sizes).plot(marker="o", logy=True)
plt.title("Lower support, many more itemsets")
plt.xlabel("Minimum support")
plt.ylabel("Frequent itemsets (log scale)")
plt.show()
No description has been provided for this image

Quick check: with 169 products, how many possible pairs and triples are there?

import math
print(math.comb(169, 2), math.comb(169, 3))
14196 790244

Apriori builds candidate itemsets level by level and rescans every basket to count them.

The FP-tree: squeeze the baskets into shared paths#

Problem: read the baskets once, store them compactly, and never list candidates.

  • Count every product once, then sort each basket from most to least common
  • Baskets that start the same way share one path in the tree
  • Mine patterns from the tree, one product at a time
counts = X.sum().sort_values(ascending=False)
counts.head(10).sort_values().plot.barh()
plt.title("The 10 most common grocery items")
plt.xlabel("Number of baskets")
plt.ylabel("Product")
plt.show()
No description has been provided for this image
rank = {item: i for i, item in enumerate(counts.index)}
for basket in baskets[:4]:
    print(sorted(basket, key=rank.get))
['citrus fruit', 'margarine', 'semi-finished bread', 'ready soups']
['yogurt', 'tropical fruit', 'coffee']
['whole milk']
['yogurt', 'pip fruit', 'cream cheese ', 'meat spreads']
nodes = set()
for basket in baskets:
    path = tuple(sorted(basket, key=rank.get))
    for k in range(1, len(path) + 1):
        nodes.add(path[:k])
print(sum(len(b) for b in baskets), len(nodes))
43367 22653
paths = [tuple(sorted(b, key=rank.get)) for b in baskets]
pd.Series([p[0] for p in paths]).value_counts().head()
whole milk          2513
other vegetables    1167
rolls/buns          1009
soda                 900
yogurt               395
Name: count, dtype: int64

To mine one item, FP-Growth collects the paths that lead to it: its prefix paths.

prefixes = [p[:p.index("curd")] for p in paths if "curd" in p]
print(len(prefixes))
above = pd.Series([i for p in prefixes for i in p]).value_counts()
above.head()
524
whole milk            257
yogurt                170
other vegetables      169
root vegetables       107
whipped/sour cream    103
Name: count, dtype: int64

Quick check: what is the support of curd and whole milk together?

print(round(above["whole milk"] / len(baskets), 4))
print(round((X["curd"] & X["whole milk"]).mean(), 4))
0.0261
0.0261

FP-Growth in mlxtend#

Problem: find the same itemsets as Apriori, faster.

Common mistake: handing FP-Growth the raw list of baskets

from mlxtend.frequent_patterns import fpgrowth
fpgrowth(baskets, min_support=0.01)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[13], line 2
      1 from mlxtend.frequent_patterns import fpgrowth
----> 2 fpgrowth(baskets, min_support=0.01)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\mlxtend\frequent_patterns\fpgrowth.py:78, in fpgrowth(df, min_support, null_values, use_colnames, max_len, verbose)
     12 def fpgrowth(
     13     df, min_support=0.5, null_values=False, use_colnames=False, max_len=None, verbose=0
     14 ):
     15     """Get frequent itemsets from a one-hot DataFrame
     16 
     17     Parameters
   (...)
     76 
     77     """
---> 78     fpc.valid_input_check(df, null_values)
     80     if min_support <= 0.0 or min_support > 1.0:
     81         raise ValueError(
     82             "`min_support` must be a positive "
     83             "number within the interval `(0, 1]`. "
     84             "Got %s." % min_support
     85         )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\mlxtend\frequent_patterns\fpcommon.py:152, in valid_input_check(df, null_values)
    141     msg = (
    142         "SparseDataFrame support has been deprecated in pandas 1.0,"
    143         " and is no longer supported in mlxtend. "
   (...)
    148         " for supporting sparse data in DataFrames."
    149     )
    150     raise TypeError(msg)
--> 152 if df.size == 0:
    153     return
    154 if hasattr(df, "sparse"):

AttributeError: 'list' object has no attribute 'size'
freq = fpgrowth(X, min_support=0.01, use_colnames=True)
print(len(freq))
freq.sort_values("support", ascending=False).head()
333
support itemsets
6 0.255516 frozenset({whole milk})
9 0.193493 frozenset({other vegetables})
13 0.183935 frozenset({rolls/buns})
24 0.174377 frozenset({soda})
3 0.139502 frozenset({yogurt})
freq_ap = apriori(X, min_support=0.01, use_colnames=True)
same = set(freq["itemsets"]) == set(freq_ap["itemsets"])
print(len(freq_ap), same)
333 True
import time
for algo in [apriori, fpgrowth]:
    start = time.perf_counter()
    found = algo(X, min_support=0.003, use_colnames=True)
    secs = time.perf_counter() - start
    print(algo.__name__, len(found), round(secs, 2))
apriori 2226 0.33
fpgrowth 2226 0.23

Common mistake: trusting product names to be clean

print(X["cream cheese"].mean())
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\indexes\base.py:3641, in Index.get_loc(self, key)
   3640 try:
-> 3641     return self._engine.get_loc(casted_key)
   3642 except KeyError as err:

File pandas/_libs/index.pyx:168, in pandas._libs.index.IndexEngine.get_loc()

File pandas/_libs/index.pyx:197, in pandas._libs.index.IndexEngine.get_loc()

File pandas/_libs/hashtable_class_helper.pxi:7668, in pandas._libs.hashtable.PyObjectHashTable.get_item()

File pandas/_libs/hashtable_class_helper.pxi:7676, in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'cream cheese'

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
Cell In[17], line 1
----> 1 print(X["cream cheese"].mean())

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\frame.py:4378, in DataFrame.__getitem__(self, key)
   4376 if self.columns.nlevels > 1:
   4377     return self._getitem_multilevel(key)
-> 4378 indexer = self.columns.get_loc(key)
   4379 if is_integer(indexer):
   4380     indexer = [indexer]

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\indexes\base.py:3648, in Index.get_loc(self, key)
   3643     if isinstance(casted_key, slice) or (
   3644         isinstance(casted_key, abc.Iterable)
   3645         and any(isinstance(x, slice) for x in casted_key)
   3646     ):
   3647         raise InvalidIndexError(key) from err
-> 3648     raise KeyError(key) from err
   3649 except TypeError:
   3650     # If we have a listlike key, _check_indexing_error will raise
   3651     #  InvalidIndexError. Otherwise we fall through and re-raise
   3652     #  the TypeError.
   3653     self._check_indexing_error(key)

KeyError: 'cream cheese'
[c for c in X.columns if c != c.strip()]
['cream cheese ', 'roll products ']
baskets = [[item.strip() for item in b] for b in baskets]
X = pd.DataFrame(te.fit_transform(baskets), columns=te.columns_)
print(round(X["cream cheese"].mean(), 4))
freq = fpgrowth(X, min_support=0.01, use_colnames=True)
print(len(freq))
0.0397
333

Quick check: at 0.3% support, how many itemsets remain if we only allow pairs or smaller?

short = fpgrowth(X, min_support=0.003, max_len=2, use_colnames=True)
print(len(short))
1276

Your turn: how many frequent itemsets have 1, 2 and 3 items? Show the top triples.

freq["size"] = freq["itemsets"].apply(len)
print(freq["size"].value_counts().sort_index())
triples = freq[freq["size"] == 3].sort_values("support",
                                              ascending=False)
triples.assign(items=triples["itemsets"].apply(sorted).str.join(", "))[
    ["items", "support"]].head().round(3)
size
1     88
2    213
3     32
Name: count, dtype: int64
items support
244 other vegetables, root vegetables, whole milk 0.023
114 other vegetables, whole milk, yogurt 0.022
158 other vegetables, rolls/buns, whole milk 0.018
124 other vegetables, tropical fruit, whole milk 0.017
112 rolls/buns, whole milk, yogurt 0.016

From itemsets to rules#

Problem: an itemset says what goes together; a rule says what to suggest next.

  • Support: share of all baskets holding both sides
  • Confidence: of the baskets with the if side, the share that also have the then side
  • Lift: confidence divided by how popular the then side is anyway
from mlxtend.frequent_patterns import association_rules
rules = association_rules(freq, metric="lift", min_threshold=1.0)
rules["if"] = rules["antecedents"].apply(sorted).str.join(", ")
rules["then"] = rules["consequents"].apply(sorted).str.join(", ")
cols = ["if", "then", "confidence", "lift"]
print(len(rules))
top_lift = rules.sort_values("lift", ascending=False)
top_lift[["if", "then", "lift"]].head().round(3)
598
if then lift
255 curd whole milk, yogurt 3.372
254 whole milk, yogurt curd 3.372
31 root vegetables citrus fruit, other vegetables 3.295
30 citrus fruit, other vegetables root vegetables 3.295
522 other vegetables, yogurt whipped/sour cream 3.267
a, b = X["butter"], X["whole milk"]
support = (a & b).mean()
confidence = (a & b).sum() / a.sum()
lift = confidence / b.mean()
print(round(support, 3), round(confidence, 3), round(lift, 2))
0.028 0.497 1.95
pick = (rules["if"] == "butter") & (rules["then"] == "whole milk")
rules.loc[pick, cols].round(3)
if then confidence lift
174 butter whole milk 0.497 1.946

Quick check: whole milk is in about a quarter of baskets. Is a rule ending in whole milk with 40% confidence impressive?

milk = X["whole milk"].mean()
print(round(milk, 3), round(0.40 / milk, 2))
0.256 1.57

Quick check: what happens if we mine rules from itemsets found at 10% support?

high = fpgrowth(X, min_support=0.1, use_colnames=True)
print(len(high), high["itemsets"].apply(len).max())
print(len(association_rules(high, metric="lift", min_threshold=1)))
8 1
0

Choosing rules worth acting on#

Problem: hundreds of rules, and most are obvious or rare.

pts = plt.scatter(rules["support"], rules["confidence"],
                  c=rules["lift"], cmap="viridis", s=15)
plt.colorbar(pts, label="Lift")
plt.title("Every rule: support, confidence and lift")
plt.xlabel("Support (share of baskets)")
plt.ylabel("Confidence")
plt.show()
No description has been provided for this image
strong = rules[(rules["confidence"] >= 0.4) & (rules["lift"] >= 2)]
print(len(strong))
strong.sort_values("confidence", ascending=False)[cols].head().round(3)
29
if then confidence lift
29 citrus fruit, root vegetables other vegetables 0.586 3.030
116 root vegetables, tropical fruit other vegetables 0.585 3.021
253 curd, yogurt whole milk 0.582 2.279
186 butter, other vegetables whole milk 0.574 2.245
122 root vegetables, tropical fruit whole milk 0.570 2.231
top = strong.sort_values("leverage", ascending=False).head(3)
top = top.assign(extra=(top["leverage"] * len(X)).round().astype(int))
top[["if", "then", "leverage", "extra"]].round(4)
if then leverage extra
372 root vegetables other vegetables 0.0263 259
499 whipped/sour cream other vegetables 0.0150 148
383 root vegetables, whole milk other vegetables 0.0137 135

Hand the shortlist to the store team as a CSV file.

import os
shortlist = strong.sort_values("lift", ascending=False)[cols]
shortlist.to_csv("m34_strong_rules.csv", index=False)
saved = pd.read_csv("m34_strong_rules.csv")
print(saved.shape)
saved.head(3).round(3)
(29, 4)
if then confidence lift
0 citrus fruit, root vegetables other vegetables 0.586 3.030
1 root vegetables, tropical fruit other vegetables 0.585 3.021
2 rolls/buns, root vegetables other vegetables 0.502 2.595
best = strong.sort_values("lift").tail(10)
labels = best["if"] + " -> " + best["then"]
plt.barh(labels, best["lift"])
plt.title("Ten strong rules ranked by lift")
plt.xlabel("Lift")
plt.ylabel("Rule")
plt.show()
No description has been provided for this image

Your turn: which rules lead to root vegetables? Show the five with the highest lift.

to_roots = rules[rules["then"] == "root vegetables"]
to_roots.sort_values("lift", ascending=False)[cols].head().round(3)
if then confidence lift
30 citrus fruit, other vegetables root vegetables 0.359 3.295
118 other vegetables, tropical fruit root vegetables 0.343 3.145
263 beef root vegetables 0.331 3.040
384 other vegetables, whole milk root vegetables 0.310 2.842
408 other vegetables, yogurt root vegetables 0.297 2.729

Mini-project: do the rules predict real baskets?#

Hide one item from a basket. Can rules guess it better than just suggesting the bestsellers? Mine rules on 80% of baskets; test on the other 20%.

from sklearn.model_selection import train_test_split
train, test = train_test_split(baskets, test_size=0.2,
                               random_state=42)
te_train = TransactionEncoder()
X_train = pd.DataFrame(te_train.fit_transform(train),
                       columns=te_train.columns_)
freq_train = fpgrowth(X_train, min_support=0.005, use_colnames=True)
shop_rules = association_rules(freq_train, metric="confidence",
                               min_threshold=0.1)
shop_rules = shop_rules.sort_values("confidence", ascending=False)
print(len(train), len(test), len(shop_rules))
7868 1967 1809
popular = list(X_train.mean().sort_values(ascending=False).index)
def recommend(basket, n=3):
    have, picks = set(basket), []
    for ante, cons in zip(shop_rules["antecedents"],
                          shop_rules["consequents"]):
        if ante <= have:
            picks += [i for i in sorted(cons) if i not in have | set(picks)]
        if len(picks) >= n:
            break
    picks += [i for i in popular if i not in have | set(picks)]
    return picks[:n]
print(recommend(["butter"]))
['whole milk', 'other vegetables', 'yogurt']
sample = next(b for b in test if len(b) >= 4)
print(sample)
print(recommend(sample[:-1]))
['whole milk', 'beverages', 'pastry', 'newspapers']
['other vegetables', 'yogurt', 'soda']
def hit_rate(suggest):
    hits = tries = 0
    for basket in test:
        if len(basket) < 2:
            continue
        for i, item in enumerate(basket):
            rest = basket[:i] + basket[i + 1:]
            hits += item in suggest(rest)
            tries += 1
    return tries, round(hits / tries, 3)
def bestsellers(basket, n=3):
    return [i for i in popular if i not in basket][:n]

print(hit_rate(recommend))
print(hit_rate(bestsellers))
(8000, 0.196)
(8000, 0.179)

Finding: rules guess the hidden item 19.6% of the time, against 17.9% for bestsellers. A real but modest edge: grocery baskets are dominated by staples.

Recap#

  • Apriori lists candidates level by level; at low support that explodes
  • FP-Growth packs baskets into a tree of shared prefixes, then mines it
  • Encode baskets as a one-hot table, and clean item names first
  • Confidence says how often a rule is right; lift says whether it beats chance
  • Judge rules on held-out baskets, against a simple baseline

Challenges#

  1. Rerun the mini-project with min_support 0.003. Does the hit rate improve?
  2. Rank shop_rules by lift instead of confidence and compare.
  3. Find the rule with the highest lift that has three items on the if side.

What's next#

Lesson 35: Sequential Pattern Mining, where the order of purchases matters.

Data Mining 35: Sequential Pattern Mining#

In this lesson you'll be able to:

  • Turn raw transactions into one time-ordered sequence per customer
  • Test whether a sequence contains a pattern, and measure its support
  • Mine every two-step pattern, and split reorders from cross-sells
  • Grow longer patterns with the Apriori trick
  • Time a follow-up offer from real purchase gaps Dataset: Online Retail, a year of real orders from a UK online gift shop (data/pandas/online_retail.csv.gz)

Warm-up: what sells together (lesson 34)#

Load the orders, keep real sales, and run FP-Growth on the invoices.

import pandas as pd
df = pd.read_csv("online_retail.csv.gz")
sales = df.dropna(subset=["CustomerID"])
sales = sales[~sales["InvoiceNo"].str.startswith("C")]
keep = (sales["Quantity"] > 0) & sales["StockCode"].str.match(r"\d")
sales = sales[keep].copy()
print(len(df), len(sales))
print(sales["CustomerID"].nunique(), sales["InvoiceNo"].nunique())
541909 396370
4335 18405
from mlxtend.preprocessing import TransactionEncoder
from mlxtend.frequent_patterns import fpgrowth
baskets = sales.groupby("InvoiceNo")["Description"].apply(set)
te = TransactionEncoder()
X = pd.DataFrame(te.fit_transform(baskets.tolist()),
                 columns=te.columns_)
freq = fpgrowth(X, min_support=0.02, use_colnames=True, max_len=2)
pairs = freq[freq["itemsets"].apply(len) == 2].copy()
pairs["items"] = pairs["itemsets"].apply(sorted).str.join(" + ")
top = pairs.sort_values("support", ascending=False)
for r in top.head(3).itertuples():
    print(r.items, round(r.support, 3))
JUMBO BAG PINK POLKADOT + JUMBO BAG RED RETROSPOT 0.03
GREEN REGENCY TEACUP AND SAUCER + ROSES REGENCY TEACUP AND SAUCER  0.029
ALARM CLOCK BAKELIKE GREEN + ALARM CLOCK BAKELIKE RED  0.029

FP-Growth sees each basket on its own. It can't say which purchase came first.

Order matters: put purchases on a timeline#

Problem: baskets forget time; sequences keep it.

  • A sequence is one customer's baskets, in time order
  • A pattern like <A, B> means: bought A, then B in a later basket
  • Support is the share of customers whose sequence contains the pattern

Common mistake: using dates before converting them from text

sales["InvoiceDate"].dt.month.head()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[4], line 1
----> 1 sales["InvoiceDate"].dt.month.head()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\generic.py:6194, in NDFrame.__getattr__(self, name)
   6187 if (
   6188     name not in self._internal_names_set
   6189     and name not in self._metadata
   6190     and name not in self._accessors
   6191     and self._info_axis._can_hold_identifiers_and_holds_name(name)
   6192 ):
   6193     return self[name]
-> 6194 return object.__getattribute__(self, name)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\accessor.py:230, in Accessor.__get__(self, obj, cls)
    227 if obj is None:
    228     # we're accessing the attribute of the class, i.e., Dataset.geo
    229     return self._accessor
--> 230 return self._accessor(obj)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\indexes\accessors.py:698, in CombinedDatetimelikeProperties.__new__(cls, data)
    695 elif isinstance(data.dtype, PeriodDtype):
    696     return PeriodProperties(data, orig)
--> 698 raise AttributeError("Can only use .dt accessor with datetimelike values")

AttributeError: Can only use .dt accessor with datetimelike values

Quick check: if we sort the raw date text, which date comes first and which comes last?

sales["InvoiceDate"].sort_values().iloc[[0, -1]].tolist()
['1/10/2011 10:32', '9/9/2011 9:52']
sales["InvoiceDate"] = pd.to_datetime(sales["InvoiceDate"],
                                      format="%m/%d/%Y %H:%M")
print(sales["InvoiceDate"].min(), sales["InvoiceDate"].max())
print(sales["InvoiceDate"].dt.month.head(3).tolist())
2010-12-01 08:26:00 2011-12-09 12:50:00
[12, 12, 12]

One sequence per customer#

Problem: turn hundreds of thousands of rows into ordered lists of baskets.

inv = (sales.groupby(["CustomerID", "InvoiceNo"])
            .agg(date=("InvoiceDate", "min"),
                 n_items=("Description", "nunique"),
                 items=("Description", set))
            .reset_index()
            .sort_values(["CustomerID", "date", "InvoiceNo"]))
print(len(inv))
inv[["CustomerID", "InvoiceNo", "date", "n_items"]].head()
18405
CustomerID InvoiceNo date n_items
0 12346.0 541431 2011-01-18 10:01:00 1
1 12347.0 537626 2010-12-07 14:57:00 31
2 12347.0 542237 2011-01-26 14:30:00 29
3 12347.0 549222 2011-04-07 10:43:00 24
4 12347.0 556201 2011-06-09 13:01:00 18
inv.loc[inv["CustomerID"] == 12347, ["date", "n_items"]]
date n_items
1 2010-12-07 14:57:00 31
2 2011-01-26 14:30:00 29
3 2011-04-07 10:43:00 24
4 2011-06-09 13:01:00 18
5 2011-08-02 08:48:00 22
6 2011-10-31 12:25:00 47
7 2011-12-07 15:52:00 11
for r in inv[inv["CustomerID"] == 12891].itertuples():
    print(r.date.date(), sorted(r.items))
2011-02-20 ['SKULLS AND CROSSBONES WRAP']
2011-05-03 ['RED RETROSPOT WRAP ']
2011-06-07 ['SKULLS AND CROSSBONES WRAP']
seqs = inv.groupby("CustomerID")["items"].apply(list)
print(len(seqs))
print(seqs.str.len().describe().round(1))
4335
count    4335.0
mean        4.2
std         7.6
min         1.0
25%         1.0
50%         2.0
75%         5.0
max       206.0
Name: items, dtype: float64

Quick check: what share of customers shopped only once?

print(round((seqs.str.len() == 1).mean(), 3))
0.347
import matplotlib.pyplot as plt
seqs.str.len().clip(upper=20).plot.hist(bins=range(1, 22))
plt.title("How many times each customer ordered")
plt.xlabel("Baskets per customer (20 means 20 or more)")
plt.ylabel("Customers")
plt.show()
No description has been provided for this image
monthly = inv.set_index("date").resample("MS").size()
monthly.plot(marker="o")
plt.title("Baskets per month")
plt.xlabel("Month")
plt.ylabel("Baskets")
plt.show()
No description has been provided for this image
repeat = seqs[seqs.str.len() >= 2]
N = len(repeat)
print(N)
2829

Does a sequence contain a pattern?#

Problem: check 'bought A, then B later' for any customer.

def contains(seq, pattern):
    k = 0
    for basket in seq:
        if pattern[k] in basket:
            k += 1
            if k == len(pattern):
                return True
    return False
seq = seqs.loc[12347]
a, b = sorted(seq[0])[0], sorted(seq[-1])[0]
print(a, "|", b)
print(contains(seq, [a, b]), contains(seq, [b, a]))
3D DOG PICTURE PLAYING CARDS | BICYCLE PUNCTURE REPAIR KIT 
True False
def support(pattern):
    hits = sum(contains(seq, pattern) for seq in repeat)
    return round(hits / N, 4)

Quick check: cakestand then cake tins, or cake tins then cakestand: which order is more common?

Common mistake: passing the steps as separate arguments

stand, tins = "REGENCY CAKESTAND 3 TIER", "SET OF 3 REGENCY CAKE TINS"
support(stand, tins)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[18], line 2
      1 stand, tins = "REGENCY CAKESTAND 3 TIER", "SET OF 3 REGENCY CAKE TINS"
----> 2 support(stand, tins)

TypeError: support() takes 1 positional argument but 2 were given
print(support([stand, tins]), support([tins, stand]))
same = sum(any(stand in b and tins in b for b in seq)
           for seq in repeat)
print(round(same / N, 4))
0.0728 0.0414
0.0615

Your turn: what share of repeat customers bought the white hanging heart T-light holder, then bought it again later?

print(support(["WHITE HANGING HEART T-LIGHT HOLDER"] * 2))
0.1432

Mining every two-step pattern#

Problem: hundreds of popular products make hundreds of thousands of ordered pairs.

from collections import Counter
bought = Counter(i for seq in repeat for i in set().union(*seq))
item_sup = pd.Series(bought) / N
frequent = set(item_sup[item_sup >= 0.05].index)
print(len(item_sup), len(frequent))
3837 481

One pass per customer: note each product's first and last basket. <A, B> is present when A's first basket comes before B's last.

pair_counts = Counter()
for seq in repeat:
    first, last = {}, {}
    for k, basket in enumerate(seq):
        for item in basket & frequent:
            first.setdefault(item, k)
            last[item] = k
    pair_counts.update((a, b) for a in first for b in last
                       if first[a] < last[b])
print(len(pair_counts))
230705
pat = (pd.Series(pair_counts, name="support").div(N)
         .rename_axis(["first", "then"]).reset_index())
pat = pat.sort_values(["support", "first", "then"],
                      ascending=[False, True, True])
print(pat.head().round(4).to_string(index=False))
                             first                               then  support
WHITE HANGING HEART T-LIGHT HOLDER WHITE HANGING HEART T-LIGHT HOLDER   0.1432
          REGENCY CAKESTAND 3 TIER           REGENCY CAKESTAND 3 TIER   0.1166
           JUMBO BAG RED RETROSPOT            JUMBO BAG RED RETROSPOT   0.1103
     ASSORTED COLOUR BIRD ORNAMENT      ASSORTED COLOUR BIRD ORNAMENT   0.1029
                     PARTY BUNTING                      PARTY BUNTING   0.0990

Quick check: does the fast count agree with our contains test for cakestand then cake tins?

row = pat[(pat["first"] == stand) & (pat["then"] == tins)]
print(row["support"].round(4).tolist(), support([stand, tins]))
[0.0728] 0.0728

Two kinds of pattern: reorders (same product again) and cross-sells (a different product later).

reorders = pat[pat["first"] == pat["then"]]
cross = pat[pat["first"] != pat["then"]]
print(len(reorders), len(cross))
print(reorders[["first", "support"]].head().round(4).to_string(index=False))
481 230224
                             first  support
WHITE HANGING HEART T-LIGHT HOLDER   0.1432
          REGENCY CAKESTAND 3 TIER   0.1166
           JUMBO BAG RED RETROSPOT   0.1103
     ASSORTED COLOUR BIRD ORNAMENT   0.1029
                     PARTY BUNTING   0.0990
top = reorders.head(10).iloc[::-1]
plt.barh(top["first"].str.title(), top["support"])
plt.title("Products customers come back for")
plt.xlabel("Share of repeat customers who reorder")
plt.ylabel("Product")
plt.show()
No description has been provided for this image
cross = cross.assign(
    confidence=cross["support"] / cross["first"].map(item_sup),
    lift=lambda t: t["confidence"] / t["then"].map(item_sup))
common = cross[cross["support"] >= 0.03]
best = common.sort_values("confidence", ascending=False, kind="stable")
for r in best.head(5).itertuples():
    print(f"{r.first} -> {r.then}: {r.confidence:.2f}, {r.lift:.1f}")
CANDLEHOLDER PINK HANGING HEART -> WHITE HANGING HEART T-LIGHT HOLDER: 0.70, 2.8
ALARM CLOCK BAKELIKE ORANGE -> ALARM CLOCK BAKELIKE RED : 0.68, 5.5
ALARM CLOCK BAKELIKE ORANGE -> ALARM CLOCK BAKELIKE GREEN: 0.64, 6.3
ALARM CLOCK BAKELIKE CHOCOLATE -> ALARM CLOCK BAKELIKE RED : 0.59, 4.7
ALARM CLOCK BAKELIKE PINK -> ALARM CLOCK BAKELIKE RED : 0.57, 4.6
plt.scatter(common["confidence"], common["lift"], s=12, alpha=0.6)
plt.title("Common cross-sells: reliable versus surprising")
plt.xlabel("Confidence")
plt.ylabel("Lift")
plt.show()
No description has been provided for this image

Your turn: what do customers buy after the pink alarm clock? Use the best table.

pink = best[best["first"] == "ALARM CLOCK BAKELIKE PINK"]
for r in pink.head(3).itertuples():
    print(r.then, round(r.support, 4), round(r.confidence, 2))
ALARM CLOCK BAKELIKE RED  0.053 0.57
ALARM CLOCK BAKELIKE GREEN 0.0467 0.51
ALARM CLOCK BAKELIKE IVORY 0.0357 0.39

Longer patterns with the Apriori trick#

Problem: three-step patterns multiply the candidates again.

If <A, B, C> is frequent, <A, B> and <B, C> must be frequent too. So we only chain frequent pairs.

top_pairs = cross[cross["support"] >= 0.05]
links = set(zip(top_pairs["first"], top_pairs["then"]))
cands = sorted((a, b, c) for a, b in links for b2, c in links
               if b == b2 and c != a)
print(len(top_pairs), len(cands))
148 898
trip = pd.Series({c: support(list(c)) for c in cands})
trip = trip.sort_values(ascending=False, kind="stable")
for (a, b, c), s in trip.head(5).items():
    print(f"{a} -> {b} -> {c}: {s}")
LUNCH BAG RED RETROSPOT -> LUNCH BAG PINK POLKADOT -> LUNCH BAG  BLACK SKULL.: 0.0421
LUNCH BAG RED RETROSPOT -> LUNCH BAG CARS BLUE -> LUNCH BAG  BLACK SKULL.: 0.0392
LUNCH BAG CARS BLUE -> LUNCH BAG RED RETROSPOT -> LUNCH BAG  BLACK SKULL.: 0.0389
LUNCH BAG RED RETROSPOT -> LUNCH BAG SUKI DESIGN  -> LUNCH BAG  BLACK SKULL.: 0.0385
LUNCH BAG CARS BLUE -> LUNCH BAG RED RETROSPOT -> LUNCH BAG SUKI DESIGN : 0.0382

Quick check: can a three-step pattern ever be more common than the two-step patterns inside it?

a, b, c = trip.index[0]
print(support([a, b]), support([b, c]), support([a, b, c]))
0.0693 0.0629 0.0421

Your turn: which three-step patterns start with the Regency cakestand?

from_stand = trip[[c[0] == stand for c in trip.index]]
print(len(from_stand))
for (a, b, c), s in from_stand.head(3).items():
    print(f"{b} -> {c}: {s}")
16
PARTY BUNTING -> SPOTTY BUNTING: 0.0209
SPOTTY BUNTING -> PARTY BUNTING: 0.0194
PARTY BUNTING -> PAPER CHAIN KIT 50'S CHRISTMAS : 0.0177

Mini-project: when should the follow-up email go out?#

After a customer's first Regency cakestand, what do they buy next, and how many days later?

has_stand = inv["items"].apply(lambda s: stand in s)
first_buy = inv[has_stand].groupby("CustomerID")["date"].min()
later = inv.join(first_buy.rename("first_buy"), on="CustomerID",
                 how="inner")
later = later[later["date"] > later["first_buy"]]
print(len(first_buy), later["CustomerID"].nunique())
881 663
after = later.explode("items")
next_buys = after.groupby("items")["CustomerID"].nunique()
next_buys.sort_values(ascending=False).head(5)
items
REGENCY CAKESTAND 3 TIER              330
SET OF 3 REGENCY CAKE TINS            206
WHITE HANGING HEART T-LIGHT HOLDER    171
PARTY BUNTING                         170
ROSES REGENCY TEACUP AND SAUCER       165
Name: CustomerID, dtype: int64
def days_to(item):
    hit = after[after["items"] == item]
    first_hit = hit.groupby("CustomerID")["date"].min()
    gap = first_hit - first_buy.loc[first_hit.index]
    return gap.dt.days

reorder_gap, tins_gap = days_to(stand), days_to(tins)
print(len(reorder_gap), reorder_gap.median())
print(len(tins_gap), tins_gap.median())
330 73.0
206 138.5
print(round(len(tins_gap) / len(first_buy), 3))
print(round(item_sup[tins], 3))
0.234
0.151
plt.hist([reorder_gap, tins_gap], bins=12,
         label=["Cakestand again", "Cake tins"])
plt.title("Days from first cakestand to the next purchase")
plt.xlabel("Days after first cakestand")
plt.ylabel("Customers")
plt.legend()
plt.show()
No description has been provided for this image

Finding: half of returning cakestand buyers (330 of 663) buy another one, after a median 73 days. The cake tins are the best cross-sell (206 customers), but later: a median 138.5 days. Caveat: one year of data cuts off longer gaps.

Recap#

  • Sequences are each customer's baskets in time order; convert dates first
  • A pattern <A, B> needs B in a later basket than A
  • Support counts customers, not baskets
  • First and last positions count every pair in one pass; the Apriori trick grows longer patterns
  • Reorders and cross-sells, plus real time gaps, drive follow-up offers

Challenges#

  1. Repeat the mini-project for the white hanging heart T-light holder.
  2. Only count <A, B> when B comes within 90 days of A. Which patterns survive?
  3. Mine patterns for customers in one country, such as Germany, and compare.

What's next#

Lesson 36: Introduction to Anomaly and Outlier Detection.

Data Mining 36: Introduction to Anomaly and Outlier Detection#

In this lesson you'll be able to:

  • Explain what an anomaly is, and the kinds you'll meet
  • Flag outliers in one column with z-scores and the IQR fence
  • Score whole rows with Isolation Forest and Local Outlier Factor
  • Explain why a flagged row is unusual
  • Check whether unusual means fraud, on real card data Datasets: Online Retail customers (data/pandas/online_retail.csv.gz) and card transactions (data/mining/creditcard.csv.gz)

Warm-up: sequences from lesson 35#

Rebuild the clean sales, and count baskets per customer again.

import numpy as np
import pandas as pd
df = pd.read_csv("online_retail.csv.gz")
sales = df.dropna(subset=["CustomerID"])
sales = sales[~sales["InvoiceNo"].str.startswith("C")]
keep = (sales["Quantity"] > 0) & sales["StockCode"].str.match(r"\d")
sales = sales[keep].copy()
baskets = sales.groupby("CustomerID")["InvoiceNo"].nunique()
print(baskets.median(), baskets.max(), baskets.idxmax())
2.0 206 12748.0

Pattern mining looks for the common. Anomaly detection looks for the rare.

What is an anomaly?#

Problem: spot the rows that don't fit, before they mislead us.

  • An anomaly (or outlier) is a row unlike the rest
  • Causes: data errors, rare but real cases, or bad behaviour such as fraud
  • Kinds: point (one odd value), contextual (odd for its time or place), collective (an odd group)
df["Quantity"].describe().round(1)
count    541909.0
mean          9.6
std         218.1
min      -80995.0
25%           1.0
50%           3.0
75%          10.0
max       80995.0
Name: Quantity, dtype: float64
cols = ["InvoiceNo", "StockCode", "Quantity", "UnitPrice",
        "CustomerID"]
big = df.nlargest(2, "Quantity")
big[cols]
InvoiceNo StockCode Quantity UnitPrice CustomerID
540421 581483 23843 80995 2.08 16446.0
61619 541431 23166 74215 1.04 12346.0

Quick check: were these giant orders real sales, or something else?

same = df["StockCode"].isin(big["StockCode"])
same &= df["CustomerID"].isin(big["CustomerID"])
df.loc[same, cols + ["InvoiceDate"]]
InvoiceNo StockCode Quantity UnitPrice CustomerID InvoiceDate
61619 541431 23166 74215 1.04 12346.0 1/18/2011 10:01
61624 C541433 23166 -74215 1.04 12346.0 1/18/2011 10:17
540421 581483 23843 80995 2.08 16446.0 12/9/2011 9:15
540422 C581484 23843 -80995 2.08 16446.0 12/9/2011 9:27
price_cols = ["InvoiceNo", "StockCode", "Description", "UnitPrice"]
df.nlargest(3, "UnitPrice")[price_cols]
InvoiceNo StockCode Description UnitPrice
222681 C556445 M Manual 38970.00
524602 C580605 AMAZONFEE AMAZON FEE 17836.46
43702 C540117 AMAZONFEE AMAZON FEE 16888.02

A first rule: how far from typical?#

Problem: flag customers whose numbers sit far from the crowd.

sales["Spend"] = sales["Quantity"] * sales["UnitPrice"]
cust = sales.groupby("CustomerID").agg(
    baskets=("InvoiceNo", "nunique"), items=("Quantity", "sum"),
    spend=("Spend", "sum"), products=("StockCode", "nunique"),
    country=("Country", "first"))
cust["basket_value"] = cust["spend"] / cust["baskets"]
print(cust.shape)
cust.head(3).round(1)
(4335, 6)
baskets items spend products country basket_value
CustomerID
12346.0 1 74215 77183.6 1 United Kingdom 77183.6
12347.0 7 2458 4310.0 103 Iceland 615.7
12348.0 4 2332 1437.2 21 Finland 359.3

Common mistake: computing z-scores on the whole table, text column included

z = (cust - cust.mean()) / cust.std()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[8], line 1
----> 1 z = (cust - cust.mean()) / cust.std()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\util\_decorators.py:336, in deprecate_nonkeyword_arguments.<locals>.decorate.<locals>.wrapper(*args, **kwargs)
    330 if len(args) > num_allow_args:
    331     warnings.warn(
    332         msg.format(arguments=_format_argument_list(allow_args)),
    333         klass,
    334         stacklevel=find_stack_level(),
    335     )
--> 336 return func(*args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\frame.py:14475, in DataFrame.mean(self, axis, skipna, numeric_only, **kwargs)
  14394 @deprecate_nonkeyword_arguments(Pandas4Warning, allowed_args=["self"], name="mean")
  14395 def mean(
  14396     self,
   (...)
  14400     **kwargs,
  14401 ) -> Series | Any:
  14402     """
  14403     Return the mean of the values over the requested axis.
  14404 
   (...)
  14473     dtype: float64
  14474     """
> 14475     result = super().mean(
  14476         axis=axis, skipna=skipna, numeric_only=numeric_only, **kwargs
  14477     )
  14478     if isinstance(result, Series):
  14479         result = result.__finalize__(self, method="mean")

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\generic.py:11819, in NDFrame.mean(self, axis, skipna, numeric_only, **kwargs)
  11811 def mean(
  11812     self,
  11813     *,
   (...)
  11817     **kwargs,
  11818 ) -> Series | float:
> 11819     return self._stat_function(
  11820         "mean", nanops.nanmean, axis, skipna, numeric_only, **kwargs
  11821     )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\generic.py:11773, in NDFrame._stat_function(self, name, func, axis, skipna, numeric_only, **kwargs)
  11769 nv.validate_func(name, (), kwargs)
  11771 validate_bool_kwarg(skipna, "skipna", none_allowed=False)
> 11773 return self._reduce(
  11774     func, name=name, axis=axis, skipna=skipna, numeric_only=numeric_only
  11775 )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\frame.py:13635, in DataFrame._reduce(self, op, name, axis, skipna, numeric_only, filter_type, **kwds)
  13631     df = df.T
  13633 # After possibly _get_data and transposing, we are now in the
  13634 #  simple case where we can use BlockManager.reduce
> 13635 res = df._mgr.reduce(blk_func)
  13636 out = df._constructor_from_mgr(res, axes=res.axes).iloc[0]
  13637 out.name = None

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\managers.py:1694, in BlockManager.reduce(self, func)
   1691 # If 2D, we assume that we're operating column-wise
   1692 assert self.ndim == 2
-> 1694 res_blocks = [blk.reduce(func) for blk in self.blocks]
   1695 index = default_index(1)  # placeholder
   1696 # shallow copy self.items not needed because DataFrame._reduce does a getitem

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\blocks.py:358, in Block.reduce(self, func)
    352 @final
    353 def reduce(self, func) -> Block:
    354     # We will apply the function and reshape the result into a single-row
    355     #  Block with the same mgr_locs; squeezing will be done at a higher level
    356     assert self.ndim == 2
--> 358     result = func(self.values)
    360     if self.values.ndim == 1:
    361         res_values = result

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\frame.py:13563, in DataFrame._reduce.<locals>.blk_func(values, axis)
  13561     if not is_1d_only_ea_dtype(values.dtype):
  13562         return values._reduce(name, axis=1, skipna=skipna, **kwds)
> 13563     return values._reduce(name, skipna=skipna, keepdims=True, **kwds)
  13564 else:
  13565     return op(values, axis=axis, skipna=skipna, **kwds)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\string_arrow.py:564, in ArrowStringArray._reduce(self, name, skipna, keepdims, **kwargs)
    562     result = self._reduce_calc(name, skipna=skipna, keepdims=keepdims, **kwargs)
    563 else:
--> 564     raise TypeError(f"Cannot perform reduction '{name}' with string dtype")
    566 if name in ("argmin", "argmax") and isinstance(result, pa.Array):
    567     return self._convert_int_result(result)

TypeError: Cannot perform reduction 'mean' with string dtype
num = cust.drop(columns="country")
z = (num - num.mean()) / num.std()
print((z.abs() > 3).sum())
baskets         52
items           31
spend           33
products        62
basket_value     7
dtype: int64
s = cust["spend"]
print(round((s.loc[14646] - s.mean()) / s.std(), 1))
31.1
q1, q3 = cust["spend"].quantile([0.25, 0.75])
fence = q3 + 1.5 * (q3 - q1)
print(round(q1, 1), round(q3, 1), round(fence, 1))
print((cust["spend"] > fence).sum())
305.4 1631.5 3620.6
425
import matplotlib.pyplot as plt
cust["spend"].plot.box(logy=True)
plt.title("Spend per customer, log scale")
plt.ylabel("Total spend (pounds, log scale)")
plt.show()
No description has been provided for this image

Your turn: apply the same IQR fence to the items column. How many customers are above it?

i1, i3 = cust["items"].quantile([0.25, 0.75])
item_fence = i3 + 1.5 * (i3 - i1)
print(item_fence, (cust["items"] > item_fence).sum())
2241.25 401

Quick check: why does the fence flag over ten times as many customers as z-scores?

print(cust["spend"].agg(["mean", "median", "std"]).round(1))
mean      2021.0
median     668.1
std       8906.5
Name: spend, dtype: float64
np.log10(cust["spend"] + 1).plot.hist(bins=40)
plt.title("Customer spend is heavily skewed")
plt.xlabel("log10 of total spend + 1 (pounds)")
plt.ylabel("Customers")
plt.show()
No description has been provided for this image
log_spend = np.log1p(cust["spend"])
zl = (log_spend - log_spend.mean()) / log_spend.std()
print((zl > 3).sum(), (zl < -3).sum())
31 8

Many features at once: Isolation Forest#

Problem: a customer can look normal in each column, but odd in the combination.

  • Make random cuts through the data, like a game of twenty questions
  • Odd rows get isolated in very few cuts; ordinary rows need many
  • Average over many random trees for an anomaly score

Common mistake: handing the model the whole table, text and all

from sklearn.ensemble import IsolationForest
IsolationForest(random_state=42).fit(cust)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_25756\557455568.py in ?()
      1 from sklearn.ensemble import IsolationForest
----> 2 IsolationForest(random_state=42).fit(cust)

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py in ?(estimator, *args, **kwargs)
   1399                     )
   1400                 ),
   1401                 callback_management_context(estimator),
   1402             ):
-> 1403                 return fit_method(estimator, *args, **kwargs)

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\ensemble\_iforest.py in ?(self, X, y, sample_weight)
    316         -------
    317         self : object
    318             Fitted estimator.
    319         """
--> 320         X = validate_data(
    321             self, X, accept_sparse=["csc"], dtype=np.float32, ensure_all_finite=False
    322         )
    323 

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py in ?(_estimator, X, y, reset, validate_separately, skip_check_array, **check_params)
   3037             out = y
   3038         else:
   3039             out = X, y
   3040     elif not no_val_X and no_val_y:
-> 3041         out = check_array(X, input_name="X", **check_params)
   3042     elif no_val_X and not no_val_y:
   3043         out = _check_y(y, **check_params)
   3044     else:

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py in ?(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, ensure_all_finite, ensure_non_negative, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)
   1035                         )
   1036                     array = xp.astype(array, dtype, copy=False)
   1037                 else:
   1038                     array = _asarray_with_order(array, order=order, dtype=dtype, xp=xp)
-> 1039             except ComplexWarning as complex_warning:
   1040                 raise ValueError(
   1041                     "Complex data not supported\n{}\n".format(array)
   1042                 ) from complex_warning

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_array_api.py in ?(array, dtype, order, copy, xp, device)
    979         # Use NumPy API to support order
    980         if copy is True:
    981             array = numpy.array(array, order=order, dtype=dtype)
    982         else:
--> 983             array = numpy.asarray(array, order=order, dtype=dtype)
    984 
    985         # At this point array is a NumPy ndarray. We convert it to an array
    986         # container that is consistent with the input's namespace.

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\generic.py in ?(self, dtype, copy)
   2023             )
   2024         values = self._values
   2025         if copy is None:
   2026             # Note: branch avoids `copy=None` for NumPy 1.x support
-> 2027             arr = np.asarray(values, dtype=dtype)
   2028         else:
   2029             arr = np.array(values, dtype=dtype, copy=copy)
   2030 

ValueError: could not convert string to float: 'United Kingdom'
feats = ["baskets", "items", "spend", "products", "basket_value"]
X = np.log1p(cust[feats])
iso = IsolationForest(contamination=0.01, random_state=42)
cust["flag"] = iso.fit_predict(X) == -1
cust["score"] = -iso.score_samples(X)
print(cust["flag"].sum())
44
top = cust.nlargest(5, "score")
top[feats + ["score"]].round(2)
baskets items spend products basket_value score
CustomerID
14646.0 73 197420 279138.02 700 3823.81 0.76
14156.0 54 57872 116729.63 713 2161.66 0.74
12748.0 206 25511 32317.32 1766 156.88 0.73
14911.0 198 80429 136275.72 1785 688.26 0.73
18102.0 60 64124 259657.30 150 4327.62 0.73
cust["score"].plot.hist(bins=40)
cutoff = cust.loc[cust["flag"], "score"].min()
plt.axvline(cutoff, color="red", linestyle="--")
plt.title("Contamination sets where the line is drawn")
plt.xlabel("Anomaly score (higher = stranger)")
plt.ylabel("Customers")
plt.show()
print(round(cutoff, 3))
No description has been provided for this image
0.662
ok, odd = cust[~cust["flag"]], cust[cust["flag"]]
plt.scatter(ok["baskets"], ok["spend"] + 1, s=6, c="grey",
            alpha=0.4, label="ordinary")
plt.scatter(odd["baskets"], odd["spend"] + 1, s=20, c="red",
            label="flagged")
plt.xscale("log")
plt.yscale("log")
plt.title("Isolation Forest flags the edges of the cloud")
plt.xlabel("Baskets (log scale)")
plt.ylabel("Total spend + 1 (log scale)")
plt.legend()
plt.show()
No description has been provided for this image

Your turn: are the flagged customers mostly big spenders or tiny ones? Compare with the median spend.

median_spend = cust["spend"].median()
print(round(median_spend, 1), (odd["spend"] > median_spend).sum(),
      len(odd))
668.1 29 44

A neighbourhood view: Local Outlier Factor#

Problem: some rows are only odd compared with their nearest neighbours.

LOF compares how crowded each row's neighbourhood is with how crowded its neighbours' are.

from sklearn.neighbors import LocalOutlierFactor
from sklearn.preprocessing import StandardScaler
X_scaled = StandardScaler().fit_transform(X)
lof = LocalOutlierFactor(n_neighbors=20, contamination=0.01)
cust["lof_flag"] = lof.fit_predict(X_scaled) == -1
print(cust["lof_flag"].sum())
44
cust["lof_score"] = -lof.negative_outlier_factor_
print(cust["lof_score"].describe().round(2))
count    4335.00
mean        1.10
std         0.17
min         0.94
25%         1.00
50%         1.06
75%         1.15
max         5.37
Name: lof_score, dtype: float64

Quick check: do Isolation Forest and LOF flag the same customers?

pd.crosstab(cust["flag"], cust["lof_flag"])
lof_flag False True
flag
False 4262 29
True 29 15
lof_only = cust[cust["lof_flag"] & ~cust["flag"]]
lof_only.sort_values("spend")[feats].head().round(1)
baskets items spend products basket_value
CustomerID
17816.0 4 30 36.6 16 9.1
18037.0 6 20 70.0 19 11.7
16112.0 4 574 195.7 15 48.9
14185.0 1 6 197.6 3 197.6
17988.0 7 43 201.1 37 28.7
plt.scatter(cust["score"], cust["lof_score"], s=6, alpha=0.5)
plt.title("Two detectors, two opinions")
plt.xlabel("Isolation Forest score")
plt.ylabel("LOF score")
plt.show()
No description has been provided for this image

Your turn: rerun LOF with 50 neighbours. How many of its flags agree with Isolation Forest now?

lof50 = LocalOutlierFactor(n_neighbors=50, contamination=0.01)
flag50 = lof50.fit_predict(X_scaled) == -1
print(flag50.sum(), (flag50 & cust["flag"]).sum())
44 22

Explaining an anomaly#

Problem: a flag is useless until someone can say why the row is odd.

odd_ones = [14646, 12346, 13256]
print(cust.loc[odd_ones, "flag"].tolist())
ratio = cust.loc[odd_ones, feats] / cust[feats].median()
ratio.round(2)
[True, True, True]
baskets items spend products basket_value
CustomerID
14646.0 36.5 520.90 417.80 20.00 13.17
12346.0 0.5 195.82 115.53 0.03 265.82
13256.0 0.5 33.09 0.00 0.03 0.00
sales.loc[sales["CustomerID"] == 13256,
          ["InvoiceDate", "Description", "Quantity", "UnitPrice"]]
InvoiceDate Description Quantity UnitPrice
502122 11/25/2011 15:57 ASSTD DESIGN 3D PAPER STICKERS 12540 0.0
  • 14646: a genuine giant, a wholesale account to look after
  • 12346: a data error: an order that was cancelled, still in our clean sales
  • 13256: 12,540 items at a price of zero: a pricing or free-goods question
import os
flagged = cust[cust["flag"]].sort_values("score", ascending=False)
flagged.to_csv("m36_flagged_customers.csv")
back = pd.read_csv("m36_flagged_customers.csv")
print(back.shape, back["CustomerID"].head(3).tolist())
(44, 11) [14646.0, 14156.0, 12748.0]

Mini-project: does unusual mean fraud?#

284,807 real card transactions, 492 of them fraud. Features V1 to V28 are anonymised by the bank. Fit Isolation Forest without labels, then use the labels only to check.

cc = pd.read_csv("creditcard.csv.gz")
features = [f"V{i}" for i in range(1, 29)] + ["Amount"]
print(cc.shape)
print(cc["Class"].sum(), round(cc["Class"].mean(), 4))
(284807, 31)
492 0.0017
iso_cc = IsolationForest(random_state=42)
iso_cc.fit(cc[features])
cc["score"] = -iso_cc.score_samples(cc[features])
top = cc.nlargest(len(cc) // 100, "score")
print(len(top), top["Class"].sum(), round(top["Class"].mean(), 3))
2848 291 0.102
for share in [0.001, 0.005, 0.01, 0.05]:
    review = cc.nlargest(int(len(cc) * share), "score")
    caught = review["Class"].sum()
    print(share, len(review), caught, round(caught / 492, 2))
0.001 284 105 0.21
0.005 1424 229 0.47
0.01 2848 291 0.59
0.05 14240 414 0.84

Quick check: how many frauds would a random 1% sample of transactions catch?

random_pick = cc.sample(frac=0.01, random_state=42)
print(len(random_pick), random_pick["Class"].sum())
2848 2
big_amounts = cc.nlargest(len(cc) // 100, "Amount")
print(len(big_amounts), big_amounts["Class"].sum())
2848 9
normal, fraud = cc[cc["Class"] == 0], cc[cc["Class"] == 1]
plt.hist([normal["score"], fraud["score"]], bins=40, density=True,
         label=["genuine", "fraud"])
plt.title("Fraud scores higher, but the groups overlap")
plt.xlabel("Anomaly score (higher = stranger)")
plt.ylabel("Density")
plt.legend()
plt.show()
No description has been provided for this image

Finding: reviewing the strangest 1% (2,848 transactions) catches 291 of 492 frauds, 59%. Fraud is 10.2% of that list against 0.17% overall, but 9 in 10 flags are still genuine.

Recap#

  • An anomaly is a row unlike the rest: an error, a rare case, or fraud
  • Z-scores and the IQR fence work on one column; mind skewed data
  • Isolation Forest isolates odd rows in few random cuts
  • LOF compares a row's neighbourhood with its neighbours'
  • Unusual is not the same as bad: explain flags, and check them against known cases

Challenges#

  1. Rebuild the customer table without the two cancelled giant orders. Who is flagged now?
  2. Run LOF on the card data (use a sample of 20,000 rows, random_state=42).
  3. Flag outliers in Quantity by country. Is 1,000 items normal everywhere?

What's next#

Lesson 37: Statistical Methods for Outlier Detection.

Data Mining 37: Statistical Methods for Outlier Detection#

In this lesson you'll be able to:

  • Flag outliers with z-scores, and spot when they mislead
  • Draw IQR fences and boxplots on skewed money data
  • Build robust z-scores from the median and MAD
  • Catch outliers in combinations with Mahalanobis distance Datasets: card transactions (data/mining/creditcard.csv.gz) and New York taxi trips (data/pandas/taxis.csv)

Warm-up: rare events, from lesson 36#

Lesson 36: anomalies are rare and come in types. Today we measure 'unusual'.

import numpy as np
import pandas as pd
cards = pd.read_csv("creditcard.csv.gz")
print(cards.shape)
cards[["Time", "V1", "V2", "Amount", "Class"]].head(3).round(2)
(284807, 31)
Time V1 V2 Amount Class
0 0.0 -1.36 -0.07 149.62 0
1 0.0 1.19 0.27 2.69 0
2 1.0 -1.36 -1.34 378.66 0

V1 to V28 are anonymised PCA components; Time is seconds from the first payment.

print(cards["Class"].value_counts())
print(round(cards["Class"].mean() * 100, 3))
Class
0    284315
1       492
Name: count, dtype: int64
0.173

Z-scores: distance from the average#

Problem: is a 2,000 dollar payment unusual? We need a common yardstick.

  • z = (value - mean) / standard deviation
  • It says how many standard deviations a value sits from the mean
  • The classic rule flags anything beyond 3 either way
amount = cards["Amount"]
amount.describe().round(2)
count    284807.00
mean         88.35
std         250.12
min           0.00
25%           5.60
50%          22.00
75%          77.16
max       25691.16
Name: Amount, dtype: float64

Common mistake: forgetting the brackets after .std

z = (amount - amount.mean()) / amount.std
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[5], line 1
----> 1 z = (amount - amount.mean()) / amount.std

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\ops\common.py:85, in _unpack_zerodim_and_defer.<locals>.new_method(self, other)
     82     other = sanitize_array(other, None)
     83     other = ensure_wrapped_if_datetimelike(other)
---> 85 return method(self, other)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arraylike.py:214, in OpsMixin.__truediv__(self, other)
    212 @unpack_zerodim_and_defer("__truediv__")
    213 def __truediv__(self, other):
--> 214     return self._arith_method(other, operator.truediv)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\series.py:6751, in Series._arith_method(self, other, op)
   6749 def _arith_method(self, other, op):
   6750     self, other = self._align_for_op(other)
-> 6751     return base.IndexOpsMixin._arith_method(self, other, op)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\base.py:1644, in IndexOpsMixin._arith_method(self, other, op)
   1641     rvalues = np.arange(rvalues.start, rvalues.stop, rvalues.step)
   1643 with np.errstate(all="ignore"):
-> 1644     result = ops.arithmetic_op(lvalues, rvalues, op)
   1646 return self._construct_result(result, name=res_name, other=other)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\ops\array_ops.py:289, in arithmetic_op(left, right, op)
    285     _bool_arith_check(op, left, right)  # type: ignore[arg-type]
    287     # error: Argument 1 to "_na_arithmetic_op" has incompatible type
    288     # "Union[ExtensionArray, ndarray[Any, Any]]"; expected "ndarray[Any, Any]"
--> 289     res_values = _na_arithmetic_op(left, right, op)  # type: ignore[arg-type]
    291 return res_values

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\ops\array_ops.py:220, in _na_arithmetic_op(left, right, op, is_cmp)
    217     func = partial(expressions.evaluate, op)
    219 try:
--> 220     result = func(left, right)
    221 except TypeError:
    222     if not is_cmp and (
    223         left.dtype == object or getattr(right, "dtype", None) == object
    224     ):
   (...)
    227         # Don't do this for comparisons, as that will handle complex numbers
    228         #  incorrectly, see GH#32047

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\computation\expressions.py:252, in evaluate(op, left_op, right_op, use_numexpr)
    249 if op_str is not None:
    250     if use_numexpr:
    251         # error: "None" not callable
--> 252         return _evaluate(op, op_str, left_op, right_op)  # type: ignore[misc]
    253 return _evaluate_standard(op, op_str, left_op, right_op)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\computation\expressions.py:83, in _evaluate_standard(op, op_str, left_op, right_op)
     81 if _TEST_MODE:
     82     _store_test_result(False)
---> 83 return op(left_op, right_op)

TypeError: unsupported operand type(s) for /: 'float' and 'method'
z = (amount - amount.mean()) / amount.std()
flagged = z.abs() > 3
print(flagged.sum())
print(round(amount[flagged].min(), 2))
4076
838.82

Quick check: are these big payments more likely to be fraud?

rate_all = cards["Class"].mean() * 100
rate_flag = cards.loc[flagged, "Class"].mean() * 100
print(round(rate_all, 3), round(rate_flag, 3))
cards.groupby("Class")["Amount"].median()
0.173 0.27
Class
0    22.00
1     9.25
Name: Amount, dtype: float64

Why skewed data fools the z-score#

Problem: a few huge payments drag the mean and standard deviation with them.

print(round(amount.skew(), 1))
amount.nlargest(5)
17.0
274771    25691.16
58465     19656.53
151296    18910.00
46841     12910.93
54018     11898.09
Name: Amount, dtype: float64
trimmed = amount[amount < amount.quantile(0.999)]
print(round(amount.std(), 2), round(trimmed.std(), 2))
250.12 193.77

Money is skewed: most payments are small, a few are huge. A log scale evens it out.

log_amt = np.log1p(amount)
print(round(log_amt.skew(), 2))
0.16
import matplotlib.pyplot as plt
ax = log_amt.plot.hist(bins=60, figsize=(7, 4))
ax.set_title("Card payments on a log scale")
ax.set_xlabel("log(1 + amount)")
ax.set_ylabel("Number of payments")
plt.show()
No description has been provided for this image

Your turn: z-scores on log_amt. How many are flagged high, how many low?

z_log = (log_amt - log_amt.mean()) / log_amt.std()
print((z_log > 3).sum(), (z_log < -3).sum())
line = np.expm1(log_amt.mean() + 3 * log_amt.std())
print(round(line, 2))
218 0
3367.2

IQR fences and the boxplot#

Problem: we want a rule built from quartiles, which extremes can't drag.

  • IQR = Q3 - Q1, the spread of the middle half
  • Tukey's fences: below Q1 - 1.5 IQR or above Q3 + 1.5 IQR
q1, q3 = amount.quantile([0.25, 0.75])
iqr = q3 - q1
upper = q3 + 1.5 * iqr
print(round(q1, 2), round(q3, 2), round(upper, 2))
5.6 77.16 184.51
over = amount > upper
print(over.sum(), round(over.mean() * 100, 1))
31904 11.2

Quick check: fences on log_amt instead. More flags, or fewer?

lq1, lq3 = log_amt.quantile([0.25, 0.75])
l_upper = lq3 + 1.5 * (lq3 - lq1)
l_over = log_amt > l_upper
print(l_over.sum(), round(np.expm1(l_upper), 2))
245 3184.77
import seaborn as sns
kind = cards["Class"].map({0: "normal", 1: "fraud"})
ax = sns.boxplot(x=log_amt, y=kind)
ax.set_title("Payment size by class (log scale)")
ax.set_xlabel("log(1 + amount)")
ax.set_ylabel("")
plt.show()
No description has been provided for this image

Robust z-scores with the median and MAD#

Problem: mean and standard deviation bend towards outliers. The median and MAD barely move.

  • MAD = median of |value - median|
  • robust z = 0.6745 * (value - median) / MAD, flag beyond 3.5

Quick check: drop the top 0.1% of payments. Which measures move?

mad_all = (amount - amount.median()).abs().median()
mad_trim = (trimmed - trimmed.median()).abs().median()
print(round(amount.mean(), 2), round(trimmed.mean(), 2))
print(amount.median(), trimmed.median())
print(round(mad_all, 2), round(mad_trim, 2))
88.35 83.89
22.0 21.99
20.22 20.21

Common mistake: calling .mad() from an old tutorial

amount.mad()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_3260\1445532913.py in ?()
----> 1 amount.mad()

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\generic.py in ?(self, name)
   6190             and name not in self._accessors
   6191             and self._info_axis._can_hold_identifiers_and_holds_name(name)
   6192         ):
   6193             return self[name]
-> 6194         return object.__getattribute__(self, name)

AttributeError: 'Series' object has no attribute 'mad'
def robust_z(s):
    med = s.median()
    mad = (s - med).abs().median()
    return 0.6745 * (s - med) / mad

print((robust_z(amount).abs() > 3.5).sum())
46571
rz_log = robust_z(log_amt)
big = rz_log.abs() > 3.5
print(big.sum())
amount[big]
3
58465     19656.53
151296    18910.00
274771    25691.16
Name: Amount, dtype: float64
summary = pd.Series({
    "z-score": flagged.sum(), "z-score on log": (z_log > 3).sum(),
    "IQR fences": over.sum(), "IQR on log": l_over.sum(),
    "robust z on log": big.sum()})
summary.sort_values()
robust z on log        3
z-score on log       218
IQR on log           245
z-score             4076
IQR fences         31904
dtype: int64

Which column hides the fraud?#

Problem: big amounts were not the fraud signal. Maybe another column is.

Choose on day 1 (the first 24 hours), keep day 2 untouched for the mini-project.

day1 = cards[cards["Time"] < 86400]
day2 = cards[cards["Time"] >= 86400]
print(len(day1), day1["Class"].sum())
print(len(day2), day2["Class"].sum())
144786 281
140021 211
v_cols = [f"V{i}" for i in range(1, 29)]
flags = day1[v_cols].apply(robust_z).abs() > 3.5
is_fraud = day1["Class"] == 1
report = pd.DataFrame({"flagged": flags.sum(),
                       "frauds": flags[is_fraud].sum()})
report["fraud_rate"] = report["frauds"] / report["flagged"]
report.sort_values("frauds", ascending=False).head().round(3)
flagged frauds fraud_rate
V14 3713 243 0.065
V17 2497 231 0.093
V12 3995 223 0.056
V10 3424 222 0.065
V27 14249 198 0.014

Quick check: how many times richer in fraud is V14's flag list than day 1 overall?

base1 = day1["Class"].mean()
print(round(report.loc["V14", "fraud_rate"] / base1, 1))
33.7
ax = day1.loc[~is_fraud, "V14"].plot.hist(bins=80, density=True,
                                          alpha=0.6, label="normal")
day1.loc[is_fraud, "V14"].plot.hist(bins=40, density=True, ax=ax,
                                    alpha=0.6, label="fraud")
ax.set_title("V14 on day 1: frauds sit far to the left")
ax.set_xlabel("V14 (anonymised component)")
ax.set_ylabel("Density")
ax.legend()
plt.show()
No description has been provided for this image

Mahalanobis distance: odd combinations#

Problem: some records look normal in each column, but odd as a pair.

taxis = pd.read_csv("taxis.csv")
trips = taxis[["distance", "fare"]]
print(len(trips))
print(trips.corr().round(2))
6433
          distance  fare
distance      1.00  0.92
fare          0.92  1.00
  • Mahalanobis distance measures distance from the centre in units that follow the correlation
  • Squared distances follow a chi-square curve, so we can set a 99.9% cut-off
X = trips.to_numpy()
centre = X.mean(axis=0)
cov_inv = np.linalg.inv(np.cov(X, rowvar=False))
diff = X - centre
print(centre.round(2))
[ 3.02 13.09]

Common mistake: multiplying the whole table at once with @

d2 = diff @ cov_inv @ diff
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[28], line 1
----> 1 d2 = diff @ cov_inv @ diff

ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 6433 is different from 2)
d2 = (diff @ cov_inv * diff).sum(axis=1)
taxis["d2"] = d2
from scipy.stats import chi2
cutoff = chi2.ppf(0.999, df=2)
joint = taxis["d2"] > cutoff
print(round(cutoff, 2), joint.sum())
13.82 158
z_trip = (trips - trips.mean()) / trips.std()
single = (z_trip.abs() > 3).any(axis=1)
hidden = joint & ~single
print(hidden.sum())
taxis.loc[hidden, ["distance", "fare", "pickup_zone"]]
6
distance fare pickup_zone
770 9.4 2.5 JFK Airport
970 0.0 45.0 NaN
4118 0.0 25.0 NaN
6089 10.1 12.0 Murray Hill-Queens
6118 0.0 25.0 Prospect Heights
6314 0.0 25.0 Jamaica
ax = taxis.plot.scatter(x="distance", y="fare", s=6, alpha=0.3,
                        color="grey", figsize=(7, 5))
taxis[joint].plot.scatter(x="distance", y="fare", s=14, ax=ax,
                         color="red", label="Mahalanobis flag")
ax.set_title("Taxi trips: fare against distance")
ax.set_xlabel("Distance (miles)")
ax.set_ylabel("Fare (dollars)")
plt.show()
No description has been provided for this image

Your turn: list the three trips with the largest Mahalanobis distance.

cols = ["distance", "fare", "total", "d2"]
taxis.nlargest(3, "d2")[cols].round(1)
distance fare total d2
622 0.0 120.0 166.0 650.1
3644 0.0 103.0 124.0 472.7
42 0.0 80.0 100.4 277.7

Mini-project: a statistical fraud alarm for day 2#

Fit Mahalanobis on day 1's four best columns. How much of day 2's fraud does it catch?

watch = ["V14", "V17", "V12", "V10"]
ref = day1[watch].to_numpy()
c_mean = ref.mean(axis=0)
c_inv = np.linalg.inv(np.cov(ref, rowvar=False))
gap = day2[watch].to_numpy() - c_mean
scores = (gap @ c_inv * gap).sum(axis=1)
print(len(scores), round(np.median(scores), 2))
140021 2.18
alarm = scores > chi2.ppf(0.999, df=4)
caught = day2.loc[alarm, "Class"].sum()
print(alarm.sum(), caught, day2["Class"].sum())
print(round(caught / alarm.sum() * 100, 1))
2949 180 211
6.1

A fraud team can only review so many cases. What if they check just the top 500?

top500 = np.argsort(-scores)[:500]
in_top = day2["Class"].iloc[top500].sum()
print(in_top, round(in_top / 500 * 100, 1))
146 29.2
recall = caught / day2["Class"].sum()
lift = (caught / alarm.sum()) / day2["Class"].mean()
print(f"Alarm catches {recall:.0%} of day-2 fraud; "
      f"alerts are {lift:.0f}x richer in fraud than average")
Alarm catches 85% of day-2 fraud; alerts are 41x richer in fraud than average

Recap#

  • z-scores measure distance from the mean in standard deviations
  • Skewed data inflates the mean and std: take logs first
  • IQR fences and robust z-scores (median, MAD) resist the extremes
  • Unusual is not the same as fraud: check flags against outcomes
  • Mahalanobis distance finds odd combinations, with a chi-square cut-off

Challenges#

  1. Use robust z-scores on log taxi fares. Which trips get flagged?
  2. Rebuild the day-2 alarm with only V14 and V17. What changes?
  3. Try a 99% chi-square cut-off instead of 99.9%. Count the extra alerts.

What's next#

Lesson 38: Distance and Density-Based Outlier Detection No bell curves needed: kNN distances, LOF and DBSCAN.

Data Mining 38: Distance and Density-Based Outlier Detection#

In this lesson you'll be able to:

  • Score outliers by the distance to their k nearest neighbours
  • Explain why scaling and the choice of k change the answer
  • Use the Local Outlier Factor, including on brand-new rows
  • Find noise points with DBSCAN and compare detectors fairly Dataset: card transactions (data/mining/creditcard.csv.gz), a 20,000-row random sample

Warm-up: lesson 37's robust z-score on V14#

Lesson 37 flagged values far from the median. One column at a time, though.

import numpy as np
import pandas as pd
cards = pd.read_csv("creditcard.csv.gz")
sample = cards.sample(20000, random_state=42)
print(len(sample), sample["Class"].sum())
20000 34
v14 = sample["V14"]
med = v14.median()
mad = (v14 - med).abs().median()
rz = 0.6745 * (v14 - med) / mad
flag = rz.abs() > 3.5
print(flag.sum(), sample.loc[flag, "Class"].sum())
452 29

The intuition: outliers have no close neighbours#

Problem: an odd record can look fine in each column, but sit alone in the full space.

import matplotlib.pyplot as plt
normal = sample[sample["Class"] == 0]
fraud = sample[sample["Class"] == 1]
ax = normal.plot.scatter(x="V14", y="V17", s=4, alpha=0.3,
                         color="grey", label="normal")
fraud.plot.scatter(x="V14", y="V17", s=20, color="red", ax=ax,
                   label="fraud")
ax.set_title("Frauds sit away from the crowd")
ax.set_xlabel("V14")
ax.set_ylabel("V17")
plt.show()
No description has been provided for this image
  • For each point, find its k nearest neighbours
  • Its outlier score is the distance to the k-th one
  • Crowded points score low; lonely points score high

kNN distance scores#

Problem: turn 'far from everyone' into one number per payment.

raw_cols = [f"V{i}" for i in range(1, 29)] + ["Amount"]
X_raw = sample[raw_cols].to_numpy()
y = sample["Class"].to_numpy()
print(X_raw.shape)
(20000, 29)

Common mistake: asking for neighbours before fitting

from sklearn.neighbors import NearestNeighbors
nn = NearestNeighbors(n_neighbors=6)
dist, _ = nn.kneighbors(X_raw)
---------------------------------------------------------------------------
NotFittedError                            Traceback (most recent call last)
Cell In[6], line 3
      1 from sklearn.neighbors import NearestNeighbors
      2 nn = NearestNeighbors(n_neighbors=6)
----> 3 dist, _ = nn.kneighbors(X_raw)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\neighbors\_base.py:808, in KNeighborsMixin.kneighbors(self, X, n_neighbors, return_distance)
    756 def kneighbors(self, X=None, n_neighbors=None, return_distance=True):
    757     """Find the K-neighbors of a point.
    758 
    759     Returns indices of and distances to the neighbors of each point.
   (...)
    806            [2]]...)
    807     """
--> 808     check_is_fitted(self)
    810     if n_neighbors is None:
    811         n_neighbors = self.n_neighbors

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:1721, in check_is_fitted(estimator, attributes, msg, all_or_any)
   1718     return
   1720 if not _is_fitted(estimator, attributes, all_or_any):
-> 1721     raise NotFittedError(msg % {"name": type(estimator).__name__})

NotFittedError: This NearestNeighbors instance is not fitted yet. Call 'fit' with appropriate arguments before using this estimator.
nn.fit(X_raw)
dist, _ = nn.kneighbors(X_raw)
print(dist[:3].round(2))
[[ 0.   48.02 48.58 48.64 48.68 48.84]
 [ 0.    4.96  6.22  6.63  6.64  6.65]
 [ 0.    1.13  1.81  2.24  2.91  3.08]]
knn_raw = dist[:, -1]
top = np.argsort(-knn_raw)[:200]
print(y[top].sum())
print(sample["Amount"].iloc[top[:5]].round(2).to_list())
14
[5239.5, 4901.0, 4275.58, 4633.95, 4338.25]

Quick check: are those five biggest scorers frauds?

print(y[top[:5]])
[0 0 0 0 0]

Scale first: distances need fair units#

Problem: Amount is in dollars, V columns are near 1, so Amount drowns them out.

sample[["V1", "V14", "V28", "Amount"]].std().round(2)
V1          2.00
V14         0.96
V28         0.34
Amount    227.97
dtype: float64
from sklearn.preprocessing import StandardScaler
feats = sample[raw_cols].copy()
feats["Amount"] = np.log1p(feats["Amount"])
X = StandardScaler().fit_transform(feats)
print(X.std(axis=0).round(2)[[0, 13, 28]])
[1. 1. 1.]
def knn_score(X, k):
    nn = NearestNeighbors(n_neighbors=k + 1).fit(X)
    return nn.kneighbors(X)[0][:, -1]

def hits(score, n=200):
    return int(y[np.argsort(-score)[:n]].sum())
knn5 = knn_score(X, 5)
print(hits(knn_raw), hits(knn5))
14 21
pd.Series(knn5).groupby(y).median().round(2)
0    2.16
1    8.44
dtype: float64

Quick check: can we rebuild one fraud's score by hand?

i = np.where(y == 1)[0][0]
gaps = np.linalg.norm(X - X[i], axis=1)
print(np.sort(gaps)[:6].round(2))
print(round(knn5[i], 2))
[ 0.    8.96 11.48 12.65 14.36 17.52]
17.52

Your turn: try k = 20. More frauds in the top 200, or fewer?

knn20 = knn_score(X, 20)
print(hits(knn20), hits(knn20, 100))
25 20

Your turn: scale, but skip the log on Amount. How many frauds in the top 200?

X_nolog = StandardScaler().fit_transform(sample[raw_cols])
print(hits(knn_score(X_nolog, 20)), hits(knn20))
25 25

Local Outlier Factor: density compared with neighbours#

Problem: a fixed distance can't serve dense and sparse regions alike.

  • LOF compares a point's local density with its neighbours' density
  • Around 1: as dense as its neighbours. Well above 1: an outlier
from sklearn.neighbors import LocalOutlierFactor
lof = LocalOutlierFactor(n_neighbors=20)
labels = lof.fit_predict(X)
lof20 = -lof.negative_outlier_factor_
print((labels == -1).sum(), y[labels == -1].sum())
print(hits(lof20))
1091 18
8

Quick check: why might LOF with 20 neighbours miss frauds?

_, idx = NearestNeighbors(n_neighbors=11).fit(X).kneighbors(X)
fraud_rows = np.where(y == 1)[0]
share = y[idx[fraud_rows, 1:]].mean()
print(round(share, 2), round(y.mean(), 4))
0.58 0.0017
lof = LocalOutlierFactor(n_neighbors=50)
lof.fit(X)
lof50 = -lof.negative_outlier_factor_
print(hits(lof50), hits(lof50, 100))
26 25
print(pd.Series(lof50).groupby(y).median().round(2))
high = lof50 > 1.5
print(high.sum(), y[high].sum())
0    1.10
1    3.31
dtype: float64
968 30
ax = pd.Series(lof50).groupby(y).plot.hist(
    bins=np.geomspace(0.9, 10, 40), alpha=0.6, density=True,
    legend=True, figsize=(7, 4))[0]
ax.set_xscale("log")
ax.set_title("LOF scores: 0 = normal, 1 = fraud")
ax.set_xlabel("LOF score (log scale)")
ax.set_ylabel("Density")
plt.show()
No description has been provided for this image

Common mistake: calling predict on a plain LOF to score new payments

lof.predict(X[:3])
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_available_if.py:32, in _AvailableIfDescriptor._check(self, obj, owner)
     31 try:
---> 32     check_result = self.check(obj)
     33 except Exception as e:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\neighbors\_lof.py:342, in LocalOutlierFactor._check_novelty_predict(self)
    336     msg = (
    337         "predict is not available when novelty=False, use "
    338         "fit_predict if you want to predict on training data. Use "
    339         "novelty=True if you want to use LOF for novelty detection "
    340         "and predict on new unseen data."
    341     )
--> 342     raise AttributeError(msg)
    343 return True

AttributeError: predict is not available when novelty=False, use fit_predict if you want to predict on training data. Use novelty=True if you want to use LOF for novelty detection and predict on new unseen data.

The above exception was the direct cause of the following exception:

AttributeError                            Traceback (most recent call last)
Cell In[23], line 1
----> 1 lof.predict(X[:3])

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_available_if.py:43, in _AvailableIfDescriptor.__get__(self, obj, owner)
     39 def __get__(self, obj, owner=None):
     40     if obj is not None:
     41         # delegate only on instances, not the classes.
     42         # this is to allow access to the docstrings.
---> 43         self._check(obj, owner=owner)
     44         out = MethodType(self.fn, obj)
     46     else:
     47         # This makes it possible to use the decorated method as an unbound method,
     48         # for instance when monkeypatching.

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_available_if.py:34, in _AvailableIfDescriptor._check(self, obj, owner)
     32     check_result = self.check(obj)
     33 except Exception as e:
---> 34     raise AttributeError(attr_err_msg) from e
     36 if not check_result:
     37     raise AttributeError(attr_err_msg)

AttributeError: This 'LocalOutlierFactor' has no attribute 'predict'

Fix: novelty=True. Score real rows held back from the sample.

rest = cards.drop(sample.index)
new = rest.groupby("Class").head(3)
new_feats = new[raw_cols].copy()
new_feats["Amount"] = np.log1p(new_feats["Amount"])
print(new["Class"].to_list())
[0, 0, 0, 1, 1, 1]
scaler = StandardScaler().fit(feats)
lof_new = LocalOutlierFactor(n_neighbors=50, novelty=True)
lof_new.fit(scaler.transform(feats))
X_new = scaler.transform(new_feats)
print(lof_new.predict(X_new))
print((-lof_new.score_samples(X_new)).round(2))
[ 1  1  1 -1  1 -1]
[1.01 1.24 1.06 2.92 1.47 2.49]

DBSCAN: points that belong to no crowd#

Problem: we want a yes or no flag, with no need to choose how many to flag.

  • DBSCAN grows clusters from dense cores (lesson 30)
  • Points within eps of no core are labelled -1: noise
d10 = np.sort(knn_score(X, 10))
ax = pd.Series(d10).plot(figsize=(7, 4))
ax.set_title("Distance to the 10th neighbour, sorted")
ax.set_xlabel("Payments, sorted")
ax.set_ylabel("Distance")
plt.show()
print(np.percentile(d10, [50, 95, 99]).round(2))
No description has been provided for this image
[2.42 5.1  8.69]
from sklearn.cluster import DBSCAN
db = DBSCAN(eps=5, min_samples=10).fit(X)
noise = db.labels_ == -1
print(db.labels_.max() + 1, noise.sum(), y[noise].sum())
1 610 29

Your turn: keep eps = 5, but require 30 neighbours for a core point.

db30 = DBSCAN(eps=5, min_samples=30).fit(X)
noise30 = db30.labels_ == -1
print(noise30.sum(), y[noise30].sum())
719 29
knn_top = np.argsort(-knn20)[:noise.sum()]
shared = np.isin(np.where(noise)[0], knn_top).sum()
print(noise.sum(), shared, y[knn_top].sum())
610 548 28

Quick check: what share of DBSCAN's noise points are fraud?

print(round(y[noise].mean() * 100, 1), round(y.mean() * 100, 2))
4.8 0.17

Comparing detectors on a fair alert budget#

Problem: each method flags a different number of rows. Fix the budget, then compare.

board = pd.Series({
    "robust z on V14": hits(rz.abs().to_numpy()),
    "kNN raw, k=5": hits(knn_raw), "kNN scaled, k=20": hits(knn20),
    "LOF k=20": hits(lof20), "LOF k=50": hits(lof50)})
board.sort_values(ascending=False)
robust z on V14     29
LOF k=50            26
kNN scaled, k=20    25
kNN raw, k=5        14
LOF k=20             8
dtype: int64
ax = board.sort_values().plot(kind="barh", figsize=(7, 4))
ax.set_title("Frauds found in a 200-payment review list")
ax.set_xlabel("Frauds in the top 200 (of 34)")
ax.set_ylabel("")
plt.show()
No description has been provided for this image
(board / 200 * 100).round(1).sort_values(ascending=False)
robust z on V14     14.5
LOF k=50            13.0
kNN scaled, k=20    12.5
kNN raw, k=5         7.0
LOF k=20             4.0
dtype: float64

Mini-project: do the detectors agree?#

Combine kNN and LOF by rank. Does the blend beat each one on a 100-payment budget?

top_knn = set(np.argsort(-knn20)[:100])
top_lof = set(np.argsort(-lof50)[:100])
both = top_knn & top_lof
print(len(both), int(y[list(both)].sum()))
72 20
ranks = pd.DataFrame({"knn": knn20, "lof": lof50}).rank(pct=True)
blend = ranks.mean(axis=1).to_numpy()
print(hits(knn20, 100), hits(lof50, 100), hits(blend, 100))
20 25 24
review = sample.iloc[np.argsort(-blend)[:100]]
review.groupby("Class")["Amount"].agg(["count", "median"]).round(2)
count median
Class
0 76 154.11
1 24 39.80

A fraud team asks: how many reviews to catch half the frauds? Three quarters?

caught = np.cumsum(y[np.argsort(-blend)])
half = np.argmax(caught >= y.sum() / 2) + 1
three_q = np.argmax(caught >= y.sum() * 0.75) + 1
print(half, three_q)
46 105
ax = pd.Series(caught[:1000]).plot(figsize=(7, 4))
ax.axhline(y.sum(), color="red", linestyle="--")
ax.set_title("Frauds caught as the team works down the list")
ax.set_xlabel("Payments reviewed")
ax.set_ylabel("Frauds caught")
plt.show()
No description has been provided for this image
found = hits(blend, 100)
random_hits = 100 * y.mean()
print(f"Blend: {found} of {y.sum()} frauds in 100 reviews; "
      f"random picks would find {random_hits:.1f}")
Blend: 24 of 34 frauds in 100 reviews; random picks would find 0.2

Recap#

  • kNN distance: lonely points have far-away neighbours
  • Scale (and log skewed columns) before any distance method
  • LOF compares local densities; k must outgrow small fraud clusters
  • Use novelty=True to score new rows; DBSCAN labels noise as -1
  • Compare detectors on the same alert budget

Challenges#

  1. Run LOF with k = 100. Does it beat k = 50?
  2. Try DBSCAN with eps = 4 and eps = 6. Count noise and frauds.
  3. Drop Amount from the features. Which detector suffers most?

What's next#

Lesson 39: Applications in Fraud Detection and Cybersecurity Rules, scores and review queues, plus spotting phishing text.

Data Mining 39: Applications in Fraud Detection and Cybersecurity#

In this lesson you'll be able to:

  • Turn domain knowledge into simple rules, and measure their lift
  • Score payments with an Isolation Forest, trained without labels
  • Size a review queue by budget, in frauds and in dollars
  • Catch phishing texts with red-flag rules and novelty detection Datasets: card transactions (data/mining/creditcard.csv.gz) and SMS messages (data/mining/sms_spam.csv)

Warm-up: from lesson 38's review lists#

Lesson 38 ranked payments by loneliness and counted frauds in a top-200 list.

import numpy as np
import pandas as pd
cards = pd.read_csv("creditcard.csv.gz")
day1 = cards[cards["Time"] < 86400]
day2 = cards[cards["Time"] >= 86400]
print(day1["Class"].sum(), day2["Class"].sum())
281 211
  • Signals: rules from experience, and scores from models
  • A queue: alerts ranked so analysts review the worst first
  • A budget: how many alerts the team can check each day

Rules first: when does fraud happen?#

Problem: fraud teams start with rules they can explain. Is any hour riskier?

cards["hour"] = (cards["Time"] // 3600) % 24
by_hour = cards.groupby("hour")["Class"].agg(["mean", "count"])
by_hour["mean"] = by_hour["mean"] * 100
by_hour.sort_values("mean", ascending=False).head().round(2)
mean count
hour
2.0 1.71 3328
4.0 1.04 2209
3.0 0.49 3492
5.0 0.37 2990
7.0 0.32 7243

Honest note: hour 0 is when recording began, not midnight. Quiet hours are likely night.

import matplotlib.pyplot as plt
ax = by_hour["mean"].plot(kind="bar", figsize=(8, 4))
ax.set_title("Fraud rate by hour since recording began")
ax.set_xlabel("Hour")
ax.set_ylabel("Fraud rate (%)")
plt.show()
No description has been provided for this image

Common mistake: combining two conditions with and

quiet = cards[(cards["hour"] >= 1) and (cards["hour"] <= 5)]
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_25840\3380649061.py in ?()
----> 1 quiet = cards[(cards["hour"] >= 1) and (cards["hour"] <= 5)]

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\generic.py in ?(self)
   1499     @final
   1500     def __bool__(self) -> NoReturn:
-> 1501         raise ValueError(
   1502             f"The truth value of a {type(self).__name__} is ambiguous. "
   1503             "Use a.empty, a.bool(), a.item(), a.any() or a.all()."
   1504         )

ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
quiet = cards["hour"].between(1, 5)
rate_q = cards.loc[quiet, "Class"].mean()
lift = rate_q / cards["Class"].mean()
print(quiet.sum(), cards.loc[quiet, "Class"].sum())
print(round(rate_q * 100, 2), round(lift, 1))
16239 118
0.73 4.2

Quick check: what share of all 492 frauds does the quiet-hours rule catch?

share = cards.loc[quiet, "Class"].sum() / cards["Class"].sum()
print(round(share * 100, 1))
24.0

Your turn: what share of ALL payments does the rule flag?

print(round(quiet.mean() * 100, 1))
5.7

Isolation Forest: anomaly scores without labels#

Problem: labels arrive late, and new fraud looks like nothing seen before.

  • Grow random trees that split on random columns at random values
  • Odd points get isolated in very few splits; ordinary points need many
  • Short average path = high anomaly score
from sklearn.ensemble import IsolationForest
feats = [f"V{i}" for i in range(1, 29)] + ["Amount"]
iso = IsolationForest(n_estimators=200, random_state=42)
iso.fit(day1[feats])
score = -iso.score_samples(day2[feats])
print(len(score), score.min().round(3), score.max().round(3))
140021 0.347 0.787
y2 = day2["Class"].to_numpy()
order = np.argsort(-score)
print(y2[order[:100]].sum())
26
cols = ["Amount", "V14", "V17", "Class"]
day2.iloc[order[:5]][cols].round(2)
Amount V14 V17 Class
274771 25691.16 -6.80 -7.17 0
173353 4861.64 -3.66 -2.57 0
220090 5964.95 -3.11 -2.49 0
173054 4543.64 -4.05 -2.50 0
176335 1676.60 -2.46 -1.78 0
ax = pd.Series(score[y2 == 0]).plot.hist(bins=60, density=True,
                                        alpha=0.6, label="normal")
pd.Series(score[y2 == 1]).plot.hist(bins=30, density=True, ax=ax,
                                    alpha=0.6, label="fraud")
ax.set_title("Isolation Forest scores on day 2")
ax.set_xlabel("Anomaly score (higher = stranger)")
ax.set_ylabel("Density")
ax.legend()
plt.show()
No description has been provided for this image
print(np.median(score[y2 == 0]).round(3))
print(np.median(score[y2 == 1]).round(3))
0.407
0.59

Quick check: the forest also has predict. How many payments does it flag by default?

flag_iso = iso.predict(day2[feats]) == -1
print(flag_iso.sum(), y2[flag_iso].sum())
6445 177

Your turn: how many frauds are in the top 1,000? How does that compare with random?

print(y2[order[:1000]].sum())
print(round(1000 * y2.mean(), 1))
77
1.5

The review queue: budgets, frauds and dollars#

Problem: managers ask 'if we fund N reviews a day, what do we catch?'

amt2 = day2["Amount"].to_numpy()
fraud_dollars = amt2[y2 == 1].sum()
print(y2.sum(), round(fraud_dollars, 2))
211 26888.86
rows = []
for budget in [100, 500, 1000, 2000]:
    picked = order[:budget]
    hit = y2[picked] == 1
    rows.append({"budget": budget, "frauds": hit.sum(),
                 "dollars": amt2[picked][hit].sum()})
queue = pd.DataFrame(rows)
queue["hit_rate"] = queue["frauds"] / queue["budget"]
queue["dollar_share"] = queue["dollars"] / fraud_dollars
queue.round(2)
budget frauds dollars hit_rate dollar_share
0 100 26 934.01 0.26 0.03
1 500 55 3248.72 0.11 0.12
2 1000 77 6328.96 0.08 0.24
3 2000 124 13853.95 0.06 0.52

Your turn: how many reviews protect half of the fraud dollars?

saved = np.cumsum(amt2[order] * y2[order])
print(np.argmax(saved >= fraud_dollars / 2) + 1)
1714

Quick check: going from 1,000 to 2,000 reviews, what happens to the hit rate?

extra = queue.set_index("budget").loc[[1000, 2000], "frauds"]
print(extra.diff().iloc[-1], round(extra.diff().iloc[-1] / 1000, 3))
47.0 0.047
ax = queue.plot(x="budget", y="dollar_share", marker="o",
                legend=False, figsize=(7, 4))
ax.set_title("Fraud dollars protected by review budget")
ax.set_xlabel("Reviews per day")
ax.set_ylabel("Share of fraud dollars caught")
plt.show()
No description has been provided for this image
night2 = cards.loc[day2.index, "hour"].between(1, 5).to_numpy()
top1000 = order[:1000]
print(night2[top1000].sum(), y2[top1000][night2[top1000]].sum())
114 54

Quick check: inside the top 1,000, are quiet-hour alerts more often fraud?

in_q = night2[top1000]
print(round(y2[top1000][in_q].mean(), 2))
print(round(y2[top1000][~in_q].mean(), 2))
0.47
0.03

Cybersecurity: red flags in phishing texts#

Problem: scam texts push you to call a number or tap a link. Can those red flags catch them?

sms = pd.read_csv("sms_spam.csv")
sms["spam"] = (sms["label"] == "spam").astype(int)
print(sms["label"].value_counts())
print(sms.loc[sms["spam"] == 1, "message"].iloc[0][:75])
label
ham     4825
spam     747
Name: count, dtype: int64
Free entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA
text = sms["message"]
sms["phone"] = text.str.contains(r"\d{5,}").astype(int)
link = r"http|www\.|\.com|\.co\.uk"
sms["link"] = text.str.contains(link, case=False).astype(int)
money = r"cash|prize|claim|\bwon\b"
sms["money"] = text.str.contains(money, case=False).astype(int)
flags = ["phone", "link", "money"]
report = pd.DataFrame({
    "flagged": sms[flags].sum(),
    "spam_caught": sms.loc[sms["spam"] == 1, flags].sum()})
report["spam_rate"] = report["spam_caught"] / report["flagged"]
report.round(3)
flagged spam_caught spam_rate
phone 588 585 0.995
link 152 136 0.895
money 243 211 0.868
fp = sms[(sms["phone"] == 1) & (sms["spam"] == 0)]
print(len(fp))
fp["message"].str[:60].to_list()
3
['MY NO. IN LUTON 0125698789 RING ME IF UR AROUND! H*',
 'Yun ah.the ubi one say if ü wan call by tomorrow.call 674412',
 'Leave it wif me lar... Ü wan to carry meh so heavy... Is da ']

Quick check: do the phone and link flags fire on the same messages?

pd.crosstab(sms["phone"], sms["link"])
link 0 1
phone
0 4927 57
1 493 95

Your turn: flag a message if it has a phone number OR a link. How much spam is caught?

either = (sms["phone"] == 1) | (sms["link"] == 1)
caught = sms.loc[either, "spam"].sum()
print(either.sum(), caught, sms["spam"].sum())
645 626 747

Novelty detection: learn normal, flag the rest#

Problem: attackers change wording. Learn only from genuine messages, so new scams still stand out.

sms["length"] = text.str.len()
sms["digits"] = text.str.count(r"\d")
sms["capitals"] = text.str.count(r"[A-Z]")
shape = ["length", "digits", "capitals"] + flags
sms.groupby("label")[shape].median()
length digits capitals phone link money
label
ham 52.0 0.0 2.0 0.0 0.0 0.0
spam 149.0 16.0 14.0 1.0 0.0 0.0
from sklearn.model_selection import train_test_split
train, test = train_test_split(sms, test_size=0.3,
                               stratify=sms["spam"], random_state=42)
ham_train = train[train["spam"] == 0]
print(len(ham_train), len(test), test["spam"].sum())
3377 1672 224

Common mistake: feeding the raw message text to the model

IsolationForest(random_state=42).fit(ham_train[["message"]])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_25840\3108366611.py in ?()
----> 1 IsolationForest(random_state=42).fit(ham_train[["message"]])

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py in ?(estimator, *args, **kwargs)
   1399                     )
   1400                 ),
   1401                 callback_management_context(estimator),
   1402             ):
-> 1403                 return fit_method(estimator, *args, **kwargs)

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\ensemble\_iforest.py in ?(self, X, y, sample_weight)
    316         -------
    317         self : object
    318             Fitted estimator.
    319         """
--> 320         X = validate_data(
    321             self, X, accept_sparse=["csc"], dtype=np.float32, ensure_all_finite=False
    322         )
    323 

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py in ?(_estimator, X, y, reset, validate_separately, skip_check_array, **check_params)
   3037             out = y
   3038         else:
   3039             out = X, y
   3040     elif not no_val_X and no_val_y:
-> 3041         out = check_array(X, input_name="X", **check_params)
   3042     elif no_val_X and not no_val_y:
   3043         out = _check_y(y, **check_params)
   3044     else:

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py in ?(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, ensure_all_finite, ensure_non_negative, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)
   1035                         )
   1036                     array = xp.astype(array, dtype, copy=False)
   1037                 else:
   1038                     array = _asarray_with_order(array, order=order, dtype=dtype, xp=xp)
-> 1039             except ComplexWarning as complex_warning:
   1040                 raise ValueError(
   1041                     "Complex data not supported\n{}\n".format(array)
   1042                 ) from complex_warning

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_array_api.py in ?(array, dtype, order, copy, xp, device)
    979         # Use NumPy API to support order
    980         if copy is True:
    981             array = numpy.array(array, order=order, dtype=dtype)
    982         else:
--> 983             array = numpy.asarray(array, order=order, dtype=dtype)
    984 
    985         # At this point array is a NumPy ndarray. We convert it to an array
    986         # container that is consistent with the input's namespace.

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\generic.py in ?(self, dtype, copy)
   2023             )
   2024         values = self._values
   2025         if copy is None:
   2026             # Note: branch avoids `copy=None` for NumPy 1.x support
-> 2027             arr = np.asarray(values, dtype=dtype)
   2028         else:
   2029             arr = np.array(values, dtype=dtype, copy=copy)
   2030 

ValueError: could not convert string to float: 'They said if its gonna snow, it will start around 8 or 9 pm tonite! They are predicting an inch of accumulation.'
guard = IsolationForest(n_estimators=200, random_state=42)
guard.fit(ham_train[shape])
y_test = test["spam"].to_numpy()
odd = -guard.score_samples(test[shape])
rank = np.argsort(-odd)
print(y_test[rank[:y_test.sum()]].sum(), y_test.sum())
194 224
flag_guard = guard.predict(test[shape]) == -1
print(flag_guard.sum(), y_test[flag_guard].sum())
345 210
ax = pd.Series(odd).groupby(y_test).plot.hist(
    bins=40, alpha=0.6, density=True, legend=True, figsize=(7, 4))[0]
ax.set_title("Message anomaly scores: 0 = ham, 1 = spam")
ax.set_xlabel("Anomaly score (higher = stranger)")
ax.set_ylabel("Density")
plt.show()
No description has been provided for this image

Quick check: which spam texts look most normal to the guard?

calm = test.iloc[rank[::-1]]
calm = calm[calm["spam"] == 1]["message"]
calm.str[:60].head(3).to_list()
['For sale - arsenal dartboard. Good condition but no doubles ',
 'Latest News! Police station toilet stolen, cops have nothing',
 'Ringtone Club: Get the UK singles chart on your mobile each ']

Mini-project: a layered defence for text messages#

Layer 1: the red-flag rule. Layer 2: the anomaly score, for what the rule lets through.

rule = ((test["phone"] == 1) | (test["link"] == 1)).to_numpy()
print(rule.sum(), y_test[rule].sum())
missed = y_test[~rule].sum()
print(missed, (~rule).sum())
197 188
36 1475
rest = np.where(~rule)[0]
worst = rest[np.argsort(-odd[rest])][:50]
print(y_test[worst].sum())
14
caught = y_test[rule].sum() + y_test[worst].sum()
false_alarms = (y_test[rule] == 0).sum() + (y_test[worst] == 0).sum()
print(caught, y_test.sum(), false_alarms, len(test))
202 224 45 1672
print(f"Two layers stop {caught / y_test.sum():.0%} of spam, with "
      f"{false_alarms} false alarms in {len(test)} messages")
Two layers stop 90% of spam, with 45 false alarms in 1672 messages

Recap#

  • Detection systems combine rules, scores, a queue and a budget
  • Simple rules are explainable; judge them by lift and by share caught
  • Isolation Forest isolates odd points in few splits, no labels needed
  • Report queues in business terms: frauds and dollars per budget
  • In security, learn normal and layer rules with anomaly scores

Challenges#

  1. Add the quiet-hours flag to the forest's features. Does the top 1,000 improve?
  2. Raise layer two to 100 reviews. Is the extra spam worth the extra false alarms?
  3. Add an exclamation-mark count to the message shape. Does the guard get better?

What's next#

Lesson 40: Hands-on Project- Credit Card Fraud Detection Train supervised models on heavily imbalanced data, judged by precision and recall.

Data Mining 40: Hands-on Project - Credit Card Fraud Detection#

In this lesson you'll be able to:

  • Measure class imbalance and see why accuracy misleads
  • Clean and split a large, rare-event dataset honestly
  • Judge a fraud model with precision, recall and the confusion matrix
  • Tackle imbalance with class weights and undersampling
  • Turn model results into money saved and alerts raised Dataset: 284,807 real European card transactions from September 2013 (data/mining/creditcard.csv.gz)

Warm-up: a simple fraud rule#

Lesson 39 met rule-based fraud flags. Try one: flag the biggest 1% of amounts.

import pandas as pd
df = pd.read_csv("creditcard.csv.gz")
print(df.shape)
df[["Time", "V1", "V2", "Amount", "Class"]].head(3)
(284807, 31)
Time V1 V2 Amount Class
0 0.0 -1.35981 -0.07278 149.62 0
1 0.0 1.19186 0.26615 2.69 0
2 1.0 -1.35835 -1.34016 378.66 0
cut = df["Amount"].quantile(0.99)
flagged = df[df["Amount"] > cut]
print(round(cut, 2), len(flagged), flagged["Class"].sum())
print(df["Class"].sum())
1017.97 2849 9
492

How rare is fraud?#

Problem: before modelling, we need to know how unbalanced the two classes are.

  • V1 to V28: anonymised PCA components (the bank hid the real fields)
  • Time: seconds since the first transaction in the file
  • Amount: payment value; Class: 1 = fraud, 0 = normal
df["Class"].value_counts()
Class
0    284315
1       492
Name: count, dtype: int64
fraud_rate = df["Class"].mean() * 100
print(f"Fraud rate: {fraud_rate:.3f}%")
Fraud rate: 0.173%

Quick check: a lazy model says never fraud for every transaction. What accuracy does it score?

lazy_accuracy = (df["Class"] == 0).mean()
print(round(lazy_accuracy, 4))
0.9983

Check data quality before modelling#

Problem: hidden repeats could leak the same transaction into both training and testing.

print(df.isna().sum().sum())
0
dupes = df.duplicated()
print(dupes.sum(), df.loc[dupes, "Class"].sum())
1081 19
cols = ["Time", "V1", "V2", "Amount", "Class"]
df.loc[df.duplicated(keep=False), cols].head(4)
Time V1 V2 Amount Class
32 26.0 -0.52991 0.87389 6.14 0
33 26.0 -0.52991 0.87389 6.14 0
34 26.0 -0.53539 0.86527 1.77 0
35 26.0 -0.53539 0.86527 1.77 0
df = df.drop_duplicates().reset_index(drop=True)
print(df.shape)
(283726, 31)

Your turn: after removing repeats, how many frauds are left, and what is the new fraud rate?

print(df["Class"].sum())
print(round(df["Class"].mean() * 100, 3))
473
0.167

Compare fraud with normal transactions#

Problem: a model can only learn fraud if fraud looks different somewhere in the data.

Quick check: is fraud mostly big purchases? Guess before the next cell.

amounts = df.groupby("Class")["Amount"]
amounts.describe()[["count", "mean", "50%", "max"]].round(2)
count mean 50% max
Class
0 283253.0 88.41 22.00 25691.16
1 473.0 123.87 9.82 2125.87
bands = [0, 1, 10, 100, 1000, 30000]
df["band"] = pd.cut(df["Amount"], bins=bands, include_lowest=True)
rate = df.groupby("band", observed=True)["Class"].mean() * 100
rate.round(3)
band
(-0.001, 1.0]        0.564
(1.0, 10.0]          0.096
(10.0, 100.0]        0.086
(100.0, 1000.0]      0.217
(1000.0, 30000.0]    0.307
Name: Class, dtype: float64
df["Hour"] = (df["Time"] // 3600 % 24).astype(int)
by_hour = df.groupby("Hour")["Class"].agg(["size", "mean"])
by_hour["fraud_pct"] = by_hour.pop("mean") * 100
by_hour.nlargest(3, "fraud_pct").round(2)
size fraud_pct
Hour
2 3308 1.45
4 2204 1.04
3 3487 0.49
import matplotlib.pyplot as plt
by_hour["fraud_pct"].plot(kind="bar", color="firebrick")
plt.title("Fraud rate by hour since the first transaction")
plt.xlabel("Hour (counted from the first transaction)")
plt.ylabel("Fraud rate (%)")
plt.show()
No description has been provided for this image
cols = ["V14", "V17", "V12", "V10"]
df.groupby("Class")[cols].mean().round(2)
V14 V17 V12 V10
Class
0 0.01 0.01 0.01 0.01
1 -6.84 -6.46 -6.10 -5.45
normal = df.loc[df["Class"] == 0, "V14"]
fraud = df.loc[df["Class"] == 1, "V14"]
plt.hist(normal, bins=60, density=True, alpha=0.5, label="normal")
plt.hist(fraud, bins=60, density=True, alpha=0.5, label="fraud")
plt.title("V14 separates fraud from normal")
plt.xlabel("V14 (anonymised component)")
plt.ylabel("Density")
plt.legend()
No description has been provided for this image

Split honestly before modelling#

Problem: with so few frauds, a careless split could put almost none in the test set.

from sklearn.model_selection import train_test_split
X = df.drop(columns=["Class", "Hour", "band"])
y = df["Class"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42)
print(len(X_train), y_train.sum())
print(len(X_test), y_test.sum())
212794 355
70932 118

Time and Amount are in seconds and euros; V1 to V28 are already centred. We scale, using training data only.

Common mistake: passing a single column (a Series) to a scaler

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train["Amount"])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[19], line 3
      1 from sklearn.preprocessing import StandardScaler
      2 scaler = StandardScaler()
----> 3 scaler.fit(X_train["Amount"])

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\preprocessing\_data.py:928, in StandardScaler.fit(self, X, y, sample_weight)
    926 # Reset internal state before fitting
    927 self._reset()
--> 928 return self.partial_fit(X, y, sample_weight)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py:1403, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1393     estimator._validate_params()
   1395 with (
   1396     config_context(
   1397         skip_parameter_validation=(
   (...)
   1401     callback_management_context(estimator),
   1402 ):
-> 1403     return fit_method(estimator, *args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\preprocessing\_data.py:965, in StandardScaler.partial_fit(self, X, y, sample_weight)
    963 xp, _, X_device = get_namespace_and_device(X)
    964 first_call = not hasattr(self, "n_samples_seen_")
--> 965 X = validate_data(
    966     self,
    967     X,
    968     accept_sparse=("csr", "csc"),
    969     dtype=supported_float_dtypes(xp, X_device),
    970     ensure_all_finite="allow-nan",
    971     reset=first_call,
    972 )
    973 n_features = X.shape[1]
    975 callback_ctx = self._init_callback_context()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:3041, in validate_data(_estimator, X, y, reset, validate_separately, skip_check_array, **check_params)
   3039         out = X, y
   3040 elif not no_val_X and no_val_y:
-> 3041     out = check_array(X, input_name="X", **check_params)
   3042 elif no_val_X and not no_val_y:
   3043     out = _check_y(y, **check_params)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:1076, in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, ensure_all_finite, ensure_non_negative, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)
   1069         else:
   1070             msg = (
   1071                 f"Expected 2D array, got 1D array instead:\narray={array}.\n"
   1072                 "Reshape your data either using array.reshape(-1, 1) if "
   1073                 "your data has a single feature or array.reshape(1, -1) "
   1074                 "if it contains a single sample."
   1075             )
-> 1076         raise ValueError(msg)
   1078 if dtype_numeric and hasattr(array.dtype, "kind") and array.dtype.kind in "USV":
   1079     raise ValueError(
   1080         "dtype='numeric' is not compatible with arrays of bytes/strings."
   1081         "Convert your data to numeric values explicitly instead."
   1082     )

ValueError: Expected a 2-dimensional container but got <class 'pandas.Series'> instead. Pass a DataFrame containing a single row (i.e. single sample) or a single column (i.e. single feature) instead.
scaler.fit(X_train[["Amount"]])
print(scaler.mean_.round(2), scaler.scale_.round(2))
[88.6] [246.99]

A pipeline does this for every column, fitting the scaler inside training only.

A baseline model judged by precision and recall#

Problem: we need scores that reward catching fraud, not just being right about normal payments.

  • Recall: of all real frauds, the share we caught
  • Precision: of all our alerts, the share that were really fraud
  • The confusion matrix counts all four outcomes
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score, recall_score
lazy = DummyClassifier(strategy="most_frequent")
lazy_pred = lazy.fit(X_train, y_train).predict(X_test)
print(round(accuracy_score(y_test, lazy_pred), 4))
print(recall_score(y_test, lazy_pred))
0.9983
0.0
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LogisticRegression
lr = make_pipeline(StandardScaler(),
                   LogisticRegression(max_iter=1000))
lr.fit(X_train, y_train)
lr_pred = lr.predict(X_test)
print(lr_pred.sum())
82
from sklearn.metrics import confusion_matrix
print(confusion_matrix(y_test, lr_pred))
[[70801    13]
 [   49    69]]
from sklearn.metrics import precision_score
print(round(precision_score(y_test, lr_pred), 3))
print(round(recall_score(y_test, lr_pred), 3))
print(round(accuracy_score(y_test, lr_pred), 4))
0.841
0.585
0.9991
from sklearn.metrics import classification_report
print(classification_report(y_test, lr_pred, digits=3))
              precision    recall  f1-score   support

           0      0.999     1.000     1.000     70814
           1      0.841     0.585     0.690       118

    accuracy                          0.999     70932
   macro avg      0.920     0.792     0.845     70932
weighted avg      0.999     0.999     0.999     70932

Common mistake: scoring probabilities with recall

lr_proba = lr.predict_proba(X_test)[:, 1]
recall_score(y_test, lr_proba)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[26], line 2
      1 lr_proba = lr.predict_proba(X_test)[:, 1]
----> 2 recall_score(y_test, lr_proba)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_param_validation.py:218, in validate_params.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    212 try:
    213     with config_context(
    214         skip_parameter_validation=(
    215             prefer_skip_nested_validation or global_skip_validation
    216         )
    217     ):
--> 218         return func(*args, **kwargs)
    219 except InvalidParameterError as e:
    220     # When the function is just a wrapper around an estimator, we allow
    221     # the function to delegate validation to the estimator, but we replace
    222     # the name of the estimator by the name of the function in the error
    223     # message to avoid confusion.
    224     msg = re.sub(
    225         r"parameter of \w+ must be",
    226         f"parameter of {func.__qualname__} must be",
    227         str(e),
    228     )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\metrics\_classification.py:2833, in recall_score(y_true, y_pred, labels, pos_label, average, sample_weight, zero_division)
   2662 @validate_params(
   2663     {
   2664         "y_true": ["array-like", "sparse matrix"],
   (...)
   2689     zero_division="warn",
   2690 ):
   2691     """Compute the recall.
   2692 
   2693     The recall is the ratio ``tp / (tp + fn)`` where ``tp`` is the number of
   (...)
   2831     array([1. , 1. , 0.5])
   2832     """
-> 2833     _, r, _, _ = precision_recall_fscore_support(
   2834         y_true,
   2835         y_pred,
   2836         labels=labels,
   2837         pos_label=pos_label,
   2838         average=average,
   2839         warn_for=("recall",),
   2840         sample_weight=sample_weight,
   2841         zero_division=zero_division,
   2842     )
   2843     return r

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_param_validation.py:191, in validate_params.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    189 global_skip_validation = get_config()["skip_parameter_validation"]
    190 if global_skip_validation:
--> 191     return func(*args, **kwargs)
    193 func_sig = signature(func)
    195 # Map *args/**kwargs to the function signature

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\metrics\_classification.py:2145, in precision_recall_fscore_support(y_true, y_pred, beta, labels, pos_label, average, warn_for, sample_weight, zero_division)
   2143 xp, _, device_ = get_namespace_and_device(y_pred)
   2144 y_true, sample_weight = move_to(y_true, sample_weight, xp=xp, device=device_)
-> 2145 labels = _check_set_wise_labels(y_true, y_pred, average, labels, pos_label)
   2147 # Calculate tp_sum, pred_sum, true_sum ###
   2148 samplewise = average == "samples"

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\metrics\_classification.py:1910, in _check_set_wise_labels(y_true, y_pred, average, labels, pos_label)
   1907     raise ValueError("average has to be one of " + str(average_options))
   1909 y_true, y_pred = attach_unique(y_true, y_pred)
-> 1910 y_type, present_labels, y_true, y_pred, _ = _check_targets(y_true, y_pred)
   1911 if average == "binary":
   1912     if y_type == "binary":

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\metrics\_classification.py:134, in _check_targets(y_true, y_pred, sample_weight)
    131     y_type = {"multiclass"}
    133 if len(y_type) > 1:
--> 134     raise ValueError(
    135         "Classification metrics can't handle a mix of {0} and {1} targets".format(
    136             type_true, type_pred
    137         )
    138     )
    140 # We can't have more than one value in y_type => The set is no more needed
    141 y_type = y_type.pop()

ValueError: Classification metrics can't handle a mix of binary and continuous targets
lr_flag = (lr_proba >= 0.5).astype(int)
print(round(recall_score(y_test, lr_flag), 3))
print(round(lr_proba.max(), 3), int((lr_proba >= 0.5).sum()))
0.585
1.0 82

Fight the imbalance#

Problem: the model barely sees fraud, so it plays safe and misses too many.

  • Class weights: make each fraud count more during training
  • Undersampling: train on all frauds plus a sample of normal rows
  • Either way, the test set stays untouched
lr_bal = make_pipeline(StandardScaler(), LogisticRegression(
    max_iter=1000, class_weight="balanced"))
bal_pred = lr_bal.fit(X_train, y_train).predict(X_test)
print(round(recall_score(y_test, bal_pred), 3))
0.89

Quick check: recall went up. What do you expect happened to precision?

print(round(precision_score(y_test, bal_pred), 3))
print(confusion_matrix(y_test, bal_pred))
0.055
[[69008  1806]
 [   13   105]]
train = X_train.assign(Class=y_train)
frauds = train[train["Class"] == 1]
normals = train[train["Class"] == 0].sample(
    n=len(frauds) * 20, random_state=42)
small = pd.concat([frauds, normals])
print(len(small), round(small["Class"].mean() * 100, 2))
7455 4.76
X_small = small.drop(columns="Class")
y_small = small["Class"]
lr_under = make_pipeline(StandardScaler(),
                         LogisticRegression(max_iter=1000))
under_pred = lr_under.fit(X_small, y_small).predict(X_test)
print(round(precision_score(y_test, under_pred), 3))
print(round(recall_score(y_test, under_pred), 3))
0.438
0.814
from sklearn.ensemble import RandomForestClassifier
forest = RandomForestClassifier(n_estimators=100, random_state=42)
forest_pred = forest.fit(X_small, y_small).predict(X_test)
print(round(precision_score(y_test, forest_pred), 3))
print(round(recall_score(y_test, forest_pred), 3))
0.701
0.797
imp = pd.Series(forest.feature_importances_, index=X.columns)
imp.nlargest(6).sort_values().plot(kind="barh", color="teal")
plt.title("Top features for spotting fraud")
plt.xlabel("Importance")
plt.show()
No description has been provided for this image

Your turn: print the forest's confusion matrix. How many frauds did it miss?

print(confusion_matrix(y_test, forest_pred))
[[70774    40]
 [   24    94]]

Mini-project: which model saves the most money?#

Compare all four models on the same test set, then count the money caught and the false alarms.

models = {"plain LR": lr, "weighted LR": lr_bal,
          "undersampled LR": lr_under, "forest": forest}
rows = []
for name, model in models.items():
    pred = model.predict(X_test)
    rows.append({"model": name, "alerts": int(pred.sum()),
                 "precision": precision_score(y_test, pred),
                 "recall": recall_score(y_test, pred)})
pd.DataFrame(rows).set_index("model").round(3)
alerts precision recall
model
plain LR 82 0.841 0.585
weighted LR 1911 0.055 0.890
undersampled LR 219 0.438 0.814
forest 134 0.701 0.797
from sklearn.metrics import PrecisionRecallDisplay
ax = plt.gca()
PrecisionRecallDisplay.from_estimator(lr, X_test, y_test, ax=ax,
                                      name="plain LR")
PrecisionRecallDisplay.from_estimator(forest, X_test, y_test,
                                      ax=ax, name="forest")
plt.title("Precision vs recall at every cut-off")
plt.show()
No description has been provided for this image
fraud_amount = X_test["Amount"][y_test == 1]
caught = fraud_amount[forest_pred[y_test == 1] == 1].sum()
missed = fraud_amount[forest_pred[y_test == 1] == 0].sum()
print(round(caught, 2), round(missed, 2))
print(round(caught / fraud_amount.sum() * 100, 1))
13699.49 3798.16
78.3
false_alarms = int(((forest_pred == 1) & (y_test == 0)).sum())
per_10k = false_alarms / (y_test == 0).sum() * 10_000
print(false_alarms, round(per_10k, 1))
40 5.6
risk = forest.predict_proba(X_test)[:, 1]
top = X_test.assign(risk=risk, fraud=y_test).nlargest(5, "risk")
top[["Amount", "V14", "risk", "fraud"]].round(2)
Amount V14 risk fraud
41828 112.33 -8.23 1.0 1
122008 219.80 -7.76 1.0 1
247381 0.00 -9.86 1.0 1
15683 99.99 -7.08 1.0 1
27641 19.02 -11.18 1.0 1

Finding: the undersampled forest catches about 4 in 5 frauds while bothering very few genuine customers.

Recap#

  • With 0.17% fraud, accuracy is useless: never-fraud scores 99.8%
  • Drop duplicates and stratify the split before any model sees the data
  • Precision counts good alerts; recall counts frauds caught
  • Class weights and undersampling trade precision for recall
  • Always test on the untouched, imbalanced test set

Challenges#

  1. Undersample with 5 and 50 normals per fraud. How do precision and recall move?
  2. Add the Hour column to the features. Does the forest improve?
  3. Give the forest class_weight="balanced" on the small table and compare.

What's next#

Lesson 41: Introduction to Time-Series Data, where the order of rows matters.

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.