Lesson 5 · Data Mining Full Course
Data Mining Full Course Part 5 (3 Hours): Market Basket Analysis, Outliers & Fraud Detection
All 8 lesson notebooks from Part 5 of the Data Mining Full Course in Python, in teaching order. Each lesson starts with a heading; the timestamps below jump…
- CourseData Mining Full Course
- Lesson5 of 6
- Video3 h 11 min
- FormatJupyter notebook · 297 code cells
- Data5 datasets
What you'll learn
- Data Mining 33: Association Rule Mining with Apriori Algorithm
- Warm-up: from grouping rows to grouping items
- From baskets to a True/False item table
- Support: how often does an itemset appear?
- The Apriori trick and choosing minsupport
- Rules: confidence and lift
- Mini-project: a cross-promotion shortlist for the store
- Challenges
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
- groceries_baskets.csv preview489.1 KB
- taxis.csv preview849.0 KB
- sms_spam.csv preview469.5 KB
- creditcard.csv.gz preview26.3 MB
- online_retail.csv.gz preview7.1 MB
📓 Full notebook
Download .ipynbData Mining Full Course, Part 5: Market Basket Analysis, Outliers and Fraud Detection#
All 8 lesson notebooks from Part 5 of the Data Mining Full Course in Python, in teaching order. Each lesson starts with a heading; the timestamps below jump to it in the video.
- 0:00 Lesson 33: Association Rule Mining with Apriori Algorithm
- 0:23:18 Lesson 34: FP-Growth and Market Basket Analysis
- 0:47:48 Lesson 35: Sequential Pattern Mining
- 1:12:07 Lesson 36: Introduction to Anomaly and Outlier Detection
- 1:36:25 Lesson 37: Statistical Methods for Outlier Detection
- 2:00:42 Lesson 38: Distance and Density-Based Outlier Detection
- 2:23:58 Lesson 39: Applications in Fraud Detection and Cybersecurity
- 2:46:46 Lesson 40: Hands-on Project- Credit Card Fraud Detection
Data files: save the datasets from the lesson page in the same folder as this notebook. Files the lessons create are written to that folder too.
Data Mining 33: Association Rule Mining with Apriori Algorithm#
In this lesson you'll be able to:
- Turn raw shopping baskets into a True/False item table
- Calculate support, confidence and lift by hand
- Find frequent itemsets with Apriori and choose min_support
- Generate, filter and read association rules
- Turn rules into a store recommendation Dataset: 9,835 real grocery baskets from one month at a store (data/mining/groceries_baskets.csv)
Warm-up: from grouping rows to grouping items#
Lesson 32 grouped countries that looked alike. Today we look for items that appear together.
with open("groceries_baskets.csv") as f:
baskets = [line.strip().split(",") for line in f]
print(len(baskets))
print(baskets[:3])
import pandas as pd
sizes = pd.Series([len(b) for b in baskets])
sizes.describe().round(2)
Quick check: what share of baskets hold just one item?
print((sizes == 1).sum(), round((sizes == 1).mean(), 3))
From baskets to a True/False item table#
Problem: Apriori needs one row per basket and one True/False column per item.
items = pd.Series([i for b in baskets for i in b])
print(items.nunique())
items.value_counts().head(8)
import matplotlib.pyplot as plt
items.value_counts().head(15).sort_values().plot(kind="barh")
plt.title("Top 15 products by number of baskets")
plt.xlabel("Number of baskets")
plt.ylabel("Product")
plt.show()
Common mistake: handing the raw list of baskets to apriori
from mlxtend.frequent_patterns import apriori
apriori(baskets, min_support=0.02)
from mlxtend.preprocessing import TransactionEncoder
te = TransactionEncoder()
onehot = pd.DataFrame(te.fit_transform(baskets), columns=te.columns_)
print(onehot.shape)
onehot[["whole milk", "yogurt", "rolls/buns", "coffee"]].head()
Support: how often does an itemset appear?#
Problem: we need a number for how common an item, or a group of items, is.
- Itemset: one or more items, such as {whole milk, yogurt}
- Support: the share of all baskets that contain the whole itemset
Common mistake: guessing a product's name
onehot["milk"].mean()
[c for c in onehot.columns if "milk" in c]
milk = onehot["whole milk"]
yog = onehot["yogurt"]
print(round(milk.mean(), 3), round(yog.mean(), 3))
Quick check: can {whole milk, yogurt} have higher support than yogurt alone?
both = milk & yog
print(both.sum(), yog.sum())
print(round(both.mean(), 3))
Your turn: what is the support of {beef, root vegetables}?
beef_root = onehot["beef"] & onehot["root vegetables"]
print(beef_root.sum(), round(beef_root.mean(), 4))
frequent = apriori(onehot, min_support=0.02, use_colnames=True)
frequent["length"] = frequent["itemsets"].apply(len)
frequent["length"].value_counts()
for _, row in frequent[frequent["length"] == 3].iterrows():
print(sorted(row["itemsets"]), round(row["support"], 3))
pairs = frequent[frequent["length"] == 2]
pairs.nlargest(5, "support").round(3)
top10 = pairs.nlargest(10, "support")
names = top10["itemsets"].apply(lambda s: " + ".join(sorted(s)))
top10.set_index(names)["support"].sort_values().plot(kind="barh")
plt.title("Ten most common product pairs")
plt.xlabel("Support (share of baskets)")
plt.ylabel("Pair")
plt.show()
The Apriori trick and choosing min_support#
Problem: 171 products make far too many combinations to count them all.
- If an itemset is rare, every bigger set containing it is rarer still
- So Apriori grows sets one item at a time, dropping rare ones early
n = onehot.shape[1]
print(n * (n - 1) // 2)
rare = [c for c in onehot.columns if onehot[c].mean() < 0.02]
print(len(rare))
a = onehot["whole milk"]
ab = a & onehot["other vegetables"]
abc = ab & onehot["rolls/buns"]
print(round(a.mean(), 3), round(ab.mean(), 3), round(abc.mean(), 3))
counts = {}
for s in [0.05, 0.03, 0.02, 0.01, 0.005]:
counts[s] = len(apriori(onehot, min_support=s, use_colnames=True))
counts = pd.Series(counts)
counts
counts.plot(marker="o", logy=True)
plt.title("Frequent itemsets by minimum support")
plt.xlabel("Minimum support")
plt.ylabel("Number of itemsets (log scale)")
plt.show()
Common mistake: giving min_support as a number of baskets
apriori(onehot, min_support=50, use_colnames=True)
share = 50 / len(onehot)
few = apriori(onehot, min_support=share, use_colnames=True)
print(round(share, 4), len(few))
Rules: confidence and lift#
Problem: knowing two items are common together isn't a recommendation yet.
- Confidence(A -> B): of the baskets with A, the share that also have B
- Lift: confidence divided by B's own support. Above 1 means A really boosts B
conf = both.sum() / yog.sum()
lift = conf / milk.mean()
print(round(conf, 3), round(lift, 3))
Quick check: is whole milk -> yogurt as strong as yogurt -> whole milk?
conf_back = both.sum() / milk.sum()
lift_back = conf_back / yog.mean()
print(round(conf_back, 3), round(lift_back, 3))
from mlxtend.frequent_patterns import association_rules
freq = apriori(onehot, min_support=0.01, use_colnames=True)
rules = association_rules(freq, num_itemsets=len(onehot),
metric="confidence", min_threshold=0.3)
print(len(rules))
rules["if"] = rules["antecedents"].apply(lambda s: ", ".join(sorted(s)))
rules["then"] = rules["consequents"].apply(lambda s: ", ".join(sorted(s)))
rules = rules.sort_values(["if", "then"], ignore_index=True)
cols = ["if", "then", "confidence", "lift"]
by_conf = rules.sort_values("confidence", ascending=False,
kind="stable")
by_conf[cols].head().round(3)
Quick check: most top rules end in whole milk. Is milk really special?
print(round((rules["then"] == "whole milk").mean(), 3))
print(round(milk.mean(), 3))
by_lift = rules.sort_values("lift", ascending=False, kind="stable")
by_lift[cols].head().round(3)
Quick check: can lift drop below 1? What would that mean?
every = association_rules(freq, num_itemsets=len(onehot),
metric="lift", min_threshold=0)
print(len(every), (every["lift"] < 1).sum())
every.nsmallest(2, "lift")[["antecedents", "consequents", "lift"]].round(3)
rules.plot.scatter(x="confidence", y="lift", alpha=0.6)
plt.axhline(1, color="grey", linestyle="--")
plt.title(f"Confidence against lift for {len(rules)} rules")
plt.xlabel("Confidence")
plt.ylabel("Lift")
plt.show()
Your turn: list the rules whose if side contains root vegetables, best lift first.
has_root = by_lift["antecedents"].apply(
lambda s: "root vegetables" in s)
root = by_lift[has_root]
print(len(root))
root[cols].head().round(3)
Mini-project: a cross-promotion shortlist for the store#
Find strong, common pair rules a store manager could act on this week.
pair = rules["antecedents"].apply(len) == 1
strong = rules[pair & (rules["lift"] > 1.5)
& (rules["support"] >= 0.015)]
strong = strong.sort_values("lift", ascending=False, kind="stable")
print(len(strong))
strong[cols + ["support"]].head(8).round(3)
strong[["if", "then"]].value_counts("then")
Your turn: find the best shortlisted rule whose then side is not a staple.
staples = ["whole milk", "other vegetables"]
fresh = strong[~strong["then"].isin(staples)]
fresh[cols].round(3)
top = strong.iloc[0]
n = round(top["support"] * len(onehot))
print(top["if"], "->", top["then"], n,
round(top["confidence"], 3), round(top["lift"], 3))
import seaborn as sns
grid = strong.pivot(index="if", columns="then", values="lift")
plt.figure(figsize=(6, 7))
sns.heatmap(grid.round(1), annot=True, cmap="Greens")
plt.title("Lift of the shortlisted rules")
plt.show()
Honesty check: a real rule should hold in both halves of the baskets.
half = len(onehot) // 2
for part in [onehot.iloc[:half], onehot.iloc[half:]]:
b, r = part["beef"], part["root vegetables"]
conf_h = (b & r).sum() / b.sum()
print(len(part), round(conf_h, 3), round(conf_h / r.mean(), 3))
import os
strong[cols + ["support"]].round(3).to_csv(
"m33_rules.csv",
index=False)
print(pd.read_csv("m33_rules.csv").shape)
Finding: beef -> root vegetables is the standout: 171 baskets, lift 3.04. 32 of the 36 shortlisted rules end in whole milk or other vegetables, the store's staples.
Recap#
- Encode baskets as a True/False table with TransactionEncoder
- Support: share of baskets containing the whole itemset
- Apriori prunes any set whose subsets are already rare
- min_support is a share between 0 and 1, not a count
- Confidence can flatter popular items; lift above 1 is the real signal
Challenges#
- Rerun the rules with min_support 0.005. How many rules have lift above 2?
- Find every rule whose then side is not whole milk or other vegetables.
- Keep only baskets with 5 or more items and repeat the mini-project.
What's next#
Lesson 34: FP-Growth and Market Basket Analysis A faster way to find the same itemsets, and deeper basket analysis.
Data Mining 34: FP-Growth and Market Basket Analysis#
In this lesson you'll be able to:
- Explain why Apriori slows down, and how an FP-tree avoids it
- Mine frequent itemsets with FP-Growth in mlxtend
- Read support, confidence and lift on association rules
- Filter hundreds of rules down to the few worth acting on
- Test a rule-based recommender on baskets it has never seen Dataset: 9,835 real grocery baskets (data/mining/groceries_baskets.csv)
Warm-up: Apriori from lesson 33#
Load the baskets, one-hot encode them, and find the common items.
import pandas as pd
from mlxtend.preprocessing import TransactionEncoder
from mlxtend.frequent_patterns import apriori
with open("groceries_baskets.csv") as f:
baskets = [line.rstrip("\n").split(",") for line in f]
print(len(baskets))
print(baskets[:3])
te = TransactionEncoder()
X = pd.DataFrame(te.fit_transform(baskets), columns=te.columns_)
print(X.shape)
freq_ap = apriori(X, min_support=0.05, use_colnames=True)
freq_ap.sort_values("support", ascending=False).head()
Why Apriori slows down#
Problem: lower the support threshold and the work explodes.
sizes = {}
for s in [0.05, 0.02, 0.01, 0.005, 0.003]:
sizes[s] = len(apriori(X, min_support=s, use_colnames=True))
print(sizes)
import matplotlib.pyplot as plt
pd.Series(sizes).plot(marker="o", logy=True)
plt.title("Lower support, many more itemsets")
plt.xlabel("Minimum support")
plt.ylabel("Frequent itemsets (log scale)")
plt.show()
Quick check: with 169 products, how many possible pairs and triples are there?
import math
print(math.comb(169, 2), math.comb(169, 3))
Apriori builds candidate itemsets level by level and rescans every basket to count them.
The FP-tree: squeeze the baskets into shared paths#
Problem: read the baskets once, store them compactly, and never list candidates.
- Count every product once, then sort each basket from most to least common
- Baskets that start the same way share one path in the tree
- Mine patterns from the tree, one product at a time
counts = X.sum().sort_values(ascending=False)
counts.head(10).sort_values().plot.barh()
plt.title("The 10 most common grocery items")
plt.xlabel("Number of baskets")
plt.ylabel("Product")
plt.show()
rank = {item: i for i, item in enumerate(counts.index)}
for basket in baskets[:4]:
print(sorted(basket, key=rank.get))
nodes = set()
for basket in baskets:
path = tuple(sorted(basket, key=rank.get))
for k in range(1, len(path) + 1):
nodes.add(path[:k])
print(sum(len(b) for b in baskets), len(nodes))
paths = [tuple(sorted(b, key=rank.get)) for b in baskets]
pd.Series([p[0] for p in paths]).value_counts().head()
To mine one item, FP-Growth collects the paths that lead to it: its prefix paths.
prefixes = [p[:p.index("curd")] for p in paths if "curd" in p]
print(len(prefixes))
above = pd.Series([i for p in prefixes for i in p]).value_counts()
above.head()
Quick check: what is the support of curd and whole milk together?
print(round(above["whole milk"] / len(baskets), 4))
print(round((X["curd"] & X["whole milk"]).mean(), 4))
FP-Growth in mlxtend#
Problem: find the same itemsets as Apriori, faster.
Common mistake: handing FP-Growth the raw list of baskets
from mlxtend.frequent_patterns import fpgrowth
fpgrowth(baskets, min_support=0.01)
freq = fpgrowth(X, min_support=0.01, use_colnames=True)
print(len(freq))
freq.sort_values("support", ascending=False).head()
freq_ap = apriori(X, min_support=0.01, use_colnames=True)
same = set(freq["itemsets"]) == set(freq_ap["itemsets"])
print(len(freq_ap), same)
import time
for algo in [apriori, fpgrowth]:
start = time.perf_counter()
found = algo(X, min_support=0.003, use_colnames=True)
secs = time.perf_counter() - start
print(algo.__name__, len(found), round(secs, 2))
Common mistake: trusting product names to be clean
print(X["cream cheese"].mean())
[c for c in X.columns if c != c.strip()]
baskets = [[item.strip() for item in b] for b in baskets]
X = pd.DataFrame(te.fit_transform(baskets), columns=te.columns_)
print(round(X["cream cheese"].mean(), 4))
freq = fpgrowth(X, min_support=0.01, use_colnames=True)
print(len(freq))
Quick check: at 0.3% support, how many itemsets remain if we only allow pairs or smaller?
short = fpgrowth(X, min_support=0.003, max_len=2, use_colnames=True)
print(len(short))
Your turn: how many frequent itemsets have 1, 2 and 3 items? Show the top triples.
freq["size"] = freq["itemsets"].apply(len)
print(freq["size"].value_counts().sort_index())
triples = freq[freq["size"] == 3].sort_values("support",
ascending=False)
triples.assign(items=triples["itemsets"].apply(sorted).str.join(", "))[
["items", "support"]].head().round(3)
From itemsets to rules#
Problem: an itemset says what goes together; a rule says what to suggest next.
- Support: share of all baskets holding both sides
- Confidence: of the baskets with the if side, the share that also have the then side
- Lift: confidence divided by how popular the then side is anyway
from mlxtend.frequent_patterns import association_rules
rules = association_rules(freq, metric="lift", min_threshold=1.0)
rules["if"] = rules["antecedents"].apply(sorted).str.join(", ")
rules["then"] = rules["consequents"].apply(sorted).str.join(", ")
cols = ["if", "then", "confidence", "lift"]
print(len(rules))
top_lift = rules.sort_values("lift", ascending=False)
top_lift[["if", "then", "lift"]].head().round(3)
a, b = X["butter"], X["whole milk"]
support = (a & b).mean()
confidence = (a & b).sum() / a.sum()
lift = confidence / b.mean()
print(round(support, 3), round(confidence, 3), round(lift, 2))
pick = (rules["if"] == "butter") & (rules["then"] == "whole milk")
rules.loc[pick, cols].round(3)
Quick check: whole milk is in about a quarter of baskets. Is a rule ending in whole milk with 40% confidence impressive?
milk = X["whole milk"].mean()
print(round(milk, 3), round(0.40 / milk, 2))
Quick check: what happens if we mine rules from itemsets found at 10% support?
high = fpgrowth(X, min_support=0.1, use_colnames=True)
print(len(high), high["itemsets"].apply(len).max())
print(len(association_rules(high, metric="lift", min_threshold=1)))
Choosing rules worth acting on#
Problem: hundreds of rules, and most are obvious or rare.
pts = plt.scatter(rules["support"], rules["confidence"],
c=rules["lift"], cmap="viridis", s=15)
plt.colorbar(pts, label="Lift")
plt.title("Every rule: support, confidence and lift")
plt.xlabel("Support (share of baskets)")
plt.ylabel("Confidence")
plt.show()
strong = rules[(rules["confidence"] >= 0.4) & (rules["lift"] >= 2)]
print(len(strong))
strong.sort_values("confidence", ascending=False)[cols].head().round(3)
top = strong.sort_values("leverage", ascending=False).head(3)
top = top.assign(extra=(top["leverage"] * len(X)).round().astype(int))
top[["if", "then", "leverage", "extra"]].round(4)
Hand the shortlist to the store team as a CSV file.
import os
shortlist = strong.sort_values("lift", ascending=False)[cols]
shortlist.to_csv("m34_strong_rules.csv", index=False)
saved = pd.read_csv("m34_strong_rules.csv")
print(saved.shape)
saved.head(3).round(3)
best = strong.sort_values("lift").tail(10)
labels = best["if"] + " -> " + best["then"]
plt.barh(labels, best["lift"])
plt.title("Ten strong rules ranked by lift")
plt.xlabel("Lift")
plt.ylabel("Rule")
plt.show()
Your turn: which rules lead to root vegetables? Show the five with the highest lift.
to_roots = rules[rules["then"] == "root vegetables"]
to_roots.sort_values("lift", ascending=False)[cols].head().round(3)
Mini-project: do the rules predict real baskets?#
Hide one item from a basket. Can rules guess it better than just suggesting the bestsellers? Mine rules on 80% of baskets; test on the other 20%.
from sklearn.model_selection import train_test_split
train, test = train_test_split(baskets, test_size=0.2,
random_state=42)
te_train = TransactionEncoder()
X_train = pd.DataFrame(te_train.fit_transform(train),
columns=te_train.columns_)
freq_train = fpgrowth(X_train, min_support=0.005, use_colnames=True)
shop_rules = association_rules(freq_train, metric="confidence",
min_threshold=0.1)
shop_rules = shop_rules.sort_values("confidence", ascending=False)
print(len(train), len(test), len(shop_rules))
popular = list(X_train.mean().sort_values(ascending=False).index)
def recommend(basket, n=3):
have, picks = set(basket), []
for ante, cons in zip(shop_rules["antecedents"],
shop_rules["consequents"]):
if ante <= have:
picks += [i for i in sorted(cons) if i not in have | set(picks)]
if len(picks) >= n:
break
picks += [i for i in popular if i not in have | set(picks)]
return picks[:n]
print(recommend(["butter"]))
sample = next(b for b in test if len(b) >= 4)
print(sample)
print(recommend(sample[:-1]))
def hit_rate(suggest):
hits = tries = 0
for basket in test:
if len(basket) < 2:
continue
for i, item in enumerate(basket):
rest = basket[:i] + basket[i + 1:]
hits += item in suggest(rest)
tries += 1
return tries, round(hits / tries, 3)
def bestsellers(basket, n=3):
return [i for i in popular if i not in basket][:n]
print(hit_rate(recommend))
print(hit_rate(bestsellers))
Finding: rules guess the hidden item 19.6% of the time, against 17.9% for bestsellers. A real but modest edge: grocery baskets are dominated by staples.
Recap#
- Apriori lists candidates level by level; at low support that explodes
- FP-Growth packs baskets into a tree of shared prefixes, then mines it
- Encode baskets as a one-hot table, and clean item names first
- Confidence says how often a rule is right; lift says whether it beats chance
- Judge rules on held-out baskets, against a simple baseline
Challenges#
- Rerun the mini-project with min_support 0.003. Does the hit rate improve?
- Rank shop_rules by lift instead of confidence and compare.
- Find the rule with the highest lift that has three items on the if side.
What's next#
Lesson 35: Sequential Pattern Mining, where the order of purchases matters.
Data Mining 35: Sequential Pattern Mining#
In this lesson you'll be able to:
- Turn raw transactions into one time-ordered sequence per customer
- Test whether a sequence contains a pattern, and measure its support
- Mine every two-step pattern, and split reorders from cross-sells
- Grow longer patterns with the Apriori trick
- Time a follow-up offer from real purchase gaps Dataset: Online Retail, a year of real orders from a UK online gift shop (data/pandas/online_retail.csv.gz)
Warm-up: what sells together (lesson 34)#
Load the orders, keep real sales, and run FP-Growth on the invoices.
import pandas as pd
df = pd.read_csv("online_retail.csv.gz")
sales = df.dropna(subset=["CustomerID"])
sales = sales[~sales["InvoiceNo"].str.startswith("C")]
keep = (sales["Quantity"] > 0) & sales["StockCode"].str.match(r"\d")
sales = sales[keep].copy()
print(len(df), len(sales))
print(sales["CustomerID"].nunique(), sales["InvoiceNo"].nunique())
from mlxtend.preprocessing import TransactionEncoder
from mlxtend.frequent_patterns import fpgrowth
baskets = sales.groupby("InvoiceNo")["Description"].apply(set)
te = TransactionEncoder()
X = pd.DataFrame(te.fit_transform(baskets.tolist()),
columns=te.columns_)
freq = fpgrowth(X, min_support=0.02, use_colnames=True, max_len=2)
pairs = freq[freq["itemsets"].apply(len) == 2].copy()
pairs["items"] = pairs["itemsets"].apply(sorted).str.join(" + ")
top = pairs.sort_values("support", ascending=False)
for r in top.head(3).itertuples():
print(r.items, round(r.support, 3))
FP-Growth sees each basket on its own. It can't say which purchase came first.
Order matters: put purchases on a timeline#
Problem: baskets forget time; sequences keep it.
- A sequence is one customer's baskets, in time order
- A pattern like <A, B> means: bought A, then B in a later basket
- Support is the share of customers whose sequence contains the pattern
Common mistake: using dates before converting them from text
sales["InvoiceDate"].dt.month.head()
Quick check: if we sort the raw date text, which date comes first and which comes last?
sales["InvoiceDate"].sort_values().iloc[[0, -1]].tolist()
sales["InvoiceDate"] = pd.to_datetime(sales["InvoiceDate"],
format="%m/%d/%Y %H:%M")
print(sales["InvoiceDate"].min(), sales["InvoiceDate"].max())
print(sales["InvoiceDate"].dt.month.head(3).tolist())
One sequence per customer#
Problem: turn hundreds of thousands of rows into ordered lists of baskets.
inv = (sales.groupby(["CustomerID", "InvoiceNo"])
.agg(date=("InvoiceDate", "min"),
n_items=("Description", "nunique"),
items=("Description", set))
.reset_index()
.sort_values(["CustomerID", "date", "InvoiceNo"]))
print(len(inv))
inv[["CustomerID", "InvoiceNo", "date", "n_items"]].head()
inv.loc[inv["CustomerID"] == 12347, ["date", "n_items"]]
for r in inv[inv["CustomerID"] == 12891].itertuples():
print(r.date.date(), sorted(r.items))
seqs = inv.groupby("CustomerID")["items"].apply(list)
print(len(seqs))
print(seqs.str.len().describe().round(1))
Quick check: what share of customers shopped only once?
print(round((seqs.str.len() == 1).mean(), 3))
import matplotlib.pyplot as plt
seqs.str.len().clip(upper=20).plot.hist(bins=range(1, 22))
plt.title("How many times each customer ordered")
plt.xlabel("Baskets per customer (20 means 20 or more)")
plt.ylabel("Customers")
plt.show()
monthly = inv.set_index("date").resample("MS").size()
monthly.plot(marker="o")
plt.title("Baskets per month")
plt.xlabel("Month")
plt.ylabel("Baskets")
plt.show()
repeat = seqs[seqs.str.len() >= 2]
N = len(repeat)
print(N)
Does a sequence contain a pattern?#
Problem: check 'bought A, then B later' for any customer.
def contains(seq, pattern):
k = 0
for basket in seq:
if pattern[k] in basket:
k += 1
if k == len(pattern):
return True
return False
seq = seqs.loc[12347]
a, b = sorted(seq[0])[0], sorted(seq[-1])[0]
print(a, "|", b)
print(contains(seq, [a, b]), contains(seq, [b, a]))
def support(pattern):
hits = sum(contains(seq, pattern) for seq in repeat)
return round(hits / N, 4)
Quick check: cakestand then cake tins, or cake tins then cakestand: which order is more common?
Common mistake: passing the steps as separate arguments
stand, tins = "REGENCY CAKESTAND 3 TIER", "SET OF 3 REGENCY CAKE TINS"
support(stand, tins)
print(support([stand, tins]), support([tins, stand]))
same = sum(any(stand in b and tins in b for b in seq)
for seq in repeat)
print(round(same / N, 4))
Your turn: what share of repeat customers bought the white hanging heart T-light holder, then bought it again later?
print(support(["WHITE HANGING HEART T-LIGHT HOLDER"] * 2))
Mining every two-step pattern#
Problem: hundreds of popular products make hundreds of thousands of ordered pairs.
from collections import Counter
bought = Counter(i for seq in repeat for i in set().union(*seq))
item_sup = pd.Series(bought) / N
frequent = set(item_sup[item_sup >= 0.05].index)
print(len(item_sup), len(frequent))
One pass per customer: note each product's first and last basket. <A, B> is present when A's first basket comes before B's last.
pair_counts = Counter()
for seq in repeat:
first, last = {}, {}
for k, basket in enumerate(seq):
for item in basket & frequent:
first.setdefault(item, k)
last[item] = k
pair_counts.update((a, b) for a in first for b in last
if first[a] < last[b])
print(len(pair_counts))
pat = (pd.Series(pair_counts, name="support").div(N)
.rename_axis(["first", "then"]).reset_index())
pat = pat.sort_values(["support", "first", "then"],
ascending=[False, True, True])
print(pat.head().round(4).to_string(index=False))
Quick check: does the fast count agree with our contains test for cakestand then cake tins?
row = pat[(pat["first"] == stand) & (pat["then"] == tins)]
print(row["support"].round(4).tolist(), support([stand, tins]))
Two kinds of pattern: reorders (same product again) and cross-sells (a different product later).
reorders = pat[pat["first"] == pat["then"]]
cross = pat[pat["first"] != pat["then"]]
print(len(reorders), len(cross))
print(reorders[["first", "support"]].head().round(4).to_string(index=False))
top = reorders.head(10).iloc[::-1]
plt.barh(top["first"].str.title(), top["support"])
plt.title("Products customers come back for")
plt.xlabel("Share of repeat customers who reorder")
plt.ylabel("Product")
plt.show()
cross = cross.assign(
confidence=cross["support"] / cross["first"].map(item_sup),
lift=lambda t: t["confidence"] / t["then"].map(item_sup))
common = cross[cross["support"] >= 0.03]
best = common.sort_values("confidence", ascending=False, kind="stable")
for r in best.head(5).itertuples():
print(f"{r.first} -> {r.then}: {r.confidence:.2f}, {r.lift:.1f}")
plt.scatter(common["confidence"], common["lift"], s=12, alpha=0.6)
plt.title("Common cross-sells: reliable versus surprising")
plt.xlabel("Confidence")
plt.ylabel("Lift")
plt.show()
Your turn: what do customers buy after the pink alarm clock? Use the best table.
pink = best[best["first"] == "ALARM CLOCK BAKELIKE PINK"]
for r in pink.head(3).itertuples():
print(r.then, round(r.support, 4), round(r.confidence, 2))
Longer patterns with the Apriori trick#
Problem: three-step patterns multiply the candidates again.
If <A, B, C> is frequent, <A, B> and <B, C> must be frequent too. So we only chain frequent pairs.
top_pairs = cross[cross["support"] >= 0.05]
links = set(zip(top_pairs["first"], top_pairs["then"]))
cands = sorted((a, b, c) for a, b in links for b2, c in links
if b == b2 and c != a)
print(len(top_pairs), len(cands))
trip = pd.Series({c: support(list(c)) for c in cands})
trip = trip.sort_values(ascending=False, kind="stable")
for (a, b, c), s in trip.head(5).items():
print(f"{a} -> {b} -> {c}: {s}")
Quick check: can a three-step pattern ever be more common than the two-step patterns inside it?
a, b, c = trip.index[0]
print(support([a, b]), support([b, c]), support([a, b, c]))
Your turn: which three-step patterns start with the Regency cakestand?
from_stand = trip[[c[0] == stand for c in trip.index]]
print(len(from_stand))
for (a, b, c), s in from_stand.head(3).items():
print(f"{b} -> {c}: {s}")
Mini-project: when should the follow-up email go out?#
After a customer's first Regency cakestand, what do they buy next, and how many days later?
has_stand = inv["items"].apply(lambda s: stand in s)
first_buy = inv[has_stand].groupby("CustomerID")["date"].min()
later = inv.join(first_buy.rename("first_buy"), on="CustomerID",
how="inner")
later = later[later["date"] > later["first_buy"]]
print(len(first_buy), later["CustomerID"].nunique())
after = later.explode("items")
next_buys = after.groupby("items")["CustomerID"].nunique()
next_buys.sort_values(ascending=False).head(5)
def days_to(item):
hit = after[after["items"] == item]
first_hit = hit.groupby("CustomerID")["date"].min()
gap = first_hit - first_buy.loc[first_hit.index]
return gap.dt.days
reorder_gap, tins_gap = days_to(stand), days_to(tins)
print(len(reorder_gap), reorder_gap.median())
print(len(tins_gap), tins_gap.median())
print(round(len(tins_gap) / len(first_buy), 3))
print(round(item_sup[tins], 3))
plt.hist([reorder_gap, tins_gap], bins=12,
label=["Cakestand again", "Cake tins"])
plt.title("Days from first cakestand to the next purchase")
plt.xlabel("Days after first cakestand")
plt.ylabel("Customers")
plt.legend()
plt.show()
Finding: half of returning cakestand buyers (330 of 663) buy another one, after a median 73 days. The cake tins are the best cross-sell (206 customers), but later: a median 138.5 days. Caveat: one year of data cuts off longer gaps.
Recap#
- Sequences are each customer's baskets in time order; convert dates first
- A pattern <A, B> needs B in a later basket than A
- Support counts customers, not baskets
- First and last positions count every pair in one pass; the Apriori trick grows longer patterns
- Reorders and cross-sells, plus real time gaps, drive follow-up offers
Challenges#
- Repeat the mini-project for the white hanging heart T-light holder.
- Only count <A, B> when B comes within 90 days of A. Which patterns survive?
- Mine patterns for customers in one country, such as Germany, and compare.
What's next#
Lesson 36: Introduction to Anomaly and Outlier Detection.
Data Mining 36: Introduction to Anomaly and Outlier Detection#
In this lesson you'll be able to:
- Explain what an anomaly is, and the kinds you'll meet
- Flag outliers in one column with z-scores and the IQR fence
- Score whole rows with Isolation Forest and Local Outlier Factor
- Explain why a flagged row is unusual
- Check whether unusual means fraud, on real card data Datasets: Online Retail customers (data/pandas/online_retail.csv.gz) and card transactions (data/mining/creditcard.csv.gz)
Warm-up: sequences from lesson 35#
Rebuild the clean sales, and count baskets per customer again.
import numpy as np
import pandas as pd
df = pd.read_csv("online_retail.csv.gz")
sales = df.dropna(subset=["CustomerID"])
sales = sales[~sales["InvoiceNo"].str.startswith("C")]
keep = (sales["Quantity"] > 0) & sales["StockCode"].str.match(r"\d")
sales = sales[keep].copy()
baskets = sales.groupby("CustomerID")["InvoiceNo"].nunique()
print(baskets.median(), baskets.max(), baskets.idxmax())
Pattern mining looks for the common. Anomaly detection looks for the rare.
What is an anomaly?#
Problem: spot the rows that don't fit, before they mislead us.
- An anomaly (or outlier) is a row unlike the rest
- Causes: data errors, rare but real cases, or bad behaviour such as fraud
- Kinds: point (one odd value), contextual (odd for its time or place), collective (an odd group)
df["Quantity"].describe().round(1)
cols = ["InvoiceNo", "StockCode", "Quantity", "UnitPrice",
"CustomerID"]
big = df.nlargest(2, "Quantity")
big[cols]
Quick check: were these giant orders real sales, or something else?
same = df["StockCode"].isin(big["StockCode"])
same &= df["CustomerID"].isin(big["CustomerID"])
df.loc[same, cols + ["InvoiceDate"]]
price_cols = ["InvoiceNo", "StockCode", "Description", "UnitPrice"]
df.nlargest(3, "UnitPrice")[price_cols]
A first rule: how far from typical?#
Problem: flag customers whose numbers sit far from the crowd.
sales["Spend"] = sales["Quantity"] * sales["UnitPrice"]
cust = sales.groupby("CustomerID").agg(
baskets=("InvoiceNo", "nunique"), items=("Quantity", "sum"),
spend=("Spend", "sum"), products=("StockCode", "nunique"),
country=("Country", "first"))
cust["basket_value"] = cust["spend"] / cust["baskets"]
print(cust.shape)
cust.head(3).round(1)
Common mistake: computing z-scores on the whole table, text column included
z = (cust - cust.mean()) / cust.std()
num = cust.drop(columns="country")
z = (num - num.mean()) / num.std()
print((z.abs() > 3).sum())
s = cust["spend"]
print(round((s.loc[14646] - s.mean()) / s.std(), 1))
q1, q3 = cust["spend"].quantile([0.25, 0.75])
fence = q3 + 1.5 * (q3 - q1)
print(round(q1, 1), round(q3, 1), round(fence, 1))
print((cust["spend"] > fence).sum())
import matplotlib.pyplot as plt
cust["spend"].plot.box(logy=True)
plt.title("Spend per customer, log scale")
plt.ylabel("Total spend (pounds, log scale)")
plt.show()
Your turn: apply the same IQR fence to the items column. How many customers are above it?
i1, i3 = cust["items"].quantile([0.25, 0.75])
item_fence = i3 + 1.5 * (i3 - i1)
print(item_fence, (cust["items"] > item_fence).sum())
Quick check: why does the fence flag over ten times as many customers as z-scores?
print(cust["spend"].agg(["mean", "median", "std"]).round(1))
np.log10(cust["spend"] + 1).plot.hist(bins=40)
plt.title("Customer spend is heavily skewed")
plt.xlabel("log10 of total spend + 1 (pounds)")
plt.ylabel("Customers")
plt.show()
log_spend = np.log1p(cust["spend"])
zl = (log_spend - log_spend.mean()) / log_spend.std()
print((zl > 3).sum(), (zl < -3).sum())
Many features at once: Isolation Forest#
Problem: a customer can look normal in each column, but odd in the combination.
- Make random cuts through the data, like a game of twenty questions
- Odd rows get isolated in very few cuts; ordinary rows need many
- Average over many random trees for an anomaly score
Common mistake: handing the model the whole table, text and all
from sklearn.ensemble import IsolationForest
IsolationForest(random_state=42).fit(cust)
feats = ["baskets", "items", "spend", "products", "basket_value"]
X = np.log1p(cust[feats])
iso = IsolationForest(contamination=0.01, random_state=42)
cust["flag"] = iso.fit_predict(X) == -1
cust["score"] = -iso.score_samples(X)
print(cust["flag"].sum())
top = cust.nlargest(5, "score")
top[feats + ["score"]].round(2)
cust["score"].plot.hist(bins=40)
cutoff = cust.loc[cust["flag"], "score"].min()
plt.axvline(cutoff, color="red", linestyle="--")
plt.title("Contamination sets where the line is drawn")
plt.xlabel("Anomaly score (higher = stranger)")
plt.ylabel("Customers")
plt.show()
print(round(cutoff, 3))
ok, odd = cust[~cust["flag"]], cust[cust["flag"]]
plt.scatter(ok["baskets"], ok["spend"] + 1, s=6, c="grey",
alpha=0.4, label="ordinary")
plt.scatter(odd["baskets"], odd["spend"] + 1, s=20, c="red",
label="flagged")
plt.xscale("log")
plt.yscale("log")
plt.title("Isolation Forest flags the edges of the cloud")
plt.xlabel("Baskets (log scale)")
plt.ylabel("Total spend + 1 (log scale)")
plt.legend()
plt.show()
Your turn: are the flagged customers mostly big spenders or tiny ones? Compare with the median spend.
median_spend = cust["spend"].median()
print(round(median_spend, 1), (odd["spend"] > median_spend).sum(),
len(odd))
A neighbourhood view: Local Outlier Factor#
Problem: some rows are only odd compared with their nearest neighbours.
LOF compares how crowded each row's neighbourhood is with how crowded its neighbours' are.
from sklearn.neighbors import LocalOutlierFactor
from sklearn.preprocessing import StandardScaler
X_scaled = StandardScaler().fit_transform(X)
lof = LocalOutlierFactor(n_neighbors=20, contamination=0.01)
cust["lof_flag"] = lof.fit_predict(X_scaled) == -1
print(cust["lof_flag"].sum())
cust["lof_score"] = -lof.negative_outlier_factor_
print(cust["lof_score"].describe().round(2))
Quick check: do Isolation Forest and LOF flag the same customers?
pd.crosstab(cust["flag"], cust["lof_flag"])
lof_only = cust[cust["lof_flag"] & ~cust["flag"]]
lof_only.sort_values("spend")[feats].head().round(1)
plt.scatter(cust["score"], cust["lof_score"], s=6, alpha=0.5)
plt.title("Two detectors, two opinions")
plt.xlabel("Isolation Forest score")
plt.ylabel("LOF score")
plt.show()
Your turn: rerun LOF with 50 neighbours. How many of its flags agree with Isolation Forest now?
lof50 = LocalOutlierFactor(n_neighbors=50, contamination=0.01)
flag50 = lof50.fit_predict(X_scaled) == -1
print(flag50.sum(), (flag50 & cust["flag"]).sum())
Explaining an anomaly#
Problem: a flag is useless until someone can say why the row is odd.
odd_ones = [14646, 12346, 13256]
print(cust.loc[odd_ones, "flag"].tolist())
ratio = cust.loc[odd_ones, feats] / cust[feats].median()
ratio.round(2)
sales.loc[sales["CustomerID"] == 13256,
["InvoiceDate", "Description", "Quantity", "UnitPrice"]]
- 14646: a genuine giant, a wholesale account to look after
- 12346: a data error: an order that was cancelled, still in our clean sales
- 13256: 12,540 items at a price of zero: a pricing or free-goods question
import os
flagged = cust[cust["flag"]].sort_values("score", ascending=False)
flagged.to_csv("m36_flagged_customers.csv")
back = pd.read_csv("m36_flagged_customers.csv")
print(back.shape, back["CustomerID"].head(3).tolist())
Mini-project: does unusual mean fraud?#
284,807 real card transactions, 492 of them fraud. Features V1 to V28 are anonymised by the bank. Fit Isolation Forest without labels, then use the labels only to check.
cc = pd.read_csv("creditcard.csv.gz")
features = [f"V{i}" for i in range(1, 29)] + ["Amount"]
print(cc.shape)
print(cc["Class"].sum(), round(cc["Class"].mean(), 4))
iso_cc = IsolationForest(random_state=42)
iso_cc.fit(cc[features])
cc["score"] = -iso_cc.score_samples(cc[features])
top = cc.nlargest(len(cc) // 100, "score")
print(len(top), top["Class"].sum(), round(top["Class"].mean(), 3))
for share in [0.001, 0.005, 0.01, 0.05]:
review = cc.nlargest(int(len(cc) * share), "score")
caught = review["Class"].sum()
print(share, len(review), caught, round(caught / 492, 2))
Quick check: how many frauds would a random 1% sample of transactions catch?
random_pick = cc.sample(frac=0.01, random_state=42)
print(len(random_pick), random_pick["Class"].sum())
big_amounts = cc.nlargest(len(cc) // 100, "Amount")
print(len(big_amounts), big_amounts["Class"].sum())
normal, fraud = cc[cc["Class"] == 0], cc[cc["Class"] == 1]
plt.hist([normal["score"], fraud["score"]], bins=40, density=True,
label=["genuine", "fraud"])
plt.title("Fraud scores higher, but the groups overlap")
plt.xlabel("Anomaly score (higher = stranger)")
plt.ylabel("Density")
plt.legend()
plt.show()
Finding: reviewing the strangest 1% (2,848 transactions) catches 291 of 492 frauds, 59%. Fraud is 10.2% of that list against 0.17% overall, but 9 in 10 flags are still genuine.
Recap#
- An anomaly is a row unlike the rest: an error, a rare case, or fraud
- Z-scores and the IQR fence work on one column; mind skewed data
- Isolation Forest isolates odd rows in few random cuts
- LOF compares a row's neighbourhood with its neighbours'
- Unusual is not the same as bad: explain flags, and check them against known cases
Challenges#
- Rebuild the customer table without the two cancelled giant orders. Who is flagged now?
- Run LOF on the card data (use a sample of 20,000 rows, random_state=42).
- Flag outliers in Quantity by country. Is 1,000 items normal everywhere?
What's next#
Lesson 37: Statistical Methods for Outlier Detection.
Data Mining 37: Statistical Methods for Outlier Detection#
In this lesson you'll be able to:
- Flag outliers with z-scores, and spot when they mislead
- Draw IQR fences and boxplots on skewed money data
- Build robust z-scores from the median and MAD
- Catch outliers in combinations with Mahalanobis distance Datasets: card transactions (data/mining/creditcard.csv.gz) and New York taxi trips (data/pandas/taxis.csv)
Warm-up: rare events, from lesson 36#
Lesson 36: anomalies are rare and come in types. Today we measure 'unusual'.
import numpy as np
import pandas as pd
cards = pd.read_csv("creditcard.csv.gz")
print(cards.shape)
cards[["Time", "V1", "V2", "Amount", "Class"]].head(3).round(2)
V1 to V28 are anonymised PCA components; Time is seconds from the first payment.
print(cards["Class"].value_counts())
print(round(cards["Class"].mean() * 100, 3))
Z-scores: distance from the average#
Problem: is a 2,000 dollar payment unusual? We need a common yardstick.
- z = (value - mean) / standard deviation
- It says how many standard deviations a value sits from the mean
- The classic rule flags anything beyond 3 either way
amount = cards["Amount"]
amount.describe().round(2)
Common mistake: forgetting the brackets after .std
z = (amount - amount.mean()) / amount.std
z = (amount - amount.mean()) / amount.std()
flagged = z.abs() > 3
print(flagged.sum())
print(round(amount[flagged].min(), 2))
Quick check: are these big payments more likely to be fraud?
rate_all = cards["Class"].mean() * 100
rate_flag = cards.loc[flagged, "Class"].mean() * 100
print(round(rate_all, 3), round(rate_flag, 3))
cards.groupby("Class")["Amount"].median()
Why skewed data fools the z-score#
Problem: a few huge payments drag the mean and standard deviation with them.
print(round(amount.skew(), 1))
amount.nlargest(5)
trimmed = amount[amount < amount.quantile(0.999)]
print(round(amount.std(), 2), round(trimmed.std(), 2))
Money is skewed: most payments are small, a few are huge. A log scale evens it out.
log_amt = np.log1p(amount)
print(round(log_amt.skew(), 2))
import matplotlib.pyplot as plt
ax = log_amt.plot.hist(bins=60, figsize=(7, 4))
ax.set_title("Card payments on a log scale")
ax.set_xlabel("log(1 + amount)")
ax.set_ylabel("Number of payments")
plt.show()
Your turn: z-scores on log_amt. How many are flagged high, how many low?
z_log = (log_amt - log_amt.mean()) / log_amt.std()
print((z_log > 3).sum(), (z_log < -3).sum())
line = np.expm1(log_amt.mean() + 3 * log_amt.std())
print(round(line, 2))
IQR fences and the boxplot#
Problem: we want a rule built from quartiles, which extremes can't drag.
- IQR = Q3 - Q1, the spread of the middle half
- Tukey's fences: below Q1 - 1.5 IQR or above Q3 + 1.5 IQR
q1, q3 = amount.quantile([0.25, 0.75])
iqr = q3 - q1
upper = q3 + 1.5 * iqr
print(round(q1, 2), round(q3, 2), round(upper, 2))
over = amount > upper
print(over.sum(), round(over.mean() * 100, 1))
Quick check: fences on log_amt instead. More flags, or fewer?
lq1, lq3 = log_amt.quantile([0.25, 0.75])
l_upper = lq3 + 1.5 * (lq3 - lq1)
l_over = log_amt > l_upper
print(l_over.sum(), round(np.expm1(l_upper), 2))
import seaborn as sns
kind = cards["Class"].map({0: "normal", 1: "fraud"})
ax = sns.boxplot(x=log_amt, y=kind)
ax.set_title("Payment size by class (log scale)")
ax.set_xlabel("log(1 + amount)")
ax.set_ylabel("")
plt.show()
Robust z-scores with the median and MAD#
Problem: mean and standard deviation bend towards outliers. The median and MAD barely move.
- MAD = median of |value - median|
- robust z = 0.6745 * (value - median) / MAD, flag beyond 3.5
Quick check: drop the top 0.1% of payments. Which measures move?
mad_all = (amount - amount.median()).abs().median()
mad_trim = (trimmed - trimmed.median()).abs().median()
print(round(amount.mean(), 2), round(trimmed.mean(), 2))
print(amount.median(), trimmed.median())
print(round(mad_all, 2), round(mad_trim, 2))
Common mistake: calling .mad() from an old tutorial
amount.mad()
def robust_z(s):
med = s.median()
mad = (s - med).abs().median()
return 0.6745 * (s - med) / mad
print((robust_z(amount).abs() > 3.5).sum())
rz_log = robust_z(log_amt)
big = rz_log.abs() > 3.5
print(big.sum())
amount[big]
summary = pd.Series({
"z-score": flagged.sum(), "z-score on log": (z_log > 3).sum(),
"IQR fences": over.sum(), "IQR on log": l_over.sum(),
"robust z on log": big.sum()})
summary.sort_values()
Which column hides the fraud?#
Problem: big amounts were not the fraud signal. Maybe another column is.
Choose on day 1 (the first 24 hours), keep day 2 untouched for the mini-project.
day1 = cards[cards["Time"] < 86400]
day2 = cards[cards["Time"] >= 86400]
print(len(day1), day1["Class"].sum())
print(len(day2), day2["Class"].sum())
v_cols = [f"V{i}" for i in range(1, 29)]
flags = day1[v_cols].apply(robust_z).abs() > 3.5
is_fraud = day1["Class"] == 1
report = pd.DataFrame({"flagged": flags.sum(),
"frauds": flags[is_fraud].sum()})
report["fraud_rate"] = report["frauds"] / report["flagged"]
report.sort_values("frauds", ascending=False).head().round(3)
Quick check: how many times richer in fraud is V14's flag list than day 1 overall?
base1 = day1["Class"].mean()
print(round(report.loc["V14", "fraud_rate"] / base1, 1))
ax = day1.loc[~is_fraud, "V14"].plot.hist(bins=80, density=True,
alpha=0.6, label="normal")
day1.loc[is_fraud, "V14"].plot.hist(bins=40, density=True, ax=ax,
alpha=0.6, label="fraud")
ax.set_title("V14 on day 1: frauds sit far to the left")
ax.set_xlabel("V14 (anonymised component)")
ax.set_ylabel("Density")
ax.legend()
plt.show()
Mahalanobis distance: odd combinations#
Problem: some records look normal in each column, but odd as a pair.
taxis = pd.read_csv("taxis.csv")
trips = taxis[["distance", "fare"]]
print(len(trips))
print(trips.corr().round(2))
- Mahalanobis distance measures distance from the centre in units that follow the correlation
- Squared distances follow a chi-square curve, so we can set a 99.9% cut-off
X = trips.to_numpy()
centre = X.mean(axis=0)
cov_inv = np.linalg.inv(np.cov(X, rowvar=False))
diff = X - centre
print(centre.round(2))
Common mistake: multiplying the whole table at once with @
d2 = diff @ cov_inv @ diff
d2 = (diff @ cov_inv * diff).sum(axis=1)
taxis["d2"] = d2
from scipy.stats import chi2
cutoff = chi2.ppf(0.999, df=2)
joint = taxis["d2"] > cutoff
print(round(cutoff, 2), joint.sum())
z_trip = (trips - trips.mean()) / trips.std()
single = (z_trip.abs() > 3).any(axis=1)
hidden = joint & ~single
print(hidden.sum())
taxis.loc[hidden, ["distance", "fare", "pickup_zone"]]
ax = taxis.plot.scatter(x="distance", y="fare", s=6, alpha=0.3,
color="grey", figsize=(7, 5))
taxis[joint].plot.scatter(x="distance", y="fare", s=14, ax=ax,
color="red", label="Mahalanobis flag")
ax.set_title("Taxi trips: fare against distance")
ax.set_xlabel("Distance (miles)")
ax.set_ylabel("Fare (dollars)")
plt.show()
Your turn: list the three trips with the largest Mahalanobis distance.
cols = ["distance", "fare", "total", "d2"]
taxis.nlargest(3, "d2")[cols].round(1)
Mini-project: a statistical fraud alarm for day 2#
Fit Mahalanobis on day 1's four best columns. How much of day 2's fraud does it catch?
watch = ["V14", "V17", "V12", "V10"]
ref = day1[watch].to_numpy()
c_mean = ref.mean(axis=0)
c_inv = np.linalg.inv(np.cov(ref, rowvar=False))
gap = day2[watch].to_numpy() - c_mean
scores = (gap @ c_inv * gap).sum(axis=1)
print(len(scores), round(np.median(scores), 2))
alarm = scores > chi2.ppf(0.999, df=4)
caught = day2.loc[alarm, "Class"].sum()
print(alarm.sum(), caught, day2["Class"].sum())
print(round(caught / alarm.sum() * 100, 1))
A fraud team can only review so many cases. What if they check just the top 500?
top500 = np.argsort(-scores)[:500]
in_top = day2["Class"].iloc[top500].sum()
print(in_top, round(in_top / 500 * 100, 1))
recall = caught / day2["Class"].sum()
lift = (caught / alarm.sum()) / day2["Class"].mean()
print(f"Alarm catches {recall:.0%} of day-2 fraud; "
f"alerts are {lift:.0f}x richer in fraud than average")
Recap#
- z-scores measure distance from the mean in standard deviations
- Skewed data inflates the mean and std: take logs first
- IQR fences and robust z-scores (median, MAD) resist the extremes
- Unusual is not the same as fraud: check flags against outcomes
- Mahalanobis distance finds odd combinations, with a chi-square cut-off
Challenges#
- Use robust z-scores on log taxi fares. Which trips get flagged?
- Rebuild the day-2 alarm with only V14 and V17. What changes?
- Try a 99% chi-square cut-off instead of 99.9%. Count the extra alerts.
What's next#
Lesson 38: Distance and Density-Based Outlier Detection No bell curves needed: kNN distances, LOF and DBSCAN.
Data Mining 38: Distance and Density-Based Outlier Detection#
In this lesson you'll be able to:
- Score outliers by the distance to their k nearest neighbours
- Explain why scaling and the choice of k change the answer
- Use the Local Outlier Factor, including on brand-new rows
- Find noise points with DBSCAN and compare detectors fairly Dataset: card transactions (data/mining/creditcard.csv.gz), a 20,000-row random sample
Warm-up: lesson 37's robust z-score on V14#
Lesson 37 flagged values far from the median. One column at a time, though.
import numpy as np
import pandas as pd
cards = pd.read_csv("creditcard.csv.gz")
sample = cards.sample(20000, random_state=42)
print(len(sample), sample["Class"].sum())
v14 = sample["V14"]
med = v14.median()
mad = (v14 - med).abs().median()
rz = 0.6745 * (v14 - med) / mad
flag = rz.abs() > 3.5
print(flag.sum(), sample.loc[flag, "Class"].sum())
The intuition: outliers have no close neighbours#
Problem: an odd record can look fine in each column, but sit alone in the full space.
import matplotlib.pyplot as plt
normal = sample[sample["Class"] == 0]
fraud = sample[sample["Class"] == 1]
ax = normal.plot.scatter(x="V14", y="V17", s=4, alpha=0.3,
color="grey", label="normal")
fraud.plot.scatter(x="V14", y="V17", s=20, color="red", ax=ax,
label="fraud")
ax.set_title("Frauds sit away from the crowd")
ax.set_xlabel("V14")
ax.set_ylabel("V17")
plt.show()
- For each point, find its k nearest neighbours
- Its outlier score is the distance to the k-th one
- Crowded points score low; lonely points score high
kNN distance scores#
Problem: turn 'far from everyone' into one number per payment.
raw_cols = [f"V{i}" for i in range(1, 29)] + ["Amount"]
X_raw = sample[raw_cols].to_numpy()
y = sample["Class"].to_numpy()
print(X_raw.shape)
Common mistake: asking for neighbours before fitting
from sklearn.neighbors import NearestNeighbors
nn = NearestNeighbors(n_neighbors=6)
dist, _ = nn.kneighbors(X_raw)
nn.fit(X_raw)
dist, _ = nn.kneighbors(X_raw)
print(dist[:3].round(2))
knn_raw = dist[:, -1]
top = np.argsort(-knn_raw)[:200]
print(y[top].sum())
print(sample["Amount"].iloc[top[:5]].round(2).to_list())
Quick check: are those five biggest scorers frauds?
print(y[top[:5]])
Scale first: distances need fair units#
Problem: Amount is in dollars, V columns are near 1, so Amount drowns them out.
sample[["V1", "V14", "V28", "Amount"]].std().round(2)
from sklearn.preprocessing import StandardScaler
feats = sample[raw_cols].copy()
feats["Amount"] = np.log1p(feats["Amount"])
X = StandardScaler().fit_transform(feats)
print(X.std(axis=0).round(2)[[0, 13, 28]])
def knn_score(X, k):
nn = NearestNeighbors(n_neighbors=k + 1).fit(X)
return nn.kneighbors(X)[0][:, -1]
def hits(score, n=200):
return int(y[np.argsort(-score)[:n]].sum())
knn5 = knn_score(X, 5)
print(hits(knn_raw), hits(knn5))
pd.Series(knn5).groupby(y).median().round(2)
Quick check: can we rebuild one fraud's score by hand?
i = np.where(y == 1)[0][0]
gaps = np.linalg.norm(X - X[i], axis=1)
print(np.sort(gaps)[:6].round(2))
print(round(knn5[i], 2))
Your turn: try k = 20. More frauds in the top 200, or fewer?
knn20 = knn_score(X, 20)
print(hits(knn20), hits(knn20, 100))
Your turn: scale, but skip the log on Amount. How many frauds in the top 200?
X_nolog = StandardScaler().fit_transform(sample[raw_cols])
print(hits(knn_score(X_nolog, 20)), hits(knn20))
Local Outlier Factor: density compared with neighbours#
Problem: a fixed distance can't serve dense and sparse regions alike.
- LOF compares a point's local density with its neighbours' density
- Around 1: as dense as its neighbours. Well above 1: an outlier
from sklearn.neighbors import LocalOutlierFactor
lof = LocalOutlierFactor(n_neighbors=20)
labels = lof.fit_predict(X)
lof20 = -lof.negative_outlier_factor_
print((labels == -1).sum(), y[labels == -1].sum())
print(hits(lof20))
Quick check: why might LOF with 20 neighbours miss frauds?
_, idx = NearestNeighbors(n_neighbors=11).fit(X).kneighbors(X)
fraud_rows = np.where(y == 1)[0]
share = y[idx[fraud_rows, 1:]].mean()
print(round(share, 2), round(y.mean(), 4))
lof = LocalOutlierFactor(n_neighbors=50)
lof.fit(X)
lof50 = -lof.negative_outlier_factor_
print(hits(lof50), hits(lof50, 100))
print(pd.Series(lof50).groupby(y).median().round(2))
high = lof50 > 1.5
print(high.sum(), y[high].sum())
ax = pd.Series(lof50).groupby(y).plot.hist(
bins=np.geomspace(0.9, 10, 40), alpha=0.6, density=True,
legend=True, figsize=(7, 4))[0]
ax.set_xscale("log")
ax.set_title("LOF scores: 0 = normal, 1 = fraud")
ax.set_xlabel("LOF score (log scale)")
ax.set_ylabel("Density")
plt.show()
Common mistake: calling predict on a plain LOF to score new payments
lof.predict(X[:3])
Fix: novelty=True. Score real rows held back from the sample.
rest = cards.drop(sample.index)
new = rest.groupby("Class").head(3)
new_feats = new[raw_cols].copy()
new_feats["Amount"] = np.log1p(new_feats["Amount"])
print(new["Class"].to_list())
scaler = StandardScaler().fit(feats)
lof_new = LocalOutlierFactor(n_neighbors=50, novelty=True)
lof_new.fit(scaler.transform(feats))
X_new = scaler.transform(new_feats)
print(lof_new.predict(X_new))
print((-lof_new.score_samples(X_new)).round(2))
DBSCAN: points that belong to no crowd#
Problem: we want a yes or no flag, with no need to choose how many to flag.
- DBSCAN grows clusters from dense cores (lesson 30)
- Points within eps of no core are labelled -1: noise
d10 = np.sort(knn_score(X, 10))
ax = pd.Series(d10).plot(figsize=(7, 4))
ax.set_title("Distance to the 10th neighbour, sorted")
ax.set_xlabel("Payments, sorted")
ax.set_ylabel("Distance")
plt.show()
print(np.percentile(d10, [50, 95, 99]).round(2))
from sklearn.cluster import DBSCAN
db = DBSCAN(eps=5, min_samples=10).fit(X)
noise = db.labels_ == -1
print(db.labels_.max() + 1, noise.sum(), y[noise].sum())
Your turn: keep eps = 5, but require 30 neighbours for a core point.
db30 = DBSCAN(eps=5, min_samples=30).fit(X)
noise30 = db30.labels_ == -1
print(noise30.sum(), y[noise30].sum())
knn_top = np.argsort(-knn20)[:noise.sum()]
shared = np.isin(np.where(noise)[0], knn_top).sum()
print(noise.sum(), shared, y[knn_top].sum())
Quick check: what share of DBSCAN's noise points are fraud?
print(round(y[noise].mean() * 100, 1), round(y.mean() * 100, 2))
Comparing detectors on a fair alert budget#
Problem: each method flags a different number of rows. Fix the budget, then compare.
board = pd.Series({
"robust z on V14": hits(rz.abs().to_numpy()),
"kNN raw, k=5": hits(knn_raw), "kNN scaled, k=20": hits(knn20),
"LOF k=20": hits(lof20), "LOF k=50": hits(lof50)})
board.sort_values(ascending=False)
ax = board.sort_values().plot(kind="barh", figsize=(7, 4))
ax.set_title("Frauds found in a 200-payment review list")
ax.set_xlabel("Frauds in the top 200 (of 34)")
ax.set_ylabel("")
plt.show()
(board / 200 * 100).round(1).sort_values(ascending=False)
Mini-project: do the detectors agree?#
Combine kNN and LOF by rank. Does the blend beat each one on a 100-payment budget?
top_knn = set(np.argsort(-knn20)[:100])
top_lof = set(np.argsort(-lof50)[:100])
both = top_knn & top_lof
print(len(both), int(y[list(both)].sum()))
ranks = pd.DataFrame({"knn": knn20, "lof": lof50}).rank(pct=True)
blend = ranks.mean(axis=1).to_numpy()
print(hits(knn20, 100), hits(lof50, 100), hits(blend, 100))
review = sample.iloc[np.argsort(-blend)[:100]]
review.groupby("Class")["Amount"].agg(["count", "median"]).round(2)
A fraud team asks: how many reviews to catch half the frauds? Three quarters?
caught = np.cumsum(y[np.argsort(-blend)])
half = np.argmax(caught >= y.sum() / 2) + 1
three_q = np.argmax(caught >= y.sum() * 0.75) + 1
print(half, three_q)
ax = pd.Series(caught[:1000]).plot(figsize=(7, 4))
ax.axhline(y.sum(), color="red", linestyle="--")
ax.set_title("Frauds caught as the team works down the list")
ax.set_xlabel("Payments reviewed")
ax.set_ylabel("Frauds caught")
plt.show()
found = hits(blend, 100)
random_hits = 100 * y.mean()
print(f"Blend: {found} of {y.sum()} frauds in 100 reviews; "
f"random picks would find {random_hits:.1f}")
Recap#
- kNN distance: lonely points have far-away neighbours
- Scale (and log skewed columns) before any distance method
- LOF compares local densities; k must outgrow small fraud clusters
- Use novelty=True to score new rows; DBSCAN labels noise as -1
- Compare detectors on the same alert budget
Challenges#
- Run LOF with k = 100. Does it beat k = 50?
- Try DBSCAN with eps = 4 and eps = 6. Count noise and frauds.
- Drop Amount from the features. Which detector suffers most?
What's next#
Lesson 39: Applications in Fraud Detection and Cybersecurity Rules, scores and review queues, plus spotting phishing text.
Data Mining 39: Applications in Fraud Detection and Cybersecurity#
In this lesson you'll be able to:
- Turn domain knowledge into simple rules, and measure their lift
- Score payments with an Isolation Forest, trained without labels
- Size a review queue by budget, in frauds and in dollars
- Catch phishing texts with red-flag rules and novelty detection Datasets: card transactions (data/mining/creditcard.csv.gz) and SMS messages (data/mining/sms_spam.csv)
Warm-up: from lesson 38's review lists#
Lesson 38 ranked payments by loneliness and counted frauds in a top-200 list.
import numpy as np
import pandas as pd
cards = pd.read_csv("creditcard.csv.gz")
day1 = cards[cards["Time"] < 86400]
day2 = cards[cards["Time"] >= 86400]
print(day1["Class"].sum(), day2["Class"].sum())
- Signals: rules from experience, and scores from models
- A queue: alerts ranked so analysts review the worst first
- A budget: how many alerts the team can check each day
Rules first: when does fraud happen?#
Problem: fraud teams start with rules they can explain. Is any hour riskier?
cards["hour"] = (cards["Time"] // 3600) % 24
by_hour = cards.groupby("hour")["Class"].agg(["mean", "count"])
by_hour["mean"] = by_hour["mean"] * 100
by_hour.sort_values("mean", ascending=False).head().round(2)
Honest note: hour 0 is when recording began, not midnight. Quiet hours are likely night.
import matplotlib.pyplot as plt
ax = by_hour["mean"].plot(kind="bar", figsize=(8, 4))
ax.set_title("Fraud rate by hour since recording began")
ax.set_xlabel("Hour")
ax.set_ylabel("Fraud rate (%)")
plt.show()
Common mistake: combining two conditions with and
quiet = cards[(cards["hour"] >= 1) and (cards["hour"] <= 5)]
quiet = cards["hour"].between(1, 5)
rate_q = cards.loc[quiet, "Class"].mean()
lift = rate_q / cards["Class"].mean()
print(quiet.sum(), cards.loc[quiet, "Class"].sum())
print(round(rate_q * 100, 2), round(lift, 1))
Quick check: what share of all 492 frauds does the quiet-hours rule catch?
share = cards.loc[quiet, "Class"].sum() / cards["Class"].sum()
print(round(share * 100, 1))
Your turn: what share of ALL payments does the rule flag?
print(round(quiet.mean() * 100, 1))
Isolation Forest: anomaly scores without labels#
Problem: labels arrive late, and new fraud looks like nothing seen before.
- Grow random trees that split on random columns at random values
- Odd points get isolated in very few splits; ordinary points need many
- Short average path = high anomaly score
from sklearn.ensemble import IsolationForest
feats = [f"V{i}" for i in range(1, 29)] + ["Amount"]
iso = IsolationForest(n_estimators=200, random_state=42)
iso.fit(day1[feats])
score = -iso.score_samples(day2[feats])
print(len(score), score.min().round(3), score.max().round(3))
y2 = day2["Class"].to_numpy()
order = np.argsort(-score)
print(y2[order[:100]].sum())
cols = ["Amount", "V14", "V17", "Class"]
day2.iloc[order[:5]][cols].round(2)
ax = pd.Series(score[y2 == 0]).plot.hist(bins=60, density=True,
alpha=0.6, label="normal")
pd.Series(score[y2 == 1]).plot.hist(bins=30, density=True, ax=ax,
alpha=0.6, label="fraud")
ax.set_title("Isolation Forest scores on day 2")
ax.set_xlabel("Anomaly score (higher = stranger)")
ax.set_ylabel("Density")
ax.legend()
plt.show()
print(np.median(score[y2 == 0]).round(3))
print(np.median(score[y2 == 1]).round(3))
Quick check: the forest also has predict. How many payments does it flag by default?
flag_iso = iso.predict(day2[feats]) == -1
print(flag_iso.sum(), y2[flag_iso].sum())
Your turn: how many frauds are in the top 1,000? How does that compare with random?
print(y2[order[:1000]].sum())
print(round(1000 * y2.mean(), 1))
The review queue: budgets, frauds and dollars#
Problem: managers ask 'if we fund N reviews a day, what do we catch?'
amt2 = day2["Amount"].to_numpy()
fraud_dollars = amt2[y2 == 1].sum()
print(y2.sum(), round(fraud_dollars, 2))
rows = []
for budget in [100, 500, 1000, 2000]:
picked = order[:budget]
hit = y2[picked] == 1
rows.append({"budget": budget, "frauds": hit.sum(),
"dollars": amt2[picked][hit].sum()})
queue = pd.DataFrame(rows)
queue["hit_rate"] = queue["frauds"] / queue["budget"]
queue["dollar_share"] = queue["dollars"] / fraud_dollars
queue.round(2)
Your turn: how many reviews protect half of the fraud dollars?
saved = np.cumsum(amt2[order] * y2[order])
print(np.argmax(saved >= fraud_dollars / 2) + 1)
Quick check: going from 1,000 to 2,000 reviews, what happens to the hit rate?
extra = queue.set_index("budget").loc[[1000, 2000], "frauds"]
print(extra.diff().iloc[-1], round(extra.diff().iloc[-1] / 1000, 3))
ax = queue.plot(x="budget", y="dollar_share", marker="o",
legend=False, figsize=(7, 4))
ax.set_title("Fraud dollars protected by review budget")
ax.set_xlabel("Reviews per day")
ax.set_ylabel("Share of fraud dollars caught")
plt.show()
night2 = cards.loc[day2.index, "hour"].between(1, 5).to_numpy()
top1000 = order[:1000]
print(night2[top1000].sum(), y2[top1000][night2[top1000]].sum())
Quick check: inside the top 1,000, are quiet-hour alerts more often fraud?
in_q = night2[top1000]
print(round(y2[top1000][in_q].mean(), 2))
print(round(y2[top1000][~in_q].mean(), 2))
Cybersecurity: red flags in phishing texts#
Problem: scam texts push you to call a number or tap a link. Can those red flags catch them?
sms = pd.read_csv("sms_spam.csv")
sms["spam"] = (sms["label"] == "spam").astype(int)
print(sms["label"].value_counts())
print(sms.loc[sms["spam"] == 1, "message"].iloc[0][:75])
text = sms["message"]
sms["phone"] = text.str.contains(r"\d{5,}").astype(int)
link = r"http|www\.|\.com|\.co\.uk"
sms["link"] = text.str.contains(link, case=False).astype(int)
money = r"cash|prize|claim|\bwon\b"
sms["money"] = text.str.contains(money, case=False).astype(int)
flags = ["phone", "link", "money"]
report = pd.DataFrame({
"flagged": sms[flags].sum(),
"spam_caught": sms.loc[sms["spam"] == 1, flags].sum()})
report["spam_rate"] = report["spam_caught"] / report["flagged"]
report.round(3)
fp = sms[(sms["phone"] == 1) & (sms["spam"] == 0)]
print(len(fp))
fp["message"].str[:60].to_list()
Quick check: do the phone and link flags fire on the same messages?
pd.crosstab(sms["phone"], sms["link"])
Your turn: flag a message if it has a phone number OR a link. How much spam is caught?
either = (sms["phone"] == 1) | (sms["link"] == 1)
caught = sms.loc[either, "spam"].sum()
print(either.sum(), caught, sms["spam"].sum())
Novelty detection: learn normal, flag the rest#
Problem: attackers change wording. Learn only from genuine messages, so new scams still stand out.
sms["length"] = text.str.len()
sms["digits"] = text.str.count(r"\d")
sms["capitals"] = text.str.count(r"[A-Z]")
shape = ["length", "digits", "capitals"] + flags
sms.groupby("label")[shape].median()
from sklearn.model_selection import train_test_split
train, test = train_test_split(sms, test_size=0.3,
stratify=sms["spam"], random_state=42)
ham_train = train[train["spam"] == 0]
print(len(ham_train), len(test), test["spam"].sum())
Common mistake: feeding the raw message text to the model
IsolationForest(random_state=42).fit(ham_train[["message"]])
guard = IsolationForest(n_estimators=200, random_state=42)
guard.fit(ham_train[shape])
y_test = test["spam"].to_numpy()
odd = -guard.score_samples(test[shape])
rank = np.argsort(-odd)
print(y_test[rank[:y_test.sum()]].sum(), y_test.sum())
flag_guard = guard.predict(test[shape]) == -1
print(flag_guard.sum(), y_test[flag_guard].sum())
ax = pd.Series(odd).groupby(y_test).plot.hist(
bins=40, alpha=0.6, density=True, legend=True, figsize=(7, 4))[0]
ax.set_title("Message anomaly scores: 0 = ham, 1 = spam")
ax.set_xlabel("Anomaly score (higher = stranger)")
ax.set_ylabel("Density")
plt.show()
Quick check: which spam texts look most normal to the guard?
calm = test.iloc[rank[::-1]]
calm = calm[calm["spam"] == 1]["message"]
calm.str[:60].head(3).to_list()
Mini-project: a layered defence for text messages#
Layer 1: the red-flag rule. Layer 2: the anomaly score, for what the rule lets through.
rule = ((test["phone"] == 1) | (test["link"] == 1)).to_numpy()
print(rule.sum(), y_test[rule].sum())
missed = y_test[~rule].sum()
print(missed, (~rule).sum())
rest = np.where(~rule)[0]
worst = rest[np.argsort(-odd[rest])][:50]
print(y_test[worst].sum())
caught = y_test[rule].sum() + y_test[worst].sum()
false_alarms = (y_test[rule] == 0).sum() + (y_test[worst] == 0).sum()
print(caught, y_test.sum(), false_alarms, len(test))
print(f"Two layers stop {caught / y_test.sum():.0%} of spam, with "
f"{false_alarms} false alarms in {len(test)} messages")
Recap#
- Detection systems combine rules, scores, a queue and a budget
- Simple rules are explainable; judge them by lift and by share caught
- Isolation Forest isolates odd points in few splits, no labels needed
- Report queues in business terms: frauds and dollars per budget
- In security, learn normal and layer rules with anomaly scores
Challenges#
- Add the quiet-hours flag to the forest's features. Does the top 1,000 improve?
- Raise layer two to 100 reviews. Is the extra spam worth the extra false alarms?
- Add an exclamation-mark count to the message shape. Does the guard get better?
What's next#
Lesson 40: Hands-on Project- Credit Card Fraud Detection Train supervised models on heavily imbalanced data, judged by precision and recall.
Data Mining 40: Hands-on Project - Credit Card Fraud Detection#
In this lesson you'll be able to:
- Measure class imbalance and see why accuracy misleads
- Clean and split a large, rare-event dataset honestly
- Judge a fraud model with precision, recall and the confusion matrix
- Tackle imbalance with class weights and undersampling
- Turn model results into money saved and alerts raised Dataset: 284,807 real European card transactions from September 2013 (data/mining/creditcard.csv.gz)
Warm-up: a simple fraud rule#
Lesson 39 met rule-based fraud flags. Try one: flag the biggest 1% of amounts.
import pandas as pd
df = pd.read_csv("creditcard.csv.gz")
print(df.shape)
df[["Time", "V1", "V2", "Amount", "Class"]].head(3)
cut = df["Amount"].quantile(0.99)
flagged = df[df["Amount"] > cut]
print(round(cut, 2), len(flagged), flagged["Class"].sum())
print(df["Class"].sum())
How rare is fraud?#
Problem: before modelling, we need to know how unbalanced the two classes are.
- V1 to V28: anonymised PCA components (the bank hid the real fields)
- Time: seconds since the first transaction in the file
- Amount: payment value; Class: 1 = fraud, 0 = normal
df["Class"].value_counts()
fraud_rate = df["Class"].mean() * 100
print(f"Fraud rate: {fraud_rate:.3f}%")
Quick check: a lazy model says never fraud for every transaction. What accuracy does it score?
lazy_accuracy = (df["Class"] == 0).mean()
print(round(lazy_accuracy, 4))
Check data quality before modelling#
Problem: hidden repeats could leak the same transaction into both training and testing.
print(df.isna().sum().sum())
dupes = df.duplicated()
print(dupes.sum(), df.loc[dupes, "Class"].sum())
cols = ["Time", "V1", "V2", "Amount", "Class"]
df.loc[df.duplicated(keep=False), cols].head(4)
df = df.drop_duplicates().reset_index(drop=True)
print(df.shape)
Your turn: after removing repeats, how many frauds are left, and what is the new fraud rate?
print(df["Class"].sum())
print(round(df["Class"].mean() * 100, 3))
Compare fraud with normal transactions#
Problem: a model can only learn fraud if fraud looks different somewhere in the data.
Quick check: is fraud mostly big purchases? Guess before the next cell.
amounts = df.groupby("Class")["Amount"]
amounts.describe()[["count", "mean", "50%", "max"]].round(2)
bands = [0, 1, 10, 100, 1000, 30000]
df["band"] = pd.cut(df["Amount"], bins=bands, include_lowest=True)
rate = df.groupby("band", observed=True)["Class"].mean() * 100
rate.round(3)
df["Hour"] = (df["Time"] // 3600 % 24).astype(int)
by_hour = df.groupby("Hour")["Class"].agg(["size", "mean"])
by_hour["fraud_pct"] = by_hour.pop("mean") * 100
by_hour.nlargest(3, "fraud_pct").round(2)
import matplotlib.pyplot as plt
by_hour["fraud_pct"].plot(kind="bar", color="firebrick")
plt.title("Fraud rate by hour since the first transaction")
plt.xlabel("Hour (counted from the first transaction)")
plt.ylabel("Fraud rate (%)")
plt.show()
cols = ["V14", "V17", "V12", "V10"]
df.groupby("Class")[cols].mean().round(2)
normal = df.loc[df["Class"] == 0, "V14"]
fraud = df.loc[df["Class"] == 1, "V14"]
plt.hist(normal, bins=60, density=True, alpha=0.5, label="normal")
plt.hist(fraud, bins=60, density=True, alpha=0.5, label="fraud")
plt.title("V14 separates fraud from normal")
plt.xlabel("V14 (anonymised component)")
plt.ylabel("Density")
plt.legend()
Split honestly before modelling#
Problem: with so few frauds, a careless split could put almost none in the test set.
from sklearn.model_selection import train_test_split
X = df.drop(columns=["Class", "Hour", "band"])
y = df["Class"]
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, stratify=y, random_state=42)
print(len(X_train), y_train.sum())
print(len(X_test), y_test.sum())
Time and Amount are in seconds and euros; V1 to V28 are already centred. We scale, using training data only.
Common mistake: passing a single column (a Series) to a scaler
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train["Amount"])
scaler.fit(X_train[["Amount"]])
print(scaler.mean_.round(2), scaler.scale_.round(2))
A pipeline does this for every column, fitting the scaler inside training only.
A baseline model judged by precision and recall#
Problem: we need scores that reward catching fraud, not just being right about normal payments.
- Recall: of all real frauds, the share we caught
- Precision: of all our alerts, the share that were really fraud
- The confusion matrix counts all four outcomes
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score, recall_score
lazy = DummyClassifier(strategy="most_frequent")
lazy_pred = lazy.fit(X_train, y_train).predict(X_test)
print(round(accuracy_score(y_test, lazy_pred), 4))
print(recall_score(y_test, lazy_pred))
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LogisticRegression
lr = make_pipeline(StandardScaler(),
LogisticRegression(max_iter=1000))
lr.fit(X_train, y_train)
lr_pred = lr.predict(X_test)
print(lr_pred.sum())
from sklearn.metrics import confusion_matrix
print(confusion_matrix(y_test, lr_pred))
from sklearn.metrics import precision_score
print(round(precision_score(y_test, lr_pred), 3))
print(round(recall_score(y_test, lr_pred), 3))
print(round(accuracy_score(y_test, lr_pred), 4))
from sklearn.metrics import classification_report
print(classification_report(y_test, lr_pred, digits=3))
Common mistake: scoring probabilities with recall
lr_proba = lr.predict_proba(X_test)[:, 1]
recall_score(y_test, lr_proba)
lr_flag = (lr_proba >= 0.5).astype(int)
print(round(recall_score(y_test, lr_flag), 3))
print(round(lr_proba.max(), 3), int((lr_proba >= 0.5).sum()))
Fight the imbalance#
Problem: the model barely sees fraud, so it plays safe and misses too many.
- Class weights: make each fraud count more during training
- Undersampling: train on all frauds plus a sample of normal rows
- Either way, the test set stays untouched
lr_bal = make_pipeline(StandardScaler(), LogisticRegression(
max_iter=1000, class_weight="balanced"))
bal_pred = lr_bal.fit(X_train, y_train).predict(X_test)
print(round(recall_score(y_test, bal_pred), 3))
Quick check: recall went up. What do you expect happened to precision?
print(round(precision_score(y_test, bal_pred), 3))
print(confusion_matrix(y_test, bal_pred))
train = X_train.assign(Class=y_train)
frauds = train[train["Class"] == 1]
normals = train[train["Class"] == 0].sample(
n=len(frauds) * 20, random_state=42)
small = pd.concat([frauds, normals])
print(len(small), round(small["Class"].mean() * 100, 2))
X_small = small.drop(columns="Class")
y_small = small["Class"]
lr_under = make_pipeline(StandardScaler(),
LogisticRegression(max_iter=1000))
under_pred = lr_under.fit(X_small, y_small).predict(X_test)
print(round(precision_score(y_test, under_pred), 3))
print(round(recall_score(y_test, under_pred), 3))
from sklearn.ensemble import RandomForestClassifier
forest = RandomForestClassifier(n_estimators=100, random_state=42)
forest_pred = forest.fit(X_small, y_small).predict(X_test)
print(round(precision_score(y_test, forest_pred), 3))
print(round(recall_score(y_test, forest_pred), 3))
imp = pd.Series(forest.feature_importances_, index=X.columns)
imp.nlargest(6).sort_values().plot(kind="barh", color="teal")
plt.title("Top features for spotting fraud")
plt.xlabel("Importance")
plt.show()
Your turn: print the forest's confusion matrix. How many frauds did it miss?
print(confusion_matrix(y_test, forest_pred))
Mini-project: which model saves the most money?#
Compare all four models on the same test set, then count the money caught and the false alarms.
models = {"plain LR": lr, "weighted LR": lr_bal,
"undersampled LR": lr_under, "forest": forest}
rows = []
for name, model in models.items():
pred = model.predict(X_test)
rows.append({"model": name, "alerts": int(pred.sum()),
"precision": precision_score(y_test, pred),
"recall": recall_score(y_test, pred)})
pd.DataFrame(rows).set_index("model").round(3)
from sklearn.metrics import PrecisionRecallDisplay
ax = plt.gca()
PrecisionRecallDisplay.from_estimator(lr, X_test, y_test, ax=ax,
name="plain LR")
PrecisionRecallDisplay.from_estimator(forest, X_test, y_test,
ax=ax, name="forest")
plt.title("Precision vs recall at every cut-off")
plt.show()
fraud_amount = X_test["Amount"][y_test == 1]
caught = fraud_amount[forest_pred[y_test == 1] == 1].sum()
missed = fraud_amount[forest_pred[y_test == 1] == 0].sum()
print(round(caught, 2), round(missed, 2))
print(round(caught / fraud_amount.sum() * 100, 1))
false_alarms = int(((forest_pred == 1) & (y_test == 0)).sum())
per_10k = false_alarms / (y_test == 0).sum() * 10_000
print(false_alarms, round(per_10k, 1))
risk = forest.predict_proba(X_test)[:, 1]
top = X_test.assign(risk=risk, fraud=y_test).nlargest(5, "risk")
top[["Amount", "V14", "risk", "fraud"]].round(2)
Finding: the undersampled forest catches about 4 in 5 frauds while bothering very few genuine customers.
Recap#
- With 0.17% fraud, accuracy is useless: never-fraud scores 99.8%
- Drop duplicates and stratify the split before any model sees the data
- Precision counts good alerts; recall counts frauds caught
- Class weights and undersampling trade precision for recall
- Always test on the untouched, imbalanced test set
Challenges#
- Undersample with 5 and 50 normals per fraud. How do precision and recall move?
- Add the Hour column to the features. Does the forest improve?
- Give the forest class_weight="balanced" on the small table and compare.
What's next#
Lesson 41: Introduction to Time-Series Data, where the order of rows matters.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.



