Lesson 6 · Data Mining Full Course
Data Mining Full Course Part 6 (3.5 Hours): Time Series Forecasting & 5 Capstone Projects
All 9 lesson notebooks from Part 6 of the Data Mining Full Course in Python, in teaching order. Each lesson starts with a heading; the timestamps below jump…
- CourseData Mining Full Course
- Lesson6 of 6
- Video3 h 42 min
- FormatJupyter notebook · 345 code cells
- Data11 datasets
What you'll learn
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
- airline_passengers.csv preview2.1 KB
- daily_min_temperatures.csv preview66.3 KB
- shampoo.csv preview519 B
- stocks.csv preview148.1 KB
- telco_churn.csv preview947.7 KB
- banknote.csv preview44.0 KB
- movie_reviews.csv preview7.5 MB
- vader_lexicon.txt preview424.0 KB
- wine.csv preview13.0 KB
- creditcard.csv.gz preview26.3 MB
- online_retail.csv.gz preview7.1 MB
📓 Full notebook
Download .ipynbData Mining Full Course, Part 6: Time Series Forecasting and 5 Capstone Projects#
All 9 lesson notebooks from Part 6 of the Data Mining Full Course in Python, in teaching order. Each lesson starts with a heading; the timestamps below jump to it in the video.
- 0:00 Lesson 41: Introduction to Time-Series Data
- 0:24:01 Lesson 42: Exponential Smoothing Techniques
- 0:48:35 Lesson 43: Forecasting with ARIMA Models
- 1:13:05 Lesson 44: Hands-on Project- Stock Price Prediction
- 1:36:49 Lesson 45: Capstone Project- Customer Churn Prediction End-to-End
- 2:00:41 Lesson 46: Capstone Project- Market Segmentation with Clustering
- 2:27:06 Lesson 47: Capstone Project- Fraud Detection in Financial Transactions
- 2:52:46 Lesson 48: Capstone Project- Sentiment Analysis using Text Mining
- 3:17:22 Lesson 49: Capstone Project- Comprehensive Data Mining Case Study
Data files: save the datasets from the lesson page in the same folder as this notebook. Files the lessons create are written to that folder too.
Data Mining 41: Introduction to Time-Series Data#
In this lesson you'll be able to:
- Turn date text into a proper time index
- Slice, resample and repair gaps in time-stamped data
- Spot trend and seasonality with charts and decomposition
- Use lags, differences and autocorrelation
- Split by time and score simple baseline forecasts Datasets: monthly airline passengers 1949-1960, daily Melbourne minimum temperatures 1981-1990
Warm-up: fraud data has a clock too#
Lesson 40 found fraud peaks at certain hours. Count all card transactions per hour.
import pandas as pd
cards = pd.read_csv("creditcard.csv.gz",
usecols=["Time", "Class"])
hour = (cards["Time"] // 3600).astype(int)
per_hour = hour.value_counts().sort_index()
print(len(per_hour), per_hour.min(), per_hour.max())
import matplotlib.pyplot as plt
per_hour.plot()
plt.title("Card transactions per hour over two days")
plt.xlabel("Hours since first transaction")
plt.ylabel("Transactions")
plt.show()
Turn date text into real dates#
Problem: CSV files store dates as plain text, so pandas can't do date maths on them yet.
raw = pd.read_csv("airline_passengers.csv")
print(raw.dtypes)
raw.head(3)
Common mistake: using date tools on a text column
raw["Month"].dt.year
raw["Month"] = pd.to_datetime(raw["Month"])
print(raw["Month"].dtype)
raw["Month"].dt.year.head(3)
Quick check: the file says 1949-01. Which exact date will pandas store for it?
air = pd.read_csv("airline_passengers.csv",
parse_dates=["Month"], index_col="Month")
air = air.asfreq("MS")
print(air.index[:2])
print(air.index.is_monotonic_increasing)
print(air.index.is_unique, len(air))
Slice and resample by time#
Problem: we often need one period, or the same data at a coarser frequency, like years instead of months.
y1960 = air.loc["1960", "Passengers"]
print(len(y1960), y1960.min(), y1960.max())
air.loc["1958-06":"1958-09"]
yearly = air["Passengers"].resample("YS").sum()
yearly.index = yearly.index.year
yearly.head()
dated = air.copy()
dated["year"] = dated.index.year
dated["month"] = dated.index.month_name().str[:3]
dated.tail(3)
quarters = air["Passengers"].resample("QS").sum()
quarters.tail(4)
Your turn: find the average monthly passengers for each year. How do 1949 and 1960 compare?
avg = air["Passengers"].resample("YS").mean()
print(round(avg.iloc[0], 1), round(avg.iloc[-1], 1))
print(round(avg.iloc[-1] / avg.iloc[0], 2))
Find and repair gaps in daily data#
Problem: real daily data has missing days, and many time-series tools expect an unbroken calendar.
temps = pd.read_csv("daily_min_temperatures.csv",
parse_dates=["Date"], index_col="Date")
print(len(temps), temps.index.min().date(), temps.index.max().date())
days = pd.date_range(temps.index.min(), temps.index.max())
print(len(days), len(days) - len(temps))
temps = temps.asfreq("D")
temps[temps["Temp"].isna()]
Quick check: fill a missing temperature with 0, or with something based on the days around it?
temps["Temp"] = temps["Temp"].interpolate(method="time")
temps.loc["1984-12-30":"1985-01-01"]
monthly = temps["Temp"].resample("MS").mean()
monthly.plot(color="steelblue")
plt.title("Melbourne monthly mean of daily minimum temperature")
plt.xlabel("Month")
plt.ylabel("Degrees Celsius")
plt.show()
t1990 = temps.loc["1990", "Temp"]
t1990.plot(alpha=0.4, label="daily")
t1990.rolling(30).mean().plot(label="30-day mean", linewidth=3)
plt.title("Melbourne 1990: daily noise vs the seasonal path")
plt.ylabel("Degrees Celsius")
plt.legend()
Spot trend and seasonality#
Problem: a raw line mixes several patterns; we want to see each one on its own.
- Trend: the long-run direction (up, down, flat)
- Seasonality: a pattern that repeats every period, like each year
- Noise: what's left, the irregular wiggles
trend = air["Passengers"].rolling(window=12).mean()
air["Passengers"].plot(label="monthly", alpha=0.6)
trend.plot(label="12-month rolling mean", linewidth=3)
plt.title("Airline passengers: trend under the seasons")
plt.xlabel("Month")
plt.ylabel("Passengers (thousands)")
plt.legend()
Quick check: which month do you think is busiest for air travel?
profile = air.groupby(air.index.month)["Passengers"].mean()
print(profile.idxmax(), round(profile.max(), 1))
print(profile.idxmin(), round(profile.min(), 1))
swing = air["Passengers"].groupby(air.index.year).agg(["min", "max"])
swing["range"] = swing["max"] - swing["min"]
swing.iloc[[0, -1]]
import seaborn as sns
grid = dated.pivot_table(index="month", columns="year",
values="Passengers", sort=False)
sns.heatmap(grid, cmap="YlGnBu")
plt.title("Passengers by month and year")
plt.xlabel("Year")
plt.ylabel("Month")
from statsmodels.tsa.seasonal import seasonal_decompose
parts = seasonal_decompose(air["Passengers"],
model="multiplicative", period=12)
parts.plot()
plt.show()
factors = parts.seasonal.iloc[:12]
factors.index = factors.index.month
factors.round(3)
Look back in time: lags, differences, autocorrelation#
Problem: forecasting uses the past, so we need columns that hold earlier values.
look = air[["Passengers"]].copy()
look["last_month"] = look["Passengers"].shift(1)
look["change"] = look["Passengers"].diff()
look["pct"] = look["Passengers"].pct_change().round(3)
look.head(3)
p = air["Passengers"]
print(round(p.autocorr(lag=1), 3))
print(round(p.autocorr(lag=6), 3))
print(round(p.autocorr(lag=12), 3))
plt.scatter(p.shift(12), p, s=12)
plt.title("This month vs the same month last year")
plt.xlabel("Passengers 12 months earlier")
plt.ylabel("Passengers this month")
daily = temps["Temp"]
print(round(daily.autocorr(lag=1), 3))
print(round(daily.autocorr(lag=365), 3))
from statsmodels.graphics.tsaplots import plot_acf
fig = plot_acf(air["Passengers"].diff().dropna(), lags=36,
title="Autocorrelation of monthly changes")
plt.xlabel("Lag (months)")
plt.ylabel("Correlation")
plt.show()
Your turn: use pct_change(12) to find the average year-on-year growth during 1960.
yoy = air["Passengers"].pct_change(12)
print(round(yoy.loc["1960"].mean() * 100, 1))
Split by time and score baseline forecasts#
Problem: a random split lets the model peek at the future. Train on the past, test on the future.
train = air.loc[:"1958", "Passengers"]
test = air.loc["1959":, "Passengers"]
print(len(train), len(test), train.iloc[-1])
- Naive: repeat the last value
- Average: repeat the mean of the last 12 months
- Seasonal naive: repeat last year's months
import numpy as np
from sklearn.metrics import mean_absolute_error
naive = np.repeat(train.iloc[-1], len(test))
mean12 = np.repeat(train.iloc[-12:].mean(), len(test))
print(round(mean_absolute_error(test, naive), 1))
print(round(mean_absolute_error(test, mean12), 1))
Quick check: naive scored worst. Will copying last year's months do better than the average?
snaive = np.tile(train.iloc[-12:].to_numpy(), 2)
print(round(mean_absolute_error(test, snaive), 1))
plt.plot(test.index, test, color="black", label="actual")
plt.plot(test.index, naive, "--", label="naive")
plt.plot(test.index, snaive, "--", label="seasonal naive")
plt.title("Baseline forecasts vs reality, 1959-1960")
plt.xlabel("Month")
plt.ylabel("Passengers (thousands)")
plt.legend()
Your turn: traffic grows each year. Scale the seasonal naive forecast by last year's growth.
growth = train.iloc[-12:].sum() / train.iloc[-24:-12].sum()
scale = np.repeat([growth, growth ** 2], 12)
grown = snaive * scale
print(round(growth, 3))
print(round(mean_absolute_error(test, grown), 1))
avg_traffic = test.mean()
print(round(avg_traffic, 1))
print(round(mean_absolute_error(test, grown) / avg_traffic * 100, 1))
Mini-project: how predictable is Melbourne's weather?#
Forecast every day of 1990 three ways, and see which simple rule misses by the least.
t = temps["Temp"]
actual = t.loc["1990"]
yesterday = t.shift(1).loc["1990"]
last_year = t.shift(365).loc["1990"]
print(len(actual), actual.isna().sum())
past = t.loc[:"1989"]
normal = past.groupby(past.index.month).mean()
month_avg = actual.index.month.map(normal)
print(normal.round(2).loc[[1, 7]])
for name, guess in [("yesterday", yesterday),
("same day last year", last_year),
("monthly average", month_avg)]:
err = mean_absolute_error(actual, guess)
print(f"{name:>20}: {err:.2f}")
jan = actual.loc["1990-01"]
jan.plot(label="actual", color="black")
yesterday.loc["1990-01"].plot(label="yesterday rule", style="--")
plt.title("Yesterday rule, January 1990")
plt.ylabel("Degrees Celsius")
plt.legend()
Finding: yesterday and the monthly average both miss by about 2 degrees; same day last year is clearly worse.
Recap#
- Parse dates and set them as the index before any time work
.locslices by date;resamplechanges the frequencyasfreqreveals missing days;interpolatefills them- Trend, seasonality and noise can be pulled apart
- Split by time and beat the naive baselines first
Challenges#
- Resample the temperatures to yearly means. Which year was warmest?
- Use a 7-day rolling mean (shifted by one day) as a fourth forecast rule.
- Decompose the monthly temperatures. Which month has the lowest seasonal value?
What's next#
Lesson 42: Exponential Smoothing Techniques, forecasts that learn trend and season.
Data Mining 42: Exponential Smoothing Techniques#
In this lesson you'll be able to:
- Explain how exponential smoothing weights the past
- Run simple exponential smoothing by hand and with statsmodels
- Add trend (Holt) and seasonality (Holt-Winters)
- Compare forecasts on a held-out period and forecast ahead Datasets: monthly airline passengers 1949-1960, monthly shampoo sales over 3 years
Warm-up: the score to beat#
Lesson 41 split by time and scored a seasonal naive forecast. Same split, same score.
import pandas as pd
import numpy as np
air = pd.read_csv("airline_passengers.csv",
parse_dates=["Month"], index_col="Month")
y = air["Passengers"].asfreq("MS")
train, test = y.loc[:"1958"], y.loc["1959":]
print(len(train), len(test))
from sklearn.metrics import mean_absolute_error
snaive = np.tile(train.iloc[-12:].to_numpy(), 2)
print(round(mean_absolute_error(test, snaive), 1))
Weigh recent months more#
Problem: a moving average treats a month from last year the same as last month.
- New level = alpha x this month + (1 - alpha) x old level
- alpha near 1: react fast; alpha near 0: stay calm and smooth
- Older months fade away by a factor of (1 - alpha) each step
for alpha in (0.2, 0.5, 0.8):
weights = [alpha * (1 - alpha) ** k for k in range(6)]
print(alpha, np.round(weights, 3))
import matplotlib.pyplot as plt
steps = np.arange(12)
plt.bar(steps - 0.2, 0.2 * 0.8 ** steps, width=0.4, label="alpha 0.2")
plt.bar(steps + 0.2, 0.8 * 0.2 ** steps, width=0.4, label="alpha 0.8")
plt.title("How much each past month counts")
plt.xlabel("Months back")
plt.ylabel("Weight")
plt.legend()
Quick check: with alpha = 0.8, how much weight is left for everything older than 2 months?
alpha = 0.8
recent = alpha + alpha * (1 - alpha)
print(round(recent, 2), round(1 - recent, 2))
Simple exponential smoothing, by hand and by library#
Problem: we want a forecast that follows the level of a series without chasing every wiggle.
alpha = 0.5
level = train.iloc[0]
for month, value in train.iloc[:6].items():
level = alpha * value + (1 - alpha) * level
print(month.strftime("%Y-%m"), value, round(level, 2))
from statsmodels.tsa.holtwinters import SimpleExpSmoothing
ses = SimpleExpSmoothing(train, initialization_method="known",
initial_level=train.iloc[0])
ses_fit = ses.fit(smoothing_level=0.5, optimized=False)
ses_fit.fittedvalues.iloc[1:7].round(2)
print(ses_fit.forecast(3).round(2))
recent = train.loc["1956":]
plt.plot(recent.index, recent, color="black", label="actual")
for a in (0.2, 0.8):
fit_a = ses.fit(smoothing_level=a, optimized=False)
plt.plot(recent.index, fit_a.fittedvalues.loc["1956":],
"--", label=f"alpha {a}")
plt.title("Low alpha smooths, high alpha chases")
plt.legend()
Common mistake: forecasting before fitting
model = SimpleExpSmoothing(train, initialization_method="estimated")
model.forecast(12)
fitted = model.fit()
print(round(fitted.params["smoothing_level"], 2))
ses_fc = fitted.forecast(len(test))
print(round(mean_absolute_error(test, ses_fc), 2))
for a in (0.2, 0.5, 0.8, 1.0):
sse = model.fit(smoothing_level=a, optimized=False).sse
print(a, round(sse))
plt.plot(test.index, test, color="black", label="actual")
for a in (0.2, 0.8):
f = model.fit(smoothing_level=a, optimized=False).forecast(24)
plt.plot(test.index, f, "--", label=f"alpha {a}")
plt.title("Simple smoothing gives flat forecasts")
plt.ylabel("Passengers (thousands)")
plt.legend()
Add a trend: Holt's method#
Problem: simple smoothing forecasts a flat line, so it falls behind any growing series.
shampoo = pd.read_csv("shampoo.csv")
print(shampoo.dtypes)
shampoo.head(3)
The Month text 1-01 means year 1, month 01. No calendar year is given.
parts = shampoo["Month"].str.split("-", expand=True).astype(int)
shampoo["month_no"] = (parts[0] - 1) * 12 + parts[1]
sales = shampoo["Sales"]
shampoo[["Month", "month_no", "Sales"]].tail(3)
plt.plot(shampoo["month_no"], sales, marker="o")
plt.title("Shampoo sales: a strong upward trend")
plt.xlabel("Month number")
plt.ylabel("Sales")
Quick check: will a trend help on shampoo? Guess how much the error drops.
from statsmodels.tsa.holtwinters import ExponentialSmoothing
s_train, s_test = sales.iloc[:30], sales.iloc[30:]
flat = SimpleExpSmoothing(s_train, initialization_method="estimated")
flat_fc = flat.fit().forecast(6)
holt = ExponentialSmoothing(s_train, trend="add",
initialization_method="estimated").fit()
holt_fc = holt.forecast(6)
print(round(mean_absolute_error(s_test, flat_fc), 1))
print(round(mean_absolute_error(s_test, holt_fc), 1))
print(holt.params["smoothing_level"].round(2))
print(holt.params["smoothing_trend"].round(2))
print(holt_fc.round(1).to_numpy())
print(holt.level.iloc[-1].round(1))
print(holt.trend.iloc[-1].round(1))
print((holt.level.iloc[-1] + holt.trend.iloc[-1]).round(1))
plt.plot(shampoo["month_no"], sales, color="black", label="actual")
test_no = shampoo["month_no"].iloc[30:]
plt.plot(test_no, flat_fc, "--", label="simple (flat)")
plt.plot(test_no, holt_fc, "--", label="Holt (trend)")
plt.title("Shampoo: Holt follows the trend")
plt.xlabel("Month number")
plt.legend()
Your turn: a straight-line trend can overshoot. Fit a damped trend (damped_trend=True) and score it.
damped = ExponentialSmoothing(s_train, trend="add", damped_trend=True,
initialization_method="estimated")
damped_fc = damped.fit().forecast(6)
print(round(mean_absolute_error(s_test, damped_fc), 1))
far_holt = holt.forecast(24)
far_damped = damped.fit().forecast(24)
print(far_holt.iloc[-1].round(1), far_damped.iloc[-1].round(1))
Add the seasons: Holt-Winters#
Problem: airline traffic has a trend and a yearly season, so we need a third smoothed part.
- Additive season: July is always about N passengers above trend
- Multiplicative season: July is always about N% above trend
hw_add = ExponentialSmoothing(train, trend="add", seasonal="add",
seasonal_periods=12,
initialization_method="estimated").fit()
add_fc = hw_add.forecast(len(test))
print(round(mean_absolute_error(test, add_fc), 1))
Quick check: lesson 41 showed the swings grow every year. Which season type should win?
hw_mul = ExponentialSmoothing(train, trend="add", seasonal="mul",
seasonal_periods=12,
initialization_method="estimated").fit()
mul_fc = hw_mul.forecast(len(test))
print(round(mean_absolute_error(test, mul_fc), 1))
names = ["smoothing_level", "smoothing_trend", "smoothing_seasonal"]
for name in names:
print(name, round(hw_mul.params[name], 2))
print(round(hw_add.aic), round(hw_mul.aic))
plt.plot(test.index, test, color="black", label="actual")
plt.plot(test.index, add_fc, "--", label="additive season")
plt.plot(test.index, mul_fc, "--", label="multiplicative season")
plt.title("Additive vs multiplicative seasons, 1959-1960")
plt.ylabel("Passengers (thousands)")
plt.legend()
factors = pd.Series(hw_mul.season[-12:].to_numpy(),
index=train.index[-12:].month_name().str[:3])
factors.plot(kind="bar", color="darkorange")
plt.axhline(1, color="grey")
plt.title("Holt-Winters seasonal factors")
plt.ylabel("Multiple of trend")
plt.show()
print(factors.idxmax(), round(factors.max(), 2))
Common mistake: passing the whole table, date text and all
raw = pd.read_csv("airline_passengers.csv")
ExponentialSmoothing(raw, trend="add")
fixed = raw.set_index(pd.to_datetime(raw["Month"]))["Passengers"]
fixed = fixed.asfreq("MS")
print(fixed.dtype, fixed.index.freqstr, len(fixed))
Compare every forecast on the same months#
Problem: we need one table that ranks all methods on the same held-out months.
holt_air = ExponentialSmoothing(train, trend="add",
initialization_method="estimated")
holt_air_fc = holt_air.fit().forecast(len(test))
forecasts = {"seasonal naive": snaive, "simple": ses_fc,
"Holt": holt_air_fc, "HW additive": add_fc,
"HW multiplicative": mul_fc}
scores = pd.Series({name: mean_absolute_error(test, f)
for name, f in forecasts.items()})
scores.sort_values().round(2)
plt.plot(test.index, test, color="black", linewidth=2, label="actual")
plt.plot(test.index, snaive, "--", label="seasonal naive")
plt.plot(test.index, mul_fc, "--", label="HW multiplicative")
plt.title("Holt-Winters vs seasonal naive, 1959-1960")
plt.ylabel("Passengers (thousands)")
plt.legend()
Your turn: in which test month did Holt-Winters miss by the most, and by how much?
miss = (test - mul_fc).abs()
print(miss.idxmax().strftime("%Y-%m"), round(miss.max(), 1))
print(round((test - mul_fc).mean(), 1))
from sklearn.metrics import root_mean_squared_error
rmse = {name: root_mean_squared_error(test, f)
for name, f in forecasts.items()}
pd.Series(rmse).sort_values().round(2)
best = scores.min()
gain = 1 - best / scores["seasonal naive"]
print(round(best / test.mean() * 100, 1), round(gain * 100))
Mini-project: forecast 1961#
Refit the winner on all 12 years, forecast 1961, and report the busiest month and the growth.
final = ExponentialSmoothing(y, trend="add", seasonal="mul",
seasonal_periods=12,
initialization_method="estimated").fit()
f1961 = final.forecast(12)
f1961.index = f1961.index.month_name().str[:3]
f1961.round(0)
print(f1961.idxmax(), round(f1961.max()))
total_1960 = y.loc["1960"].sum()
print(total_1960, round(f1961.sum()))
print(round((f1961.sum() / total_1960 - 1) * 100, 1))
last = y.loc["1960"].to_numpy()
growth = (f1961.to_numpy() / last - 1) * 100
pd.Series(growth, index=f1961.index).round(1).head(6)
summer = ["Jun", "Jul", "Aug"]
extra = f1961[summer].sum() - y.loc["1960-06":"1960-08"].sum()
print(round(extra))
y.loc["1958":].plot(label="actual", color="black")
final.forecast(12).plot(label="1961 forecast", style="--")
plt.title("Airline passengers: Holt-Winters forecast for 1961")
plt.xlabel("Month")
plt.ylabel("Passengers (thousands)")
plt.legend()
Finding: the model expects another record year, peaking in summer. Forecasts are guides, not promises.
Recap#
- Exponential smoothing weights recent values most; alpha sets how fast
- Simple smoothing gives a flat forecast: fine for level-only series
- Holt adds a trend; damping stops it overshooting
- Holt-Winters adds seasons: multiplicative when swings grow
- Fit first, score on held-out months, then refit on all data
Challenges#
- Hold out only 1960 (12 months). Does Holt-Winters still beat seasonal naive?
- Try
damped_trend=Truein the multiplicative Holt-Winters model. - Apply Holt-Winters to monthly Melbourne temperatures from lesson 41.
What's next#
Lesson 43: Forecasting with ARIMA Models.
Data Mining 43: Forecasting with ARIMA Models#
In this lesson you'll be able to:
- Test whether a series is stationary, and fix it with logs and differencing
- Read ACF and PACF plots to choose the p, d and q orders
- Fit ARIMA and seasonal ARIMA with statsmodels and forecast ahead
- Judge a model on a held-out test period and check its residuals
- Produce a forecast with a confidence band a planner can use Datasets: airline passengers 1949-1960 (data/mining/airline_passengers.csv), plus shampoo sales for practice
Warm-up: the benchmark from lesson 42#
Lesson 42 forecast with exponential smoothing. Its score is the bar ARIMA must clear.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
path = "airline_passengers.csv"
df = pd.read_csv(path, parse_dates=["Month"], index_col="Month")
y = df["Passengers"].asfreq("MS")
print(y.shape)
y.tail(3)
from statsmodels.tsa.holtwinters import ExponentialSmoothing
train, test = y[:-24], y[-24:]
hw = ExponentialSmoothing(train, trend="add", seasonal="mul",
seasonal_periods=12).fit()
hw_pred = hw.forecast(24)
hw_mape = ((test - hw_pred).abs() / test).mean() * 100
print("Holt-Winters MAPE:", round(hw_mape, 1), "%")
What ARIMA does, and why it needs a steady series#
Problem: ARIMA learns patterns that repeat, so the series must not drift.
- AR, autoregressive: predict from the last few values
- I, integrated: model the changes, not the levels
- MA, moving average: correct using the last few errors The orders p, d and q say how much of each to use.
roll = y.rolling(12)
ax = y.plot(figsize=(9, 4), label="Passengers")
roll.mean().plot(ax=ax, label="12-month mean")
roll.std().plot(ax=ax, label="12-month std")
ax.set_title("Airline passengers: level and spread both grow")
ax.set_xlabel("Month")
ax.set_ylabel("Passengers (thousands)")
ax.legend()
from statsmodels.tsa.stattools import adfuller
adf = adfuller(y, result_object=True)
print("ADF statistic:", round(adf.statistic, 2))
print("p-value:", round(adf.pvalue, 3))
Quick check#
With that p-value, can we call the raw series stationary?
is_steady = adf.pvalue < 0.05
print("Stationary?", is_steady)
Make the series steady with logs and differencing#
Problem: remove the growing swings, the trend and the seasons.
log_y = np.log(y)
fig, axes = plt.subplots(1, 2, figsize=(10, 3.5))
y.plot(ax=axes[0], title="Raw passengers")
log_y.plot(ax=axes[1], title="Log of passengers")
axes[0].set_ylabel("Passengers (thousands)")
axes[1].set_ylabel("log(passengers)")
fig.tight_layout()
diff1 = log_y.diff()
check = log_y - log_y.shift(1)
print("Same as diff?", diff1.equals(check))
diff1.head(3).round(3)
Common mistake: testing a differenced series that still holds its missing first value
adfuller(diff1, result_object=True)
adf1 = adfuller(diff1.dropna(), result_object=True)
print("p-value after one difference:", round(adf1.pvalue, 3))
diff12 = diff1.diff(12).dropna()
adf12 = adfuller(diff12, result_object=True)
print("p-value after seasonal difference:", round(adf12.pvalue, 4))
print(len(diff12), "months left")
ax = diff12.plot(figsize=(9, 3.5))
ax.axhline(0, color="grey", linewidth=1)
ax.set_title("Log passengers after both differences")
ax.set_xlabel("Month")
ax.set_ylabel("Change in log(passengers)")
Your turn#
Load data/mining/shampoo.csv. Test the Sales column, then its first difference. Is one difference enough?
shampoo = pd.read_csv("shampoo.csv")
sales = shampoo["Sales"]
p_raw = adfuller(sales, result_object=True).pvalue
p_d1 = adfuller(sales.diff().dropna(), result_object=True).pvalue
print("Raw sales p-value:", round(p_raw, 3))
print("Steady after one difference?", p_d1 < 0.05)
Read p and q from the ACF and PACF plots#
Problem: which past months and past errors actually matter?
- ACF: how strongly each month echoes the month k steps back
- PACF: the same echo with the in-between months removed A sharp ACF cut-off hints at q; a sharp PACF cut-off hints at p.
from statsmodels.graphics.tsaplots import plot_acf, plot_pacf
fig, axes = plt.subplots(1, 2, figsize=(11, 3.5))
plot_acf(diff12, lags=24, ax=axes[0])
plot_pacf(diff12, lags=24, ax=axes[1], method="ywm")
for ax in axes:
ax.set_xlabel("Lag (months)")
fig.tight_layout()
from statsmodels.tsa.stattools import acf
values = pd.Series(acf(diff12, nlags=24))
band = 1.96 / np.sqrt(len(diff12))
strong = values[values.abs() > band].drop(0)
strong.round(3)
Quick check#
The two biggest spikes are negative, at lag 1 and lag 12. Which orders do they suggest?
- Big lag 1 spike: q = 1, one past error
- Lag 12 spike: a seasonal q of 1, last year's error This is the famous airline model: (0, 1, 1) x (0, 1, 1, 12).
Fit a first ARIMA and forecast#
Problem: turn chosen orders into a fitted model and real forecasts.
from statsmodels.tsa.arima.model import ARIMA
log_train = np.log(train)
plain = ARIMA(log_train, order=(1, 1, 1))
plain_fit = plain.fit()
plain_fit.params.round(3)
Common mistake: asking the model, rather than the fitted result, for a forecast
plain.forecast(24)
plain_pred = np.exp(plain_fit.forecast(24))
plain_pred.head(3).round(1)
def mape(actual, pred):
return ((actual - pred).abs() / actual).mean() * 100
snaive = y[-36:-24].to_list() * 2
snaive = pd.Series(snaive, index=test.index)
print("Seasonal naive:", round(mape(test, snaive), 1))
print("ARIMA(1,1,1): ", round(mape(test, plain_pred), 1))
ax = train[-48:].plot(figsize=(9, 4), label="Train")
test.plot(ax=ax, label="Actual")
plain_pred.plot(ax=ax, label="ARIMA(1,1,1)", style="--")
ax.set_title("Plain ARIMA misses the summer peaks")
ax.set_xlabel("Month")
ax.set_ylabel("Passengers (thousands)")
ax.legend()
Add the seasons with seasonal ARIMA#
Problem: teach ARIMA that July looks like last July.
sarima = ARIMA(log_train, order=(0, 1, 1),
seasonal_order=(0, 1, 1, 12))
sar_fit = sarima.fit()
print(sar_fit.params.round(3))
print("AIC:", round(sar_fit.aic, 1))
sar_pred = np.exp(sar_fit.forecast(24))
print("Seasonal ARIMA:", round(mape(test, sar_pred), 1))
print("Holt-Winters: ", round(hw_mape, 1))
ax = test.plot(figsize=(9, 4), label="Actual", linewidth=2)
sar_pred.plot(ax=ax, label="Seasonal ARIMA", style="--")
hw_pred.plot(ax=ax, label="Holt-Winters", style=":")
ax.set_title("Both seasonal models track the peaks, 1959-1960")
ax.set_xlabel("Month")
ax.set_ylabel("Passengers (thousands)")
ax.legend()
rows = []
for p, q in [(0, 1), (1, 0), (1, 1)]:
fit = ARIMA(log_train, order=(p, 1, q),
seasonal_order=(0, 1, 1, 12)).fit()
err = mape(test, np.exp(fit.forecast(24)))
rows.append({"order": (p, 1, q), "aic": fit.aic, "mape": err})
pd.DataFrame(rows).round(1)
Choose with AIC (training data only). Test MAPE is the final check, not a dial to tune.
Your turn#
Fit a seasonal ARIMA with a seasonal p of 1: seasonal_order=(1, 1, 1, 12). Does the AIC improve?
alt = ARIMA(log_train, order=(0, 1, 1),
seasonal_order=(1, 1, 1, 12)).fit()
print("AIC with seasonal p=1:", round(alt.aic, 1))
print("Airline model AIC: ", round(sar_fit.aic, 1))
Check the leftovers with residual diagnostics#
Problem: a good model leaves only random noise behind.
resid = sar_fit.resid[13:]
print("Mean:", round(resid.mean(), 4))
print("Std: ", round(resid.std(), 4))
fig, axes = plt.subplots(1, 2, figsize=(10, 3.5))
resid.plot(ax=axes[0], title="Residuals over time")
resid.plot.hist(ax=axes[1], bins=20, title="Residual spread")
axes[0].set_ylabel("Error (log units)")
axes[1].set_xlabel("Error (log units)")
fig.tight_layout()
from statsmodels.stats.diagnostic import acorr_ljungbox
acorr_ljungbox(resid, lags=[6, 12, 24]).round(3)
Quick check#
All p-values are above 0.05. Is there pattern left for the model to learn?
lb = acorr_ljungbox(resid, lags=[6, 12, 24])
print("Leftover pattern?", (lb["lb_pvalue"] < 0.05).any())
Mini-project: plan capacity for 1961#
Question: how many passengers should the airline plan for, and how sure can it be?
scores = {
"Seasonal naive": mape(test, snaive),
"ARIMA(1,1,1)": mape(test, plain_pred),
"Seasonal ARIMA": mape(test, sar_pred),
"Holt-Winters": hw_mape,
}
pd.Series(scores, name="MAPE %").sort_values().round(1)
final = ARIMA(np.log(y), order=(0, 1, 1),
seasonal_order=(0, 1, 1, 12)).fit()
fc = final.get_forecast(12)
plan = np.exp(fc.conf_int(alpha=0.05))
plan.columns = ["low", "high"]
plan["forecast"] = np.exp(fc.predicted_mean)
plan.round(0).head(6)
ax = y[-36:].plot(figsize=(9, 4), label="Actual")
plan["forecast"].plot(ax=ax, label="1961 forecast", style="--")
ax.fill_between(plan.index, plan["low"], plan["high"], alpha=0.25,
label="95% interval")
ax.set_title("Airline passengers: forecast for 1961")
ax.set_xlabel("Month")
ax.set_ylabel("Passengers (thousands)")
ax.legend()
total_1960 = y["1960"].sum()
total_1961 = plan["forecast"].sum()
growth = (total_1961 / total_1960 - 1) * 100
peak = plan["forecast"].idxmax().strftime("%B")
print(round(total_1960), round(total_1961), round(growth, 1), peak)
Finding: plan for about 9% more passengers in 1961, peaking in July, with the band widening each month.
Recap#
- ARIMA = past values (p) + differencing (d) + past errors (q)
- Test stationarity with adfuller; fix it with logs, diff and seasonal diff
- ACF and PACF spikes suggest q and p; AIC compares candidates
- Seasonal data needs seasonal_order; always beat a naive baseline
- Check residuals with Ljung-Box, then forecast with get_forecast intervals
Challenges#
- Fit ARIMA to the shampoo sales (no seasons) and score the last 6 months.
- Refit the airline model on the raw series, without logs. How does MAPE change?
- Hold back 12 months instead of 24. Do the model rankings stay the same?
What's next#
Lesson 44: Hands-on Project- Stock Price Prediction Real share prices, a naive benchmark, and an honest look at what can be predicted.
Data Mining 44: Hands-on Project- Stock Price Prediction#
In this lesson you'll be able to:
- Turn prices into daily returns and test them for patterns
- Build the naive forecast that every price model must beat
- Engineer past-only features without leaking the future
- Compare regression, random forest and ARIMA with walk-forward tests
- Judge a trading signal honestly against buy-and-hold Dataset: daily adjusted closing prices, 2007-2016, for MSFT, IBM, SBUX, AAPL and the S&P 500 (data/pandas/stocks.csv)
Warm-up: stationarity from lesson 43#
Lesson 43 tested series with adfuller before modelling. Try it on Apple.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
path = "stocks.csv"
s = pd.read_csv(path, parse_dates=["Date"], index_col="Date")
print(s.shape)
s.head(3).round(2)
from statsmodels.tsa.stattools import adfuller
aapl = s["AAPL"]
p_price = adfuller(aapl, result_object=True).pvalue
p_ret = adfuller(aapl.pct_change().dropna(),
result_object=True).pvalue
print("Price p-value: ", round(p_price, 3))
print("Return p-value below 0.001?", p_ret < 0.001)
Look before you predict: nine years of prices#
Problem: compare five series on very different price scales.
growth = s / s.iloc[0] * 100
ax = growth.plot(figsize=(9, 4))
ax.set_title("Growth of 100 invested on 3 January 2007")
ax.set_xlabel("Date")
ax.set_ylabel("Value (start = 100)")
growth.iloc[-1].sort_values(ascending=False).round(0)
Common mistake: computing returns on a table that still holds the Date text column
raw = pd.read_csv(path)
raw.pct_change()
rets = s.pct_change().dropna()
rets.head(3).round(4)
Daily returns: what actually changes#
Problem: find out whether yesterday's move tells us anything about today's.
rets.describe().loc[["mean", "std", "min", "max"]].round(4)
ax = rets["AAPL"].plot.hist(bins=80, figsize=(8, 3.5))
ax.set_title("Apple daily returns, 2007-2016")
ax.set_xlabel("Daily return (0.05 = 5%)")
ax.set_ylabel("Number of days")
rets["AAPL"].nsmallest(3).round(3)
Quick check#
If Apple rose yesterday, is it more likely to rise today? Guess, then check.
r = rets["AAPL"]
print("Return vs yesterday:", round(r.autocorr(1), 3))
print("Size vs yesterday: ", round(r.abs().autocorr(1), 3))
vol = r.rolling(20).std() * 100
ax = vol.plot(figsize=(9, 3.5))
ax.set_title("Apple 20-day volatility: storms come in clusters")
ax.set_xlabel("Date")
ax.set_ylabel("Daily std (%)")
print("Calmest:", round(vol.min(), 2), " Stormiest:", round(vol.max(), 2))
ax = plt.figure(figsize=(5, 5)).gca()
ax.scatter(r.shift(1), r, s=4, alpha=0.4)
ax.set_title("Apple: today vs yesterday")
ax.set_xlabel("Yesterday's return")
ax.set_ylabel("Today's return")
rets.corr().round(2)["AAPL"]
The naive forecast: tomorrow equals today#
Problem: set the bar any model has to clear.
cut = int(len(aapl) * 0.8)
train, test = aapl[:cut], aapl[cut:]
start = test.index[0]
print(len(train), len(test), start.date())
naive = aapl.shift(1)[cut:]
naive_mae = (test - naive).abs().mean()
naive_mape = ((test - naive).abs() / test).mean() * 100
print("Naive MAE: $", round(naive_mae, 3))
print("Naive MAPE:", round(naive_mape, 2), "%")
ax = test[:60].plot(figsize=(9, 4), label="Actual")
naive[:60].plot(ax=ax, label="Naive forecast", style="--")
ax.set_title("A forecast that looks brilliant, and is just a lag")
ax.set_xlabel("Date")
ax.set_ylabel("Price ($)")
ax.legend()
from sklearn.metrics import r2_score
print("R2 of naive prices:", round(r2_score(test, naive), 3))
err = (test - naive).abs()
by_year = err.groupby(err.index.year).mean()
by_year.round(3)
Warning: a forecast that hugs the price line may be yesterday's price in disguise. Always compare with naive.
Your turn#
Compute the naive MAPE for Microsoft (MSFT) over the same test days.
msft = s["MSFT"]
m_naive = msft.shift(1)[cut:]
m_err = (msft[cut:] - m_naive).abs() / msft[cut:]
print("MSFT naive MAPE:", round(m_err.mean() * 100, 2), "%")
Build features from the past only#
Problem: describe each day using only what was known at that day's close.
feat = pd.DataFrame({"price": aapl, "ret": aapl.pct_change()})
for k in range(1, 5):
feat[f"ret_lag{k}"] = feat["ret"].shift(k)
feat["ma5_gap"] = aapl / aapl.rolling(5).mean() - 1
feat["ma20_gap"] = aapl / aapl.rolling(20).mean() - 1
feat["vol20"] = feat["ret"].rolling(20).std()
feat.isna().sum()
feat["target"] = feat["ret"].shift(-1)
feat["next_price"] = aapl.shift(-1)
cols = ["ret"] + [c for c in feat.columns if c.startswith(
("ret_lag", "ma", "vol"))]
print(cols)
feat[["ret", "target"]].tail(3).round(4)
Common mistake: fitting on rows that rolling windows and shifts left empty
from sklearn.linear_model import LinearRegression
LinearRegression().fit(feat[cols], feat["target"])
data = feat.dropna()
tr = data[data.index < start]
te = data[data.index >= start]
print(len(feat), len(data), len(tr), len(te))
links = tr[cols].corrwith(tr["target"])
links.sort_values().round(3)
Can a model beat the naive forecast?#
Problem: predict tomorrow's return, then turn it into a price.
lin = LinearRegression().fit(tr[cols], tr["target"])
lin_ret = lin.predict(te[cols])
print("R2 on test:", round(lin.score(te[cols], te["target"]), 3))
print("Mean predicted return:", round(lin_ret.mean(), 5))
def price_mae(pred_ret):
pred_price = te["price"] * (1 + pred_ret)
return (te["next_price"] - pred_price).abs().mean()
base = price_mae(0)
print("Naive: ", round(base, 4))
print("Linear: ", round(price_mae(lin_ret), 4))
from sklearn.ensemble import RandomForestRegressor
rf = RandomForestRegressor(n_estimators=200, min_samples_leaf=20,
random_state=42, n_jobs=-1)
rf.fit(tr[cols], tr["target"])
rf_ret = rf.predict(te[cols])
print("Forest: ", round(price_mae(rf_ret), 4))
imp = pd.Series(rf.feature_importances_, index=cols)
imp.sort_values(ascending=False).round(3)
board = pd.Series({
"Naive": base,
"Linear": price_mae(lin_ret),
"Random forest": price_mae(rf_ret),
"Average return": price_mae(tr["target"].mean()),
}, name="MAE $")
board.sort_values().round(4)
Quick check#
Up days are about half the test period. Is 51% direction accuracy impressive?
up = te["target"] > 0
print("Share of up days: ", round(up.mean(), 3))
print("Linear direction: ", round(((lin_ret > 0) == up).mean(), 3))
print("Forest direction: ", round(((rf_ret > 0) == up).mean(), 3))
ARIMA and walk-forward checks#
Problem: try the lesson 43 tool, and test on more than one period.
from statsmodels.tsa.arima.model import ARIMA
log_p = np.log(aapl).reset_index(drop=True)
arima = ARIMA(log_p[:cut], order=(1, 1, 1)).fit()
arima.params.round(3)
walk = arima.append(log_p[cut:])
ar_pred = np.exp(walk.fittedvalues[cut:])
ar_mae = (test.to_numpy() - ar_pred.to_numpy())
print("ARIMA MAE:", round(np.abs(ar_mae).mean(), 3))
print("Naive MAE:", round(naive_mae, 3))
- Walk-forward testing: train on the past, test on the next block, then slide forward scikit-learn's TimeSeriesSplit makes the blocks, never shuffling time.
from sklearn.model_selection import TimeSeriesSplit
rows = []
for a, b in TimeSeriesSplit(n_splits=5).split(data):
f_tr, f_te = data.iloc[a], data.iloc[b]
m = LinearRegression().fit(f_tr[cols], f_tr["target"])
pred = f_te["price"] * (1 + m.predict(f_te[cols]))
err = (f_te["next_price"] - pred).abs().mean()
nv = (f_te["next_price"] - f_te["price"]).abs().mean()
rows.append({"start": f_te.index[0].date(), "naive": nv,
"model": err})
folds = pd.DataFrame(rows).round(3)
folds
Your turn#
In how many folds does the model beat naive, and by how much at most?
gap = folds["naive"] - folds["model"]
print("Folds won:", (gap > 0).sum(), "of", len(folds))
print("Best margin: $", round(gap.max(), 3))
Mini-project: would the signal have made money?#
Question: hold Apple only on days the linear model predicts a rise. Does that beat buy-and-hold?
signal = pd.Series(lin_ret > 0, index=te.index)
strat = te["target"] * signal
hold = te["target"]
curves = pd.DataFrame({"Signal": (1 + strat).cumprod(),
"Buy and hold": (1 + hold).cumprod()})
print("Days invested:", round(signal.mean() * 100, 1), "%")
(curves.iloc[-1] - 1).mul(100).round(1)
ax = curves.mul(100).plot(figsize=(9, 4))
ax.set_title("Signal vs buy-and-hold on the test period")
ax.set_xlabel("Date")
ax.set_ylabel("Value of 100 invested")
worst = curves / curves.cummax() - 1
print("Worst drop (%):")
worst.min().mul(100).round(1)
yearly = curves.groupby(curves.index.year).last()
yearly.round(3)
Finding: no model beat the naive forecast in a useful way, and the signal earned about half of buy-and-hold.
Recap#
- Model returns, not raw prices: prices drift, returns are steadier
- The naive forecast is the bar, and it is very hard to clear
- Build features from the past only, and split strictly by time
- Check across several periods with TimeSeriesSplit
- A chart that hugs the price line can be just a one-day lag
Challenges#
- Repeat the model comparison for the S&P 500 (GSPC).
- Add yesterday's S&P 500 return as a feature. Does Apple's error improve?
- Predict weekly instead of daily returns. Is naive still as hard to beat?
What's next#
Lesson 45: Capstone Project- Customer Churn Prediction End-to-End Pipelines, cross-validation, a cost-based threshold and a saved model.
Data Mining 45: Capstone Project- Customer Churn Prediction End-to-End#
In this lesson you'll be able to:
- Bundle cleaning, encoding and a model into one scikit-learn Pipeline
- Compare and tune models with cross-validation on training data only
- Choose a decision threshold from business costs, not the default 0.5
- Save the finished model, reload it and score new customers
- Turn predictions into a costed retention plan Dataset: IBM Telco customer churn, 7,043 customers (data/mining/telco_churn.csv)
Warm-up: the baseline to beat#
Lesson 44 showed that every model must beat a naive baseline. Here is churn's.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
df = pd.read_csv("telco_churn.csv")
print(df.shape)
df["Churn"].value_counts(normalize=True).round(3)
from sklearn.dummy import DummyClassifier
y_all = (df["Churn"] == "Yes").astype(int)
lazy = DummyClassifier(strategy="most_frequent")
lazy.fit(df[["tenure"]], y_all)
print("Accuracy:", round(lazy.score(df[["tenure"]], y_all), 3))
Set the data up once, the right way#
Problem: split before learning anything, and keep some real customers back for the very end.
new_customers = df.tail(5)
known = df.iloc[:-5]
X = known.drop(columns=["customerID", "Churn"])
y = (known["Churn"] == "Yes").astype(int)
print(X.shape, y.mean().round(3))
num_cols = ["tenure", "MonthlyCharges", "TotalCharges"]
cat_cols = [c for c in X.columns if c not in num_cols]
print(len(num_cols), "numeric,", len(cat_cols), "categorical")
print(cat_cols[:4])
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, stratify=y, random_state=42)
print(len(X_train), len(X_test))
print(y_train.mean().round(3), y_test.mean().round(3))
The test set now stays locked until the final check. Every choice uses training data only.
Build one pipeline for every preparation step#
Problem: apply the same cleaning to training rows, test rows and next month's customers.
- ColumnTransformer: different recipes for different columns
- Numbers: fill gaps with the median, then scale
- Categories: one-hot encode, ignoring unseen values
- Pipeline: preparation and model, fitted together
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
numeric = Pipeline([("fill", SimpleImputer(strategy="median")),
("scale", StandardScaler())])
prep = ColumnTransformer([
("num", numeric, num_cols),
("cat", OneHotEncoder(handle_unknown="ignore"), cat_cols)])
Common mistake: fitting the pipeline while TotalCharges is still text
prep.fit(X_train)
for part in (X_train, X_test, new_customers):
part["TotalCharges"] = pd.to_numeric(
part["TotalCharges"], errors="coerce")
print(X_train["TotalCharges"].isna().sum(), "gaps in training")
prep.fit(X_train)
names = prep.get_feature_names_out()
print(len(names), "model inputs")
print(names[:3], names[-2:])
Quick check#
The imputer fills gaps with a median. Which rows was that median computed from?
fill = prep.named_transformers_["num"]["fill"]
print("Learned median:", fill.statistics_.round(2))
print("Train median: ", X_train[num_cols].median().to_numpy())
Compare models with cross-validation#
Problem: one train/test split can be lucky; we need a steadier score.
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
logit = Pipeline([("prep", prep),
("model", LogisticRegression(max_iter=1000))])
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
auc = cross_val_score(logit, X_train, y_train, cv=cv,
scoring="roc_auc")
print(auc.round(3), round(auc.mean(), 3))
The preparation is refitted inside every fold, so no fold ever learns from the rows it is scored on.
from sklearn.tree import DecisionTreeClassifier
tree = Pipeline([("prep", prep),
("model", DecisionTreeClassifier(
max_depth=5, random_state=42))])
t_auc = cross_val_score(tree, X_train, y_train, cv=cv,
scoring="roc_auc")
print(t_auc.round(3), round(t_auc.mean(), 3))
Your turn#
Cross-validate a RandomForestClassifier pipeline (100 trees, min_samples_leaf=5, random_state=42) on the same folds.
from sklearn.ensemble import RandomForestClassifier
forest = Pipeline([("prep", prep),
("model", RandomForestClassifier(
n_estimators=100, min_samples_leaf=5,
random_state=42))])
f_auc = cross_val_score(forest, X_train, y_train, cv=cv,
scoring="roc_auc")
print(f_auc.round(3), round(f_auc.mean(), 3))
board = pd.DataFrame({"mean_auc": [auc.mean(), t_auc.mean(),
f_auc.mean()],
"spread": [auc.std(), t_auc.std(),
f_auc.std()]},
index=["logistic", "tree", "forest"])
board.sort_values("mean_auc", ascending=False).round(3)
from sklearn.model_selection import cross_validate
scores = cross_validate(logit, X_train, y_train, cv=cv,
scoring=["precision", "recall", "f1"])
pd.DataFrame(scores).filter(like="test").mean().round(3)
Tune the winner with GridSearchCV#
Problem: pick settings fairly, using cross-validation on training rows only.
from sklearn.model_selection import GridSearchCV
grid = {"model__C": np.logspace(-2, 1, 4),
"model__class_weight": [None, "balanced"]}
search = GridSearchCV(logit, grid, cv=cv, scoring="roc_auc")
search.fit(X_train, y_train)
print(pd.Series(search.best_params_).to_string())
print("Best CV AUC:", round(search.best_score_, 4))
res = pd.DataFrame(search.cv_results_)
res = res[["param_model__C", "param_model__class_weight",
"mean_test_score"]]
res.sort_values("mean_test_score", ascending=False).head(4).round(4)
Quick check#
The top four settings are within 0.001 AUC. Is tuning worth much here?
scores = res["mean_test_score"]
print("Best minus worst:", round(scores.max() - scores.min(), 4))
best = search.best_estimator_
from sklearn.metrics import roc_auc_score
test_proba = best.predict_proba(X_test)[:, 1]
print("Test AUC:", round(roc_auc_score(y_test, test_proba), 3))
coefs = pd.Series(best["model"].coef_[0],
index=best["prep"].get_feature_names_out())
coefs = coefs.sort_values()
pd.concat([coefs.head(3), coefs.tail(3)]).round(2)
Choose the threshold by business cost#
Problem: 0.5 is a default, not a business decision.
Assumed costs for this lesson (swap in your own):
- Contacting a customer with an offer: $100
- Missing a customer who then leaves: $500 in lost revenue
from sklearn.model_selection import cross_val_predict
oof = cross_val_predict(best, X_train, y_train, cv=cv,
method="predict_proba")[:, 1]
def cost(proba, truth, cut, offer=100, lost=500):
called = proba >= cut
missed = (~called) & (truth == 1)
return called.sum() * offer + missed.sum() * lost
print(len(oof), round(oof.mean(), 3))
cuts = np.arange(0.05, 0.95, 0.05).round(2)
bill = pd.Series([cost(oof, y_train.to_numpy(), c) for c in cuts],
index=cuts)
best_cut = bill.idxmin()
print("Cheapest threshold:", best_cut)
bill.loc[[0.1, best_cut, 0.5, 0.8]]
ax = (bill / 1000).plot(figsize=(8, 3.5), marker="o")
ax.axvline(best_cut, color="grey", linestyle="--")
ax.set_title("Cost of each threshold, training customers")
ax.set_xlabel("Threshold (call if probability >= this)")
ax.set_ylabel("Total cost ($ thousands)")
from sklearn.metrics import precision_score, recall_score
for c in (0.5, best_cut):
flag = oof >= c
print(c, round(precision_score(y_train, flag), 3),
round(recall_score(y_train, flag), 3))
Your turn#
Calls get dearer: $200 each. Find the cheapest threshold with the cost helper.
dear = pd.Series([cost(oof, y_train.to_numpy(), c, offer=200)
for c in cuts], index=cuts)
print("Cheapest threshold at $200 a call:", dear.idxmin())
Quick check#
Why is the cheapest threshold well below 0.5?
A miss costs 5x a call, so calling pays whenever churn risk tops about 100 / 500 = 0.2.
Save the model and reload it#
Problem: hand the finished pipeline and its threshold to someone else.
import os
import joblib
model_path = "m45_churn_model.joblib"
package = {"pipeline": best, "threshold": float(best_cut)}
joblib.dump(package, model_path)
print("Saved:", os.path.exists(model_path))
loaded = joblib.load(model_path)
model, cut = loaded["pipeline"], loaded["threshold"]
same = (model.predict_proba(X_test)[:, 1] == test_proba).all()
print("Threshold:", cut, " Identical:", same)
Common mistake: sending new rows that are missing a column the pipeline expects
model.predict_proba(new_customers.drop(columns=["tenure"]))
fresh = new_customers.drop(columns=["customerID", "Churn"])
risk = model.predict_proba(fresh)[:, 1]
result = new_customers[["customerID", "Contract", "tenure"]].copy()
result["risk"] = risk.round(3)
result["call"] = risk >= cut
result["actual"] = new_customers["Churn"]
result
Mini-project: price the retention plan#
Question: on the locked test customers, what does our plan cost compared with the alternatives?
truth = y_test.to_numpy()
plans = pd.Series({
"Call nobody": cost(test_proba, truth, 1.01),
"Call everyone": cost(test_proba, truth, 0.0),
"Default 0.5": cost(test_proba, truth, 0.5),
"Cost-based": cost(test_proba, truth, cut),
}, name="cost $")
plans.sort_values()
called = test_proba >= cut
caught = (called & (truth == 1)).sum()
print("Customers called:", called.sum(), "of", len(truth))
print("Churners reached:", caught, "of", truth.sum())
saving = plans["Call nobody"] - plans["Cost-based"]
print("Saving vs doing nothing: $", saving)
ax = (plans / 1000).sort_values().plot.barh(figsize=(7, 3))
ax.set_title("Cost of each retention plan, test customers")
ax.set_xlabel("Total cost ($ thousands)")
ax.set_ylabel("Plan")
Finding: the cost-based threshold is the cheapest plan, reaching most churners for far less than doing nothing.
Recap#
- Split first, and lock the test set until the final check
- ColumnTransformer + Pipeline = preparation that cannot leak
- Cross-validate to compare models and tune settings
- Pick the threshold from business costs, using out-of-fold predictions
- Save the pipeline and threshold together with joblib, and reload to check
Challenges#
- Change the costs to $50 per call and $800 per lost customer. Where does the threshold move?
- Add the random forest to the grid search and compare its tuned AUC.
- Add a new feature inside the pipeline: average monthly spend = TotalCharges / tenure.
What's next#
Lesson 46: Capstone Project- Market Segmentation with Clustering RFM customer segments from real online retail transactions.
Data Mining 46: Capstone Project- Market Segmentation with Clustering#
In this lesson you'll be able to:
- Clean a raw transaction log for customer analysis
- Turn transactions into RFM features: recency, frequency, monetary
- Tame skewed features and choose k with elbow and silhouette
- Profile, name and act on customer segments
- Package the segmenter in a pipeline for new customers Dataset: Online Retail, a UK online gift shop, Dec 2010 to Dec 2011 (data/pandas/online_retail.csv.gz)
Warm-up: from a churn label to no label at all#
Lesson 45 predicted churn end to end with a pipeline. Today there is no answer column.
import pandas as pd
df = pd.read_csv("online_retail.csv.gz")
print(df.shape)
df.head(3)
print(df.columns.to_list())
Cleaning the transaction log#
Problem: segments built on anonymous sales, refunds or dates stored as text are wrong.
df.isna().sum()
Quick check: what percentage of lines have no customer ID?
missing = df["CustomerID"].isna().mean() * 100
print(round(missing, 1))
cancel = df["InvoiceNo"].str.startswith("C")
print(cancel.sum(), (df["Quantity"] <= 0).sum())
df.loc[cancel, ["InvoiceNo", "Quantity", "UnitPrice"]].head(3)
print((df["UnitPrice"] <= 0).sum())
shop = df.dropna(subset=["CustomerID"])
keep = (shop["Quantity"] > 0) & (shop["UnitPrice"] > 0)
shop = shop[keep].copy()
print(len(shop), shop["CustomerID"].nunique())
Common mistake: doing date maths on dates stored as text
latest = shop["InvoiceDate"].max()
print(latest)
gap = latest - shop["InvoiceDate"]
fmt = "%m/%d/%Y %H:%M"
shop["InvoiceDate"] = pd.to_datetime(shop["InvoiceDate"], format=fmt)
print(shop["InvoiceDate"].min(), shop["InvoiceDate"].max())
shop["CustomerID"] = shop["CustomerID"].astype(int)
shop["Revenue"] = shop["Quantity"] * shop["UnitPrice"]
shop[["InvoiceNo", "CustomerID", "Revenue"]].head(3)
Turning transactions into RFM features#
Problem: clustering needs one row per customer, not one per sale line.
- Recency: days since the last purchase (lower is better)
- Frequency: number of separate orders
- Monetary: total spend in pounds
snapshot = shop["InvoiceDate"].max() + pd.Timedelta(days=1)
print(snapshot)
rfm = shop.groupby("CustomerID").agg(
last_buy=("InvoiceDate", "max"),
Frequency=("InvoiceNo", "nunique"),
Monetary=("Revenue", "sum"))
rfm["Recency"] = (snapshot - rfm["last_buy"]).dt.days
features = ["Recency", "Frequency", "Monetary"]
rfm = rfm[features]
print(rfm.shape)
rfm.head().round(1)
Quick check: a customer has 100 lines in the log. Is their frequency 100?
lines = shop.groupby("CustomerID").size()
top = lines.idxmax()
print(top, lines[top], rfm.loc[top, "Frequency"])
rfm.describe().round(1)
Your turn: find the 5 biggest spenders and their share of all revenue.
top5 = rfm["Monetary"].nlargest(5)
print(top5.round(0))
print(round(top5.sum() / rfm["Monetary"].sum() * 100, 1))
Taming skewed features before clustering#
Problem: a few huge spenders and pounds-versus-days scales would swamp the distances.
import numpy as np
import matplotlib.pyplot as plt
fig, axes = plt.subplots(1, 2, figsize=(10, 4))
rfm["Monetary"].plot.hist(bins=50, ax=axes[0])
axes[0].set_title("Monetary, raw")
np.log1p(rfm["Monetary"]).plot.hist(bins=50, ax=axes[1])
axes[1].set_title("Monetary, log1p")
axes[0].set_xlabel("Pounds")
axes[1].set_xlabel("log(1 + pounds)")
plt.tight_layout()
plt.show()
print(rfm.skew().round(2))
print(np.log1p(rfm).skew().round(2))
Common mistake: feeding k-means a table that still has a text column
country = shop.groupby("CustomerID")["Country"].first()
rfm_c = rfm.join(country)
from sklearn.cluster import KMeans
KMeans(n_clusters=4, n_init=10, random_state=42).fit(rfm_c)
from sklearn.preprocessing import StandardScaler
X_log = np.log1p(rfm_c[features])
scaler = StandardScaler()
X = scaler.fit_transform(X_log)
print(abs(X.mean(axis=0)).round(2), X.std(axis=0).round(2))
Quick check: what happens if we cluster the raw numbers instead?
raw = KMeans(n_clusters=4, n_init=10, random_state=42)
raw.fit(rfm[features])
print(pd.Series(raw.labels_).value_counts().to_list())
Choosing the number of segments#
Problem: k-means needs k up front. Elbow and silhouette advise; the business decides.
K-means in plain words:
- Drop k centres among the customers
- Give each customer to the nearest centre
- Move each centre to the middle of its customers, and repeat
from sklearn.metrics import silhouette_score
ks = range(2, 8)
inertia, silhouette = [], []
for k in ks:
km = KMeans(n_clusters=k, n_init=10, random_state=42).fit(X)
inertia.append(round(km.inertia_))
silhouette.append(round(silhouette_score(X, km.labels_), 3))
print(inertia)
print(silhouette)
fig, axes = plt.subplots(1, 2, figsize=(10, 4))
axes[0].plot(ks, inertia, marker="o")
axes[0].set_title("Elbow: inertia by k")
axes[1].plot(ks, silhouette, marker="o", color="tab:orange")
axes[1].set_title("Silhouette by k")
for ax in axes:
ax.set_xlabel("Number of segments, k")
plt.show()
k = 2 only splits good from bad customers. k = 3 and 4 tie on silhouette, so we take 4: more to act on.
km = KMeans(n_clusters=4, n_init=10, random_state=42).fit(X)
rfm["cluster"] = km.labels_
print(sorted(rfm["cluster"].value_counts().to_list()))
Profiling and naming the segments#
Problem: cluster 0, 1, 2, 3 means nothing to a marketer, and the numbers are arbitrary.
centres = scaler.inverse_transform(km.cluster_centers_)
centres = pd.DataFrame(np.expm1(centres), columns=features)
centres.sort_values("Monetary").reset_index(drop=True).round(1)
med = rfm.groupby("cluster")["Monetary"].median()
rank = {c: i for i, c in enumerate(med.sort_values().index)}
rfm["segment"] = rfm["cluster"].map(rank)
profile = rfm.groupby("segment")[features].median()
profile["customers"] = rfm["segment"].value_counts()
profile.round(1)
- 0 Lost: last bought months ago, one small order
- 1 Newcomers: recent, but only a couple of orders
- 2 Regulars: steady buyers, cooling a little
- 3 Champions: recent, frequent, high spend
names = {0: "Lost", 1: "Newcomers", 2: "Regulars", 3: "Champions"}
rfm["name"] = rfm["segment"].map(names)
rfm["name"].value_counts()
import seaborn as sns
sns.scatterplot(data=rfm, x="Recency", y="Monetary", hue="name",
s=10, alpha=0.6)
plt.yscale("log")
plt.title("Customer segments: recency vs spend")
plt.xlabel("Days since last purchase")
plt.ylabel("Total spend (pounds, log scale)")
plt.show()
Quick check: which segment brings in the most revenue, and how much?
total = rfm["Monetary"].sum()
rev = rfm.groupby("name")["Monetary"].sum() / total * 100
cust = rfm["name"].value_counts(normalize=True) * 100
share = pd.DataFrame({"customers %": cust, "revenue %": rev})
share.sort_values("revenue %").round(1)
share.sort_values("revenue %").plot.barh(figsize=(8, 4))
plt.title("Share of customers vs share of revenue")
plt.xlabel("Percent")
plt.ylabel("Segment")
plt.tight_layout()
plt.show()
Your turn: what percentage of each segment lives outside the UK?
abroad = rfm_c["Country"] != "United Kingdom"
pct_abroad = abroad.groupby(rfm["name"]).mean() * 100
print(pct_abroad.sort_values().round(1))
Packaging the segmenter for new customers#
Problem: new customers arrive every day. Log, scale and k-means must run as one unit.
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import FunctionTransformer
segmenter = make_pipeline(
FunctionTransformer(np.log1p),
StandardScaler(),
KMeans(n_clusters=4, n_init=10, random_state=42))
print(list(segmenter.named_steps))
from sklearn.model_selection import train_test_split
old_ids, new_ids = train_test_split(rfm.index, test_size=0.2,
random_state=42)
segmenter.fit(rfm.loc[old_ids, features])
new_labels = segmenter.predict(rfm.loc[new_ids, features])
print(len(old_ids), len(new_ids))
from sklearn.metrics import adjusted_rand_score
full = rfm.loc[new_ids, "segment"]
print(round(adjusted_rand_score(full, new_labels), 3))
Your turn: refit k-means on X with random_state=7. Do the segments survive?
km7 = KMeans(n_clusters=4, n_init=10, random_state=7).fit(X)
print(round(adjusted_rand_score(rfm["segment"], km7.labels_), 3))
Mini-project: who should get the win-back campaign?#
Valuable customers (Regulars or Champions) who haven't bought for 90+ days. How many are there, and how much have they spent?
valuable = rfm["name"].isin(["Regulars", "Champions"])
quiet = rfm["Recency"] > 90
at_risk = rfm[valuable & quiet]
print(len(at_risk), round(at_risk["Monetary"].sum()))
at_risk["name"].value_counts()
at_risk.join(country)["Country"].value_counts().head(3)
cols = features + ["name"]
at_risk.sort_values("Monetary", ascending=False)[cols].head()
Quick check: the top name spent 77,184 pounds in one order, then vanished. Suspicious?
first = at_risk["Monetary"].idxmax()
cols_raw = ["InvoiceNo", "Quantity", "UnitPrice"]
df.loc[df["CustomerID"] == first, cols_raw]
at_risk = at_risk.drop(index=first)
print(len(at_risk), round(at_risk["Monetary"].sum()))
import os
path = "m46_win_back.csv"
at_risk[cols].to_csv(path)
back = pd.read_csv(path, index_col="CustomerID")
print(back.shape)
pct = at_risk["Monetary"].sum() / total * 100
print(round(pct, 1), round(len(at_risk) / len(rfm) * 100, 1))
Recap#
- Clean first: drop anonymous lines, refunds and zero prices; parse dates; check odd results in the raw log
- RFM turns a transaction log into one row per customer
- Log and scale skewed features before k-means
- Elbow and silhouette advise on k; the business decides
- Rank clusters, name them, and package the steps in a pipeline
Challenges#
- Segment UK customers only. Do the same four segments appear?
- Try k = 5 and profile it. Which segment splits in two?
- Subtract each customer's refunds from Monetary. Who leaves the win-back list?
What's next#
Lesson 47: Capstone Project- Fraud Detection in Financial Transactions Cost-based thresholds, and anomaly detection vs supervised learning.
Data Mining 47: Capstone Project- Fraud Detection in Financial Transactions#
In this lesson you'll be able to:
- Put a money value on missed frauds and on false alarms
- Choose a decision threshold by cost, using a validation set
- Detect fraud without labels using Isolation Forest
- Compare anomaly detection with supervised learning honestly
- Test the idea on a second problem: forged banknotes Datasets: European card transactions, Sept 2013 (data/mining/creditcard.csv.gz) and banknotes (data/mining/banknote.csv)
Warm-up: back to the card transactions#
Lesson 46 found groups with no labels. Fraud hunting often starts the same way.
import numpy as np
import pandas as pd
cc = pd.read_csv("creditcard.csv.gz")
print(cc.shape)
print(cc["Class"].value_counts().to_list())
print(round(cc["Class"].mean() * 100, 3))
cc[["Time", "V1", "V2", "V28", "Amount", "Class"]].head(3)
Putting a price on mistakes#
Problem: precision and recall treat all frauds alike. A missed 2,000 fraud is not a missed 1.
- Missed fraud: the bank loses the transaction amount
- False alarm: an analyst checks it; we assume 10 euros each
cc.groupby("Class")["Amount"].describe().round(1)
Quick check: how much money did all the frauds move in total?
fraud_total = cc.loc[cc["Class"] == 1, "Amount"].sum()
print(round(fraud_total, 2))
Common mistake: combining two true/false columns with
and
y_true = cc["Class"]
flag = np.zeros(len(cc))
missed = (y_true == 1) and (flag == 0)
def cost(y_true, flag, amount, alarm_cost=10):
missed = (y_true == 1) & (flag == 0)
false_alarm = (y_true == 0) & (flag == 1)
total = amount[missed].sum() + alarm_cost * false_alarm.sum()
return round(float(total), 2)
amount = cc["Amount"]
print(cost(y_true, np.zeros(len(cc)), amount))
print(cost(y_true, np.ones(len(cc)), amount))
Splitting honestly and training a supervised model#
Problem: if we choose a threshold on the test set, the test score is no longer honest.
- Train 60%: the model learns
- Validation 20%: we choose the threshold
- Test 20%: we report the result, once
from sklearn.model_selection import train_test_split
X = cc.drop(columns=["Time", "Class"])
y = cc["Class"]
X_train, X_rest, y_train, y_rest = train_test_split(
X, y, test_size=0.4, stratify=y, random_state=42)
X_val, X_test, y_val, y_test = train_test_split(
X_rest, y_rest, test_size=0.5, stratify=y_rest, random_state=42)
print(len(X_train), len(X_val), len(X_test))
print(y_train.sum(), y_val.sum(), y_test.sum())
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
model = make_pipeline(StandardScaler(),
LogisticRegression(max_iter=1000))
model.fit(X_train, y_train)
p_val = model.predict_proba(X_val)[:, 1]
print((p_val >= 0.5).sum())
print(round(np.median(p_val[y_val == 1]), 3))
print(round(np.median(p_val[y_val == 0]), 5))
Common mistake: scoring new rows that have different columns
new_rows = cc.drop(columns="Class").head(3)
model.predict_proba(new_rows)
new_rows = new_rows[X_train.columns]
print(model.predict_proba(new_rows)[:, 1].round(4))
Quick check: at the default 0.5 cut-off, does the model beat flagging nothing?
amt_val = X_val["Amount"]
print(cost(y_val, np.zeros(len(y_val)), amt_val))
print(cost(y_val, (p_val >= 0.5).astype(int), amt_val))
Choosing the threshold by cost#
Problem: 0.5 assumes both mistakes cost the same. Here they clearly don't.
thresholds = np.linspace(0.01, 0.99, 99).round(2)
costs = [cost(y_val, (p_val >= t).astype(int), amt_val)
for t in thresholds]
best = thresholds[int(np.argmin(costs))]
print(best, min(costs))
import matplotlib.pyplot as plt
plt.plot(thresholds, costs)
plt.axvline(best, color="red", linestyle="--", label="cheapest")
plt.axvline(0.5, color="grey", linestyle=":", label="default 0.5")
plt.title("Validation cost by threshold")
plt.xlabel("Threshold")
plt.ylabel("Cost (euros)")
plt.legend()
plt.show()
p_test = model.predict_proba(X_test)[:, 1]
amt_test = X_test["Amount"]
flag_05 = (p_test >= 0.5).astype(int)
flag_best = (p_test >= best).astype(int)
print(cost(y_test, flag_05, amt_test))
print(cost(y_test, flag_best, amt_test))
from sklearn.metrics import confusion_matrix
print(confusion_matrix(y_test, flag_05))
print(confusion_matrix(y_test, flag_best))
lost = amt_test[(y_test == 1) & (flag_best == 0)].sum()
alarms = 10 * ((y_test == 0) & (flag_best == 1)).sum()
print(round(lost, 2), alarms)
Your turn: a false alarm now means a blocked card and a phone call, 200 euros. Find the best threshold.
costs200 = [cost(y_val, (p_val >= t).astype(int), amt_val, 200)
for t in thresholds]
print(thresholds[int(np.argmin(costs200))])
Catching fraud without labels: Isolation Forest#
Problem: fraud labels arrive weeks late, and new tricks have no labels at all.
- Build random trees that split the data at random cut points
- Odd transactions get isolated in very few cuts
- Fewer cuts means a higher anomaly score
from sklearn.ensemble import IsolationForest
iso = IsolationForest(n_estimators=200, random_state=42)
iso.fit(X_train)
print(iso.max_samples_)
Quick check: Isolation Forest's predict gives two values. Are they 0 and 1?
values, counts = np.unique(iso.predict(X_val), return_counts=True)
print(values, counts)
s_val = -iso.score_samples(X_val)
s_test = -iso.score_samples(X_test)
print(round(s_val[y_val == 1].mean(), 3))
print(round(s_val[y_val == 0].mean(), 3))
plt.hist(s_val[y_val == 0], bins=50, density=True, alpha=0.6,
label="normal")
plt.hist(s_val[y_val == 1], bins=50, density=True, alpha=0.6,
label="fraud")
plt.title("Isolation Forest scores (validation)")
plt.xlabel("Anomaly score (higher = stranger)")
plt.ylabel("Density")
plt.legend()
plt.show()
shares = [0.001, 0.002, 0.005, 0.01, 0.02]
iso_costs = []
for share in shares:
cut = np.quantile(s_val, 1 - share)
iso_flag = (s_val >= cut).astype(int)
iso_costs.append(cost(y_val, iso_flag, amt_val))
print(iso_costs)
best_share = shares[int(np.argmin(iso_costs))]
cut = np.quantile(s_val, 1 - best_share)
iso_flag = (s_test >= cut).astype(int)
print(best_share, cost(y_test, iso_flag, amt_test))
Head to head: anomaly detection vs supervised learning#
Problem: which approach should the fraud team run, and what does each one buy them?
from sklearn.metrics import average_precision_score
print(round(average_precision_score(y_test, p_test), 3))
print(round(average_precision_score(y_test, s_test), 3))
def summary(flag):
caught = int(((y_test == 1) & (flag == 1)).sum())
alarms = int(((y_test == 0) & (flag == 1)).sum())
return {"caught": caught, "false alarms": alarms,
"cost": cost(y_test, flag, amt_test)}
results = pd.DataFrame.from_dict({
"flag nothing": summary(np.zeros(len(y_test))),
"logistic 0.5": summary(flag_05),
"logistic tuned": summary(flag_best),
"isolation forest": summary(iso_flag)}, orient="index")
results
results["cost"].plot.barh(color="tab:red")
plt.title("Test-set cost by approach")
plt.xlabel("Cost (euros)")
plt.ylabel("Approach")
plt.tight_layout()
plt.show()
Quick check: does Isolation Forest catch any frauds that the tuned model misses?
only_iso = (y_test == 1) & (iso_flag == 1) & (flag_best == 0)
print(only_iso.sum(), round(amt_test[only_iso].sum(), 2))
So when is anomaly detection still worth running?
- Before labels exist, for example on a new product or market
- To spot new fraud patterns the supervised model never saw
- To decide which transactions are worth labelling first
A second case: forged banknotes#
Problem: anomaly detection assumes the odd cases are rare. What if they aren't?
bn = pd.read_csv("banknote.csv")
print(bn.shape)
print(bn["forged"].value_counts().to_list())
Xb = bn.drop(columns="forged")
yb = bn["forged"]
Xb_train, Xb_test, yb_train, yb_test = train_test_split(
Xb, yb, test_size=0.3, stratify=yb, random_state=42)
print(len(Xb_train), len(Xb_test))
import seaborn as sns
sns.scatterplot(data=bn, x="variance", y="skewness", hue="forged",
s=12, alpha=0.6)
plt.title("Banknotes: genuine (0) vs forged (1)")
plt.xlabel("Variance of wavelet image")
plt.ylabel("Skewness of wavelet image")
plt.show()
from sklearn.metrics import roc_auc_score
lr_b = make_pipeline(StandardScaler(), LogisticRegression())
lr_b.fit(Xb_train, yb_train)
p_b = lr_b.predict_proba(Xb_test)[:, 1]
print(round(roc_auc_score(yb_test, p_b), 4))
Quick check: 44% of these notes are forged. Can Isolation Forest spot them as anomalies?
iso_all = IsolationForest(n_estimators=200, random_state=42)
iso_all.fit(Xb_train)
s_all = -iso_all.score_samples(Xb_test)
print(round(roc_auc_score(yb_test, s_all), 3))
genuine = Xb_train[yb_train == 0]
iso_gen = IsolationForest(n_estimators=200, random_state=42)
iso_gen.fit(genuine)
s_gen = -iso_gen.score_samples(Xb_test)
print(round(roc_auc_score(yb_test, s_gen), 3))
Your turn: try Local Outlier Factor (lesson 38) with novelty=True, scaled, on genuine notes only.
from sklearn.neighbors import LocalOutlierFactor
lof = make_pipeline(StandardScaler(),
LocalOutlierFactor(novelty=True))
lof.fit(genuine)
s_lof = -lof.score_samples(Xb_test)
print(round(roc_auc_score(yb_test, s_lof), 3))
Mini-project: the business case for the fraud team#
Using the test set: how much does the tuned model save, how many alerts does it raise, and what slips through?
nothing = results.loc["flag nothing", "cost"]
tuned = results.loc["logistic tuned", "cost"]
print(round(nothing - tuned, 2))
print(round((nothing - tuned) / nothing * 100, 1))
alerts = flag_best.sum()
print(alerts, round(alerts / len(y_test) * 10000, 1))
print(round((nothing - tuned) / alerts, 2))
missed = (y_test == 1) & (flag_best == 0)
print(missed.sum())
amt_test[missed].sort_values(ascending=False).head(3)
import os
path = "m47_results.csv"
results.to_csv(path, index_label="method")
back = pd.read_csv(path, index_col="method")
back["cost"]
Recap#
- Price both mistakes: missed fraud loses the amount, false alarms cost reviews
- Train, validate, test: choose the threshold on validation, report on test once
- The cheapest threshold is rarely 0.5, and moves with the price of an alarm
- Isolation Forest needs no labels but trails a supervised model
- Anomaly detection works when anomalies are rare, or when you fit on normal data
Challenges#
- Add
class_weight="balanced"to the logistic model. Does the tuned test cost improve? - Add the Isolation Forest score as an extra feature for the logistic model.
- Make the false-alarm cost depend on the amount (say 1% of it) and redo the threshold search.
What's next#
Lesson 48: Capstone Project- Sentiment Analysis using Text Mining Turn real text into numbers and classify it as positive or negative.
Data Mining 48: Capstone Project- Sentiment Analysis using Text Mining#
In this lesson you'll be able to:
- Explore a collection of real text reviews
- Score sentiment with a ready-made dictionary (VADER)
- Turn text into numbers with bag of words and TF-IDF
- Train, tune and test a sentiment classifier
- Read which words drive the model, and pick the right tool Dataset: 2,000 movie reviews (data/mining/movie_reviews.csv) and the VADER lexicon
Warm-up: is this data balanced?#
Lesson 47 showed that rare classes make accuracy misleading. Check the balance first.
import pandas as pd
df = pd.read_csv("movie_reviews.csv")
print(df.shape)
df["sentiment"].value_counts()
Meet the reviews#
Problem: before mining text, learn how long and how clean it is.
first = df.loc[0, "text"]
print(len(first))
print(first[:250])
df["words"] = df["text"].str.split().str.len()
df.groupby("sentiment")["words"].describe().round(1)
Quick check: in this data, do critics write more when they like a film, or less?
import matplotlib.pyplot as plt
for label in ["neg", "pos"]:
part = df.loc[df["sentiment"] == label, "words"]
plt.hist(part, bins=40, alpha=0.5, label=label)
plt.title("Review length by sentiment")
plt.xlabel("Words in review")
plt.ylabel("Number of reviews")
plt.legend()
plt.show()
has_capitals = df["text"].str.contains("[A-Z]")
print(has_capitals.sum())
A ready-made dictionary: VADER#
Problem: score sentiment today, with no labelled data and no training.
- VADER: about 7,500 words, each rated by people from -4 to +4
- Plus rules for 'not', 'very', capitals and exclamation marks
- It adds everything up into one compound score from -1 to +1
lex = pd.read_csv("vader_lexicon.txt", sep="\t",
header=None, usecols=[0, 1],
names=["token", "score"])
print(lex.shape)
picks = ["great", "fun", "boring", "waste", "horrendous"]
lex[lex["token"].isin(picks)]
Your turn: how many lexicon words are positive, and how many are negative?
print((lex["score"] > 0).sum())
print((lex["score"] < 0).sum())
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
sia = SentimentIntensityAnalyzer()
short = df.loc[df["words"].idxmin(), "text"]
print(short)
print(sia.polarity_scores(short))
Quick check: VADER rates 'good' as positive. What happens to 'not good'?
for phrase in ["good", "not good"]:
print(phrase, sia.polarity_scores(phrase)["compound"])
Common mistake: comparing the whole score dictionary with 0
scores = sia.polarity_scores(short)
scores > 0
print(scores["compound"])
print(scores["compound"] > 0)
A fair test: hold out unseen reviews#
Problem: the dictionary and a trained model must face the same unseen reviews.
from sklearn.model_selection import train_test_split
y = (df["sentiment"] == "pos").astype(int)
X_train, X_test, y_train, y_test = train_test_split(
df["text"], y, test_size=0.2, random_state=42, stratify=y)
print(len(X_train), len(X_test), y_test.mean())
Scoring the test reviews with VADER#
Problem: does a general word list understand film critics?
def compound(text):
return sia.polarity_scores(text)["compound"]
vader = X_test.apply(compound)
vader.describe().round(3)
from sklearn.metrics import accuracy_score
vader_pred = (vader >= 0).astype(int)
print(accuracy_score(y_test, vader_pred))
pd.crosstab(y_test, vader_pred,
rownames=["actual"], colnames=["vader"])
vader.groupby(y_test).plot.hist(bins=30, alpha=0.5, legend=True)
plt.title("VADER compound score by real sentiment")
plt.xlabel("Compound score (0 = negative, 1 = positive label)")
plt.ylabel("Number of reviews")
plt.show()
Why? Long reviews retell the plot, and even a pan praises the actors or the music. Every positive word pushes the total up.
Let's test that idea: score one misjudged review sentence by sentence.
missed = X_test[(y_test == 0) & (vader_pred == 1)]
example = missed.iloc[0]
sentences = pd.Series(example.split(" . "))
sent_scores = sentences.apply(compound)
print(len(sentences), (sent_scores > 0).sum(), (sent_scores < 0).sum())
print(round(vader[missed.index[0]], 3))
Your turn: what share of the negative test reviews did VADER call positive?
print(vader_pred[y_test == 0].mean().round(3))
Turning text into numbers#
Problem: a model can't read sentences, so each review becomes a row of word counts.
- Bag of words: one column per word, counting how often it appears
- Word order is lost; stop words like 'the' are dropped
from sklearn.feature_extraction.text import CountVectorizer
counter = CountVectorizer(stop_words="english", min_df=3)
counts = counter.fit_transform(X_train)
print(counts.shape)
totals = pd.Series(counts.sum(axis=0).A1,
index=counter.get_feature_names_out())
totals.nlargest(8)
Quick check: 'not' flipped VADER's score. Is 'not' still in our vocabulary?
print("not" in counter.vocabulary_)
print(len(counter.vocabulary_))
- TF-IDF: like counts, but words found in almost every review weigh less
- Rare, specific words like 'boring' stand out more
from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer(stop_words="english", min_df=3)
tfidf.fit(X_train)
idf = pd.Series(tfidf.idf_, index=tfidf.get_feature_names_out())
print(idf[["film", "movie", "great", "boring"]].round(2))
Training and tuning a sentiment classifier#
Problem: learn from 1,600 labelled reviews which words signal praise or a pan.
- Pipeline: vectoriser + model, so each cross-validation fold builds its own vocabulary
- Baseline: Naive Bayes (lesson 16). Challenger: logistic regression (lesson 17)
from sklearn.pipeline import make_pipeline
from sklearn.naive_bayes import MultinomialNB
from sklearn.model_selection import cross_val_score
nb_model = make_pipeline(
TfidfVectorizer(stop_words="english", min_df=3), MultinomialNB())
nb_cv = cross_val_score(nb_model, X_train, y_train, cv=5)
print(nb_cv.mean().round(3))
Quick check: dropping stop words also dropped 'not'. Does keeping them help?
keep_all = make_pipeline(TfidfVectorizer(min_df=3), MultinomialNB())
keep_cv = cross_val_score(keep_all, X_train, y_train, cv=5)
print(keep_cv.mean().round(3))
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV
lr_model = make_pipeline(
TfidfVectorizer(stop_words="english", min_df=3),
LogisticRegression(max_iter=1000))
grid = GridSearchCV(lr_model, {"logisticregression__C": [1, 10, 100]},
cv=5)
grid.fit(X_train, y_train)
print(grid.best_params_, round(grid.best_score_, 3))
Only now, once, do we touch the test set.
model = grid.best_estimator_
nb_model.fit(X_train, y_train)
print("Naive Bayes:", nb_model.score(X_test, y_test))
print("Logistic:", model.score(X_test, y_test))
from sklearn.metrics import classification_report
model_pred = model.predict(X_test)
print(classification_report(y_test, model_pred,
target_names=["neg", "pos"]))
Common mistake: building a fresh vectoriser on the test reviews
lr = model[-1]
wrong = TfidfVectorizer(stop_words="english", min_df=3)
X_wrong = wrong.fit_transform(X_test)
lr.predict(X_wrong)
vocab = model[0]
X_right = vocab.transform(X_test)
print(X_right.shape)
print((lr.predict(X_right) == model_pred).all())
What did the model learn?#
Problem: a business wants to know why, not just how accurate.
weights = pd.Series(lr.coef_[0],
index=vocab.get_feature_names_out())
weights = weights.sort_values()
print(weights.head(6).round(2))
print(weights.tail(6).round(2))
top = pd.concat([weights.head(8), weights.tail(8)])
colours = ["tab:red" if w < 0 else "tab:blue" for w in top]
top.plot.barh(color=colours)
plt.title("Words that move the sentiment model")
plt.xlabel("Logistic regression weight")
plt.ylabel("Word")
plt.show()
Notice 'plot' and 'script': critics mention them mostly when complaining. And 'matrix' is a film title: the model also learns which films critics liked.
Your turn: how sure is the model about the shortest review?
print(model.predict_proba([short]).round(3))
Mini-project: dictionary or trained model?#
A studio wants to track reviews automatically. Which tool should it use, and where does each one fail?
results = pd.DataFrame({
"actual": y_test, "vader": vader_pred, "model": model_pred,
"words": X_test.str.split().str.len()})
results["vader_ok"] = results["vader"] == results["actual"]
results["model_ok"] = results["model"] == results["actual"]
results[["vader_ok", "model_ok"]].mean().round(3)
labels = ["short", "medium", "long"]
results["length"] = pd.qcut(results["words"], 3, labels=labels)
by_len = results.groupby("length", observed=True)
acc = by_len[["vader_ok", "model_ok"]].mean().round(3)
acc
acc.plot.bar(rot=0)
plt.title("Accuracy by review length")
plt.xlabel("Review length (thirds of the test set)")
plt.ylabel("Accuracy")
plt.show()
agree = results["vader"] == results["model"]
print(agree.mean().round(3))
print(results.loc[agree, "model_ok"].mean().round(3))
print(results.loc[~agree, "model_ok"].mean().round(3))
Finding: the trained model wins at every length. VADER is a quick start when no labels exist. When both tools agree, the verdict is right more often: a cheap confidence check.
Recap#
- Check text first: length, balance and how it was cleaned
- VADER scores sentiment with no training, but leans positive on film reviews
- Bag of words and TF-IDF turn each review into a row of numbers
- Fit the vectoriser on training text only; a pipeline does it for you
- Read the weights: they explain the model in plain words
Challenges#
- Add ngram_range=(1, 2) to the TF-IDF step. Does 'not good' help accuracy?
- Find VADER's best cut-off on 200 training reviews, then test it once.
- Train the same pipeline on data/mining/sms_spam.csv. Which words mark spam?
What's next#
Lesson 49: Capstone Project- Comprehensive Data Mining Case Study The whole KDD process, start to finish, on one new dataset.
Data Mining 49: Capstone Project- Comprehensive Data Mining Case Study#
In this lesson you'll be able to:
- Run every step of the KDD process on one dataset
- Check, clean and scale data before mining it
- Find natural groups with k-means and test them against the truth
- Compare classifiers fairly and test the winner once
- Turn the results into a clear, practical finding Dataset: chemical analysis of 178 Italian wines (data/mining/wine.csv)
Warm-up: numbers first#
Lesson 48 had to turn text into numbers. Here the numbers come ready-made.
import pandas as pd
df = pd.read_csv("wine.csv")
print(df.shape)
df.iloc[:3, [0, 6, 9, 12, 13]]
The KDD process: our map for today#
- Selection: pick the data and the question
- Preprocessing: check and clean
- Transformation: scale, reduce, reshape
- Data mining: clustering and classification
- Interpretation: turn patterns into a finding
Step 1: Selection: what question can this data answer?#
Problem: a lab wants to confirm a wine's grape cultivar from its chemistry alone.
df["cultivar"].value_counts()
print(df.columns[:-1].to_list())
The cultivars are coded class_0, class_1 and class_2 in the original data.
Step 2: Preprocessing: check before you trust#
Problem: hidden gaps, duplicates or odd values would spoil every later step.
Quick check: how many missing values and duplicate rows do you expect in lab data?
print(df.isna().sum().sum())
print(df.duplicated().sum())
summary = df.describe().T[["mean", "std", "min", "max"]]
summary.round(2)
picks = ["alcohol", "flavanoids", "color_intensity", "proline"]
df.groupby("cultivar")[picks].mean().round(2)
import matplotlib.pyplot as plt
df.boxplot(column="flavanoids", by="cultivar")
plt.title("Flavanoids by cultivar")
plt.suptitle("")
plt.xlabel("Cultivar")
plt.ylabel("Flavanoids")
plt.show()
Quick check: are there extreme values? Count readings more than 3 standard deviations from the mean.
chem = df.drop(columns="cultivar")
z = (chem - chem.mean()) / chem.std()
extreme = (z.abs() > 3).sum()
print(extreme[extreme > 0])
print(extreme.sum(), z.size)
A handful of extremes in real lab readings: we keep them, and scaling handles the rest.
Common mistake: scaling the whole table, label column included
from sklearn.preprocessing import StandardScaler
StandardScaler().fit_transform(df)
X = df.drop(columns="cultivar")
y = df["cultivar"]
print(X.shape, y.shape)
Step 3: Transformation: put every chemical on the same scale#
Problem: proline runs into the hundreds, hue stays near 1, so distances are lopsided.
X_scaled = StandardScaler().fit_transform(X)
scaled = pd.DataFrame(X_scaled, columns=X.columns)
scaled[["hue", "proline"]].describe().round(2)
- PCA squeezes 13 measurements into 2 new axes that keep the most spread
- A picture only: the models still use all 13 columns
from sklearn.decomposition import PCA
pca = PCA(n_components=2)
coords = pca.fit_transform(X_scaled)
print(pca.explained_variance_ratio_.round(3))
for name in sorted(y.unique()):
mask = (y == name).to_numpy()
plt.scatter(coords[mask, 0], coords[mask, 1], label=name,
alpha=0.7)
plt.title("178 wines on two PCA axes")
plt.xlabel("PC1")
plt.ylabel("PC2")
plt.legend()
plt.show()
loadings = pd.Series(pca.components_[0], index=X.columns)
loadings.sort_values().round(2).iloc[[0, 1, -2, -1]]
Step 4a: Mining without labels: do the wines group themselves?#
Problem: if nobody had labelled the wines, would the chemistry still reveal three groups?
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
for k in range(2, 7):
km = KMeans(n_clusters=k, n_init=10, random_state=42)
labels = km.fit_predict(X_scaled)
print(k, round(silhouette_score(X_scaled, labels), 3))
ks = range(2, 7)
sil = [silhouette_score(X_scaled, KMeans(n_clusters=k, n_init=10,
random_state=42).fit_predict(X_scaled)) for k in ks]
plt.plot(list(ks), sil, marker="o")
plt.title("Silhouette score by number of clusters")
plt.xlabel("Number of clusters (k)")
plt.ylabel("Silhouette score")
plt.show()
Your turn: with k = 2, where do the three cultivars end up?
km2 = KMeans(n_clusters=2, n_init=10, random_state=42)
two = km2.fit_predict(X_scaled)
pd.crosstab(two, df["cultivar"], rownames=["cluster"])
km = KMeans(n_clusters=3, n_init=10, random_state=42)
df["cluster"] = km.fit_predict(X_scaled)
order = df.groupby("cluster")["proline"].mean().rank()
df["cluster"] = df["cluster"].map(order.astype(int) - 1)
df.groupby("cluster")[picks].mean().round(2)
Now, and only now, compare the clusters with the real cultivars.
pd.crosstab(df["cluster"], df["cultivar"])
from sklearn.metrics import adjusted_rand_score
print(round(adjusted_rand_score(y, df["cluster"]), 3))
Quick check: would k-means find the cultivars as well without scaling?
raw = KMeans(n_clusters=3, n_init=10, random_state=42).fit(X)
print(round(adjusted_rand_score(y, raw.labels_), 3))
Step 4b: Mining with labels: predict the cultivar#
Problem: the lab wants a model that names the cultivar of a new wine, with an honest score.
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=42, stratify=y)
print(len(X_train), len(X_test))
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
models = {
"logistic": make_pipeline(StandardScaler(),
LogisticRegression(max_iter=1000)),
"knn": make_pipeline(StandardScaler(), KNeighborsClassifier()),
"tree": DecisionTreeClassifier(random_state=42),
"forest": RandomForestClassifier(random_state=42),
}
from sklearn.model_selection import cross_val_score
for name, model in models.items():
s = cross_val_score(model, X_train, y_train, cv=5)
print(f"{name:9} {s.mean():.3f} +/- {s.std():.3f}")
Your turn: score k-nearest neighbours again, without the scaler.
knn_raw = KNeighborsClassifier()
s = cross_val_score(knn_raw, X_train, y_train, cv=5)
print(round(s.mean(), 3))
Logistic regression wins on training data. Only now do we open the test set, once.
best = models["logistic"].fit(X_train, y_train)
print(best.score(X_test, y_test))
test_pred = best.predict(X_test)
pd.crosstab(y_test, test_pred, rownames=["actual"],
colnames=["predicted"])
A new bottle arrives: one real wine held back in the test set.
Common mistake: passing one row as a flat Series
bottle = X_test.iloc[0]
best.predict(bottle)
bottle = X_test.iloc[[0]]
print(best.predict(bottle), y_test.iloc[0])
print(best.predict_proba(bottle).round(3))
Step 5: Interpretation: which chemistry matters?#
Problem: the lab needs to know which measurements carry the signal, not just a score.
forest = models["forest"].fit(X_train, y_train)
imp = pd.Series(forest.feature_importances_, index=X.columns)
imp = imp.sort_values(ascending=False)
imp.head(6).round(3)
imp.sort_values().plot.barh()
plt.title("What tells the cultivars apart")
plt.xlabel("Random forest importance")
plt.ylabel("Measurement")
plt.show()
lr = best[-1]
coef = pd.DataFrame(lr.coef_, index=lr.classes_, columns=X.columns)
coef.loc["class_2"].sort_values().round(2).iloc[[0, 1, -2, -1]]
Your turn: save the cultivar profiles for the lab, then read them back.
import os
top4 = imp.index[:4].to_list()
profile = df.groupby("cultivar")[top4].mean().round(2)
profile.to_csv("m49_cultivar_profiles.csv")
pd.read_csv("m49_cultivar_profiles.csv")
Mini-project: which two tests are enough?#
Each lab test costs money. Can two measurements identify the cultivar almost as well as 13?
one = make_pipeline(StandardScaler(),
LogisticRegression(max_iter=1000))
for col in top4:
s = cross_val_score(one, X_train[[col]], y_train, cv=5)
print(f"{col:16} {s.mean():.3f}")
from itertools import combinations
rows = []
for pair in combinations(top4, 2):
pipe = make_pipeline(StandardScaler(),
LogisticRegression(max_iter=1000))
s = cross_val_score(pipe, X_train[list(pair)], y_train, cv=5)
rows.append({"pair": " + ".join(pair), "cv": s.mean()})
pairs = pd.DataFrame(rows).sort_values("cv", ascending=False)
pairs.round(3)
best_pair = pairs.iloc[0]["pair"].split(" + ")
pipe = make_pipeline(StandardScaler(),
LogisticRegression(max_iter=1000))
pipe.fit(X_train[best_pair], y_train)
print(best_pair, round(pipe.score(X_test[best_pair], y_test), 3))
for name in sorted(y.unique()):
part = df[df["cultivar"] == name]
plt.scatter(part[best_pair[0]], part[best_pair[1]],
label=name, alpha=0.7)
plt.title("Two lab tests, three cultivars")
plt.xlabel(best_pair[0])
plt.ylabel(best_pair[1])
plt.legend()
plt.show()
Finding: colour intensity + flavanoids name about 87% of new wines; all 13 tests named all 45. Two tests make a cheap first screen; the full panel is worth it when a wrong label is costly.
Recap#
- Start with the question, then check the data before trusting it
- Scale features before distance-based methods like k-means and kNN
- Clustering finds groups without labels; check them when labels exist
- Compare models with cross-validation; open the test set once
- Finish with a finding someone can act on
Challenges#
- Run the same five KDD steps on data/mining/breast_cancer.csv.
- Tune C for the logistic regression with GridSearchCV. Does cross-validation improve?
- Try three tests instead of two. How close do you get to all 13?
What's next#
This is the end of the Data Mining series. Congratulations! Keep going with the Pandas series to sharpen your data wrangling. Or explore the channel's other series for new projects.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.



