Mathew K Analytics

Lesson 6 · Data Mining Full Course

Data Mining Full Course Part 6 (3.5 Hours): Time Series Forecasting & 5 Capstone Projects

All 9 lesson notebooks from Part 6 of the Data Mining Full Course in Python, in teaching order. Each lesson starts with a heading; the timestamps below jump…

What you'll learn

Datasets used in this lesson

Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.

📓 Full notebook

Download .ipynb

Data Mining Full Course, Part 6: Time Series Forecasting and 5 Capstone Projects#

All 9 lesson notebooks from Part 6 of the Data Mining Full Course in Python, in teaching order. Each lesson starts with a heading; the timestamps below jump to it in the video.

  • 0:00 Lesson 41: Introduction to Time-Series Data
  • 0:24:01 Lesson 42: Exponential Smoothing Techniques
  • 0:48:35 Lesson 43: Forecasting with ARIMA Models
  • 1:13:05 Lesson 44: Hands-on Project- Stock Price Prediction
  • 1:36:49 Lesson 45: Capstone Project- Customer Churn Prediction End-to-End
  • 2:00:41 Lesson 46: Capstone Project- Market Segmentation with Clustering
  • 2:27:06 Lesson 47: Capstone Project- Fraud Detection in Financial Transactions
  • 2:52:46 Lesson 48: Capstone Project- Sentiment Analysis using Text Mining
  • 3:17:22 Lesson 49: Capstone Project- Comprehensive Data Mining Case Study

Data files: save the datasets from the lesson page in the same folder as this notebook. Files the lessons create are written to that folder too.

Data Mining 41: Introduction to Time-Series Data#

In this lesson you'll be able to:

  • Turn date text into a proper time index
  • Slice, resample and repair gaps in time-stamped data
  • Spot trend and seasonality with charts and decomposition
  • Use lags, differences and autocorrelation
  • Split by time and score simple baseline forecasts Datasets: monthly airline passengers 1949-1960, daily Melbourne minimum temperatures 1981-1990

Warm-up: fraud data has a clock too#

Lesson 40 found fraud peaks at certain hours. Count all card transactions per hour.

import pandas as pd
cards = pd.read_csv("creditcard.csv.gz",
                    usecols=["Time", "Class"])
hour = (cards["Time"] // 3600).astype(int)
per_hour = hour.value_counts().sort_index()
print(len(per_hour), per_hour.min(), per_hour.max())
48 1082 9895
import matplotlib.pyplot as plt
per_hour.plot()
plt.title("Card transactions per hour over two days")
plt.xlabel("Hours since first transaction")
plt.ylabel("Transactions")
plt.show()
No description has been provided for this image

Turn date text into real dates#

Problem: CSV files store dates as plain text, so pandas can't do date maths on them yet.

raw = pd.read_csv("airline_passengers.csv")
print(raw.dtypes)
raw.head(3)
Month           str
Passengers    int64
dtype: object
Month Passengers
0 1949-01 112
1 1949-02 118
2 1949-03 132

Common mistake: using date tools on a text column

raw["Month"].dt.year
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[5], line 1
----> 1 raw["Month"].dt.year

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\generic.py:6194, in NDFrame.__getattr__(self, name)
   6187 if (
   6188     name not in self._internal_names_set
   6189     and name not in self._metadata
   6190     and name not in self._accessors
   6191     and self._info_axis._can_hold_identifiers_and_holds_name(name)
   6192 ):
   6193     return self[name]
-> 6194 return object.__getattribute__(self, name)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\accessor.py:230, in Accessor.__get__(self, obj, cls)
    227 if obj is None:
    228     # we're accessing the attribute of the class, i.e., Dataset.geo
    229     return self._accessor
--> 230 return self._accessor(obj)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\indexes\accessors.py:698, in CombinedDatetimelikeProperties.__new__(cls, data)
    695 elif isinstance(data.dtype, PeriodDtype):
    696     return PeriodProperties(data, orig)
--> 698 raise AttributeError("Can only use .dt accessor with datetimelike values")

AttributeError: Can only use .dt accessor with datetimelike values
raw["Month"] = pd.to_datetime(raw["Month"])
print(raw["Month"].dtype)
raw["Month"].dt.year.head(3)
datetime64[us]
0    1949
1    1949
2    1949
Name: Month, dtype: int32

Quick check: the file says 1949-01. Which exact date will pandas store for it?

air = pd.read_csv("airline_passengers.csv",
                  parse_dates=["Month"], index_col="Month")
air = air.asfreq("MS")
print(air.index[:2])
DatetimeIndex(['1949-01-01', '1949-02-01'], dtype='datetime64[us]', name='Month', freq='MS')
print(air.index.is_monotonic_increasing)
print(air.index.is_unique, len(air))
True
True 144

Slice and resample by time#

Problem: we often need one period, or the same data at a coarser frequency, like years instead of months.

y1960 = air.loc["1960", "Passengers"]
print(len(y1960), y1960.min(), y1960.max())
12 390 622
air.loc["1958-06":"1958-09"]
Passengers
Month
1958-06-01 435
1958-07-01 491
1958-08-01 505
1958-09-01 404
yearly = air["Passengers"].resample("YS").sum()
yearly.index = yearly.index.year
yearly.head()
Month
1949    1520
1950    1676
1951    2042
1952    2364
1953    2700
Name: Passengers, dtype: int64
dated = air.copy()
dated["year"] = dated.index.year
dated["month"] = dated.index.month_name().str[:3]
dated.tail(3)
Passengers year month
Month
1960-10-01 461 1960 Oct
1960-11-01 390 1960 Nov
1960-12-01 432 1960 Dec
quarters = air["Passengers"].resample("QS").sum()
quarters.tail(4)
Month
1960-01-01    1227
1960-04-01    1468
1960-07-01    1736
1960-10-01    1283
Freq: QS-JAN, Name: Passengers, dtype: int64

Your turn: find the average monthly passengers for each year. How do 1949 and 1960 compare?

avg = air["Passengers"].resample("YS").mean()
print(round(avg.iloc[0], 1), round(avg.iloc[-1], 1))
print(round(avg.iloc[-1] / avg.iloc[0], 2))
126.7 476.2
3.76

Find and repair gaps in daily data#

Problem: real daily data has missing days, and many time-series tools expect an unbroken calendar.

temps = pd.read_csv("daily_min_temperatures.csv",
                    parse_dates=["Date"], index_col="Date")
print(len(temps), temps.index.min().date(), temps.index.max().date())
3650 1981-01-01 1990-12-31
days = pd.date_range(temps.index.min(), temps.index.max())
print(len(days), len(days) - len(temps))
3652 2
temps = temps.asfreq("D")
temps[temps["Temp"].isna()]
Temp
Date
1984-12-31 NaN
1988-12-31 NaN

Quick check: fill a missing temperature with 0, or with something based on the days around it?

temps["Temp"] = temps["Temp"].interpolate(method="time")
temps.loc["1984-12-30":"1985-01-01"]
Temp
Date
1984-12-30 16.40
1984-12-31 14.85
1985-01-01 13.30
monthly = temps["Temp"].resample("MS").mean()
monthly.plot(color="steelblue")
plt.title("Melbourne monthly mean of daily minimum temperature")
plt.xlabel("Month")
plt.ylabel("Degrees Celsius")
plt.show()
No description has been provided for this image
t1990 = temps.loc["1990", "Temp"]
t1990.plot(alpha=0.4, label="daily")
t1990.rolling(30).mean().plot(label="30-day mean", linewidth=3)
plt.title("Melbourne 1990: daily noise vs the seasonal path")
plt.ylabel("Degrees Celsius")
plt.legend()
No description has been provided for this image

Spot trend and seasonality#

Problem: a raw line mixes several patterns; we want to see each one on its own.

  • Trend: the long-run direction (up, down, flat)
  • Seasonality: a pattern that repeats every period, like each year
  • Noise: what's left, the irregular wiggles
trend = air["Passengers"].rolling(window=12).mean()
air["Passengers"].plot(label="monthly", alpha=0.6)
trend.plot(label="12-month rolling mean", linewidth=3)
plt.title("Airline passengers: trend under the seasons")
plt.xlabel("Month")
plt.ylabel("Passengers (thousands)")
plt.legend()
No description has been provided for this image

Quick check: which month do you think is busiest for air travel?

profile = air.groupby(air.index.month)["Passengers"].mean()
print(profile.idxmax(), round(profile.max(), 1))
print(profile.idxmin(), round(profile.min(), 1))
7 351.3
11 232.8
swing = air["Passengers"].groupby(air.index.year).agg(["min", "max"])
swing["range"] = swing["max"] - swing["min"]
swing.iloc[[0, -1]]
min max range
Month
1949 104 148 44
1960 390 622 232
import seaborn as sns
grid = dated.pivot_table(index="month", columns="year",
                         values="Passengers", sort=False)
sns.heatmap(grid, cmap="YlGnBu")
plt.title("Passengers by month and year")
plt.xlabel("Year")
plt.ylabel("Month")
No description has been provided for this image
from statsmodels.tsa.seasonal import seasonal_decompose
parts = seasonal_decompose(air["Passengers"],
                           model="multiplicative", period=12)
parts.plot()
plt.show()
No description has been provided for this image
factors = parts.seasonal.iloc[:12]
factors.index = factors.index.month
factors.round(3)
Month
1     0.910
2     0.884
3     1.007
4     0.976
5     0.981
6     1.113
7     1.227
8     1.220
9     1.060
10    0.922
11    0.801
12    0.899
Name: seasonal, dtype: float64

Look back in time: lags, differences, autocorrelation#

Problem: forecasting uses the past, so we need columns that hold earlier values.

look = air[["Passengers"]].copy()
look["last_month"] = look["Passengers"].shift(1)
look["change"] = look["Passengers"].diff()
look["pct"] = look["Passengers"].pct_change().round(3)
look.head(3)
Passengers last_month change pct
Month
1949-01-01 112 NaN NaN NaN
1949-02-01 118 112.0 6.0 0.054
1949-03-01 132 118.0 14.0 0.119
p = air["Passengers"]
print(round(p.autocorr(lag=1), 3))
print(round(p.autocorr(lag=6), 3))
print(round(p.autocorr(lag=12), 3))
0.96
0.784
0.991
plt.scatter(p.shift(12), p, s=12)
plt.title("This month vs the same month last year")
plt.xlabel("Passengers 12 months earlier")
plt.ylabel("Passengers this month")
No description has been provided for this image
daily = temps["Temp"]
print(round(daily.autocorr(lag=1), 3))
print(round(daily.autocorr(lag=365), 3))
0.775
0.535
from statsmodels.graphics.tsaplots import plot_acf
fig = plot_acf(air["Passengers"].diff().dropna(), lags=36,
               title="Autocorrelation of monthly changes")
plt.xlabel("Lag (months)")
plt.ylabel("Correlation")
plt.show()
No description has been provided for this image

Your turn: use pct_change(12) to find the average year-on-year growth during 1960.

yoy = air["Passengers"].pct_change(12)
print(round(yoy.loc["1960"].mean() * 100, 1))
11.2

Split by time and score baseline forecasts#

Problem: a random split lets the model peek at the future. Train on the past, test on the future.

train = air.loc[:"1958", "Passengers"]
test = air.loc["1959":, "Passengers"]
print(len(train), len(test), train.iloc[-1])
120 24 337
  • Naive: repeat the last value
  • Average: repeat the mean of the last 12 months
  • Seasonal naive: repeat last year's months
import numpy as np
from sklearn.metrics import mean_absolute_error
naive = np.repeat(train.iloc[-1], len(test))
mean12 = np.repeat(train.iloc[-12:].mean(), len(test))
print(round(mean_absolute_error(test, naive), 1))
print(round(mean_absolute_error(test, mean12), 1))
115.2
77.8

Quick check: naive scored worst. Will copying last year's months do better than the average?

snaive = np.tile(train.iloc[-12:].to_numpy(), 2)
print(round(mean_absolute_error(test, snaive), 1))
71.2
plt.plot(test.index, test, color="black", label="actual")
plt.plot(test.index, naive, "--", label="naive")
plt.plot(test.index, snaive, "--", label="seasonal naive")
plt.title("Baseline forecasts vs reality, 1959-1960")
plt.xlabel("Month")
plt.ylabel("Passengers (thousands)")
plt.legend()
No description has been provided for this image

Your turn: traffic grows each year. Scale the seasonal naive forecast by last year's growth.

growth = train.iloc[-12:].sum() / train.iloc[-24:-12].sum()
scale = np.repeat([growth, growth ** 2], 12)
grown = snaive * scale
print(round(growth, 3))
print(round(mean_absolute_error(test, grown), 1))
1.034
51.5
avg_traffic = test.mean()
print(round(avg_traffic, 1))
print(round(mean_absolute_error(test, grown) / avg_traffic * 100, 1))
452.2
11.4

Mini-project: how predictable is Melbourne's weather?#

Forecast every day of 1990 three ways, and see which simple rule misses by the least.

t = temps["Temp"]
actual = t.loc["1990"]
yesterday = t.shift(1).loc["1990"]
last_year = t.shift(365).loc["1990"]
print(len(actual), actual.isna().sum())
365 0
past = t.loc[:"1989"]
normal = past.groupby(past.index.month).mean()
month_avg = actual.index.month.map(normal)
print(normal.round(2).loc[[1, 7]])
Date
1    14.97
7     6.53
Name: Temp, dtype: float64
for name, guess in [("yesterday", yesterday),
                    ("same day last year", last_year),
                    ("monthly average", month_avg)]:
    err = mean_absolute_error(actual, guess)
    print(f"{name:>20}: {err:.2f}")
           yesterday: 2.02
  same day last year: 2.87
     monthly average: 2.03
jan = actual.loc["1990-01"]
jan.plot(label="actual", color="black")
yesterday.loc["1990-01"].plot(label="yesterday rule", style="--")
plt.title("Yesterday rule, January 1990")
plt.ylabel("Degrees Celsius")
plt.legend()
No description has been provided for this image

Finding: yesterday and the monthly average both miss by about 2 degrees; same day last year is clearly worse.

Recap#

  • Parse dates and set them as the index before any time work
  • .loc slices by date; resample changes the frequency
  • asfreq reveals missing days; interpolate fills them
  • Trend, seasonality and noise can be pulled apart
  • Split by time and beat the naive baselines first

Challenges#

  1. Resample the temperatures to yearly means. Which year was warmest?
  2. Use a 7-day rolling mean (shifted by one day) as a fourth forecast rule.
  3. Decompose the monthly temperatures. Which month has the lowest seasonal value?

What's next#

Lesson 42: Exponential Smoothing Techniques, forecasts that learn trend and season.

Data Mining 42: Exponential Smoothing Techniques#

In this lesson you'll be able to:

  • Explain how exponential smoothing weights the past
  • Run simple exponential smoothing by hand and with statsmodels
  • Add trend (Holt) and seasonality (Holt-Winters)
  • Compare forecasts on a held-out period and forecast ahead Datasets: monthly airline passengers 1949-1960, monthly shampoo sales over 3 years

Warm-up: the score to beat#

Lesson 41 split by time and scored a seasonal naive forecast. Same split, same score.

import pandas as pd
import numpy as np
air = pd.read_csv("airline_passengers.csv",
                  parse_dates=["Month"], index_col="Month")
y = air["Passengers"].asfreq("MS")
train, test = y.loc[:"1958"], y.loc["1959":]
print(len(train), len(test))
120 24
from sklearn.metrics import mean_absolute_error
snaive = np.tile(train.iloc[-12:].to_numpy(), 2)
print(round(mean_absolute_error(test, snaive), 1))
71.2

Weigh recent months more#

Problem: a moving average treats a month from last year the same as last month.

  • New level = alpha x this month + (1 - alpha) x old level
  • alpha near 1: react fast; alpha near 0: stay calm and smooth
  • Older months fade away by a factor of (1 - alpha) each step
for alpha in (0.2, 0.5, 0.8):
    weights = [alpha * (1 - alpha) ** k for k in range(6)]
    print(alpha, np.round(weights, 3))
0.2 [0.2   0.16  0.128 0.102 0.082 0.066]
0.5 [0.5   0.25  0.125 0.062 0.031 0.016]
0.8 [0.8   0.16  0.032 0.006 0.001 0.   ]
import matplotlib.pyplot as plt
steps = np.arange(12)
plt.bar(steps - 0.2, 0.2 * 0.8 ** steps, width=0.4, label="alpha 0.2")
plt.bar(steps + 0.2, 0.8 * 0.2 ** steps, width=0.4, label="alpha 0.8")
plt.title("How much each past month counts")
plt.xlabel("Months back")
plt.ylabel("Weight")
plt.legend()
No description has been provided for this image

Quick check: with alpha = 0.8, how much weight is left for everything older than 2 months?

alpha = 0.8
recent = alpha + alpha * (1 - alpha)
print(round(recent, 2), round(1 - recent, 2))
0.96 0.04

Simple exponential smoothing, by hand and by library#

Problem: we want a forecast that follows the level of a series without chasing every wiggle.

alpha = 0.5
level = train.iloc[0]
for month, value in train.iloc[:6].items():
    level = alpha * value + (1 - alpha) * level
    print(month.strftime("%Y-%m"), value, round(level, 2))
1949-01 112 112.0
1949-02 118 115.0
1949-03 132 123.5
1949-04 129 126.25
1949-05 121 123.62
1949-06 135 129.31
from statsmodels.tsa.holtwinters import SimpleExpSmoothing
ses = SimpleExpSmoothing(train, initialization_method="known",
                         initial_level=train.iloc[0])
ses_fit = ses.fit(smoothing_level=0.5, optimized=False)
ses_fit.fittedvalues.iloc[1:7].round(2)
Month
1949-02-01    112.00
1949-03-01    115.00
1949-04-01    123.50
1949-05-01    126.25
1949-06-01    123.62
1949-07-01    129.31
Freq: MS, dtype: float64
print(ses_fit.forecast(3).round(2))
1959-01-01    345.75
1959-02-01    345.75
1959-03-01    345.75
Freq: MS, dtype: float64
recent = train.loc["1956":]
plt.plot(recent.index, recent, color="black", label="actual")
for a in (0.2, 0.8):
    fit_a = ses.fit(smoothing_level=a, optimized=False)
    plt.plot(recent.index, fit_a.fittedvalues.loc["1956":],
             "--", label=f"alpha {a}")
plt.title("Low alpha smooths, high alpha chases")
plt.legend()
No description has been provided for this image

Common mistake: forecasting before fitting

model = SimpleExpSmoothing(train, initialization_method="estimated")
model.forecast(12)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[11], line 2
      1 model = SimpleExpSmoothing(train, initialization_method="estimated")
----> 2 model.forecast(12)

AttributeError: 'SimpleExpSmoothing' object has no attribute 'forecast'
fitted = model.fit()
print(round(fitted.params["smoothing_level"], 2))
ses_fc = fitted.forecast(len(test))
print(round(mean_absolute_error(test, ses_fc), 2))
1.0
115.25
for a in (0.2, 0.5, 0.8, 1.0):
    sse = model.fit(smoothing_level=a, optimized=False).sse
    print(a, round(sse))
0.2 193114
0.5 147514
0.8 112569
1.0 98193
plt.plot(test.index, test, color="black", label="actual")
for a in (0.2, 0.8):
    f = model.fit(smoothing_level=a, optimized=False).forecast(24)
    plt.plot(test.index, f, "--", label=f"alpha {a}")
plt.title("Simple smoothing gives flat forecasts")
plt.ylabel("Passengers (thousands)")
plt.legend()
No description has been provided for this image

Add a trend: Holt's method#

Problem: simple smoothing forecasts a flat line, so it falls behind any growing series.

shampoo = pd.read_csv("shampoo.csv")
print(shampoo.dtypes)
shampoo.head(3)
Month        str
Sales    float64
dtype: object
Month Sales
0 1-01 266.0
1 1-02 145.9
2 1-03 183.1

The Month text 1-01 means year 1, month 01. No calendar year is given.

parts = shampoo["Month"].str.split("-", expand=True).astype(int)
shampoo["month_no"] = (parts[0] - 1) * 12 + parts[1]
sales = shampoo["Sales"]
shampoo[["Month", "month_no", "Sales"]].tail(3)
Month month_no Sales
33 3-10 34 475.3
34 3-11 35 581.3
35 3-12 36 646.9
plt.plot(shampoo["month_no"], sales, marker="o")
plt.title("Shampoo sales: a strong upward trend")
plt.xlabel("Month number")
plt.ylabel("Sales")
No description has been provided for this image

Quick check: will a trend help on shampoo? Guess how much the error drops.

from statsmodels.tsa.holtwinters import ExponentialSmoothing
s_train, s_test = sales.iloc[:30], sales.iloc[30:]
flat = SimpleExpSmoothing(s_train, initialization_method="estimated")
flat_fc = flat.fit().forecast(6)
holt = ExponentialSmoothing(s_train, trend="add",
                            initialization_method="estimated").fit()
holt_fc = holt.forecast(6)
print(round(mean_absolute_error(s_test, flat_fc), 1))
print(round(mean_absolute_error(s_test, holt_fc), 1))
156.2
100.5
print(holt.params["smoothing_level"].round(2))
print(holt.params["smoothing_trend"].round(2))
print(holt_fc.round(1).to_numpy())
0.2
0.2
[443.7 459.8 475.8 491.9 507.9 524. ]
print(holt.level.iloc[-1].round(1))
print(holt.trend.iloc[-1].round(1))
print((holt.level.iloc[-1] + holt.trend.iloc[-1]).round(1))
427.6
16.1
443.7
plt.plot(shampoo["month_no"], sales, color="black", label="actual")
test_no = shampoo["month_no"].iloc[30:]
plt.plot(test_no, flat_fc, "--", label="simple (flat)")
plt.plot(test_no, holt_fc, "--", label="Holt (trend)")
plt.title("Shampoo: Holt follows the trend")
plt.xlabel("Month number")
plt.legend()
No description has been provided for this image

Your turn: a straight-line trend can overshoot. Fit a damped trend (damped_trend=True) and score it.

damped = ExponentialSmoothing(s_train, trend="add", damped_trend=True,
                              initialization_method="estimated")
damped_fc = damped.fit().forecast(6)
print(round(mean_absolute_error(s_test, damped_fc), 1))
101.6
far_holt = holt.forecast(24)
far_damped = damped.fit().forecast(24)
print(far_holt.iloc[-1].round(1), far_damped.iloc[-1].round(1))
813.0 782.1

Add the seasons: Holt-Winters#

Problem: airline traffic has a trend and a yearly season, so we need a third smoothed part.

  • Additive season: July is always about N passengers above trend
  • Multiplicative season: July is always about N% above trend
hw_add = ExponentialSmoothing(train, trend="add", seasonal="add",
                              seasonal_periods=12,
                              initialization_method="estimated").fit()
add_fc = hw_add.forecast(len(test))
print(round(mean_absolute_error(test, add_fc), 1))
31.1

Quick check: lesson 41 showed the swings grow every year. Which season type should win?

hw_mul = ExponentialSmoothing(train, trend="add", seasonal="mul",
                              seasonal_periods=12,
                              initialization_method="estimated").fit()
mul_fc = hw_mul.forecast(len(test))
print(round(mean_absolute_error(test, mul_fc), 1))
29.0
names = ["smoothing_level", "smoothing_trend", "smoothing_seasonal"]
for name in names:
    print(name, round(hw_mul.params[name], 2))
smoothing_level 0.36
smoothing_trend 0.0
smoothing_seasonal 0.64
print(round(hw_add.aic), round(hw_mul.aic))
619 572
plt.plot(test.index, test, color="black", label="actual")
plt.plot(test.index, add_fc, "--", label="additive season")
plt.plot(test.index, mul_fc, "--", label="multiplicative season")
plt.title("Additive vs multiplicative seasons, 1959-1960")
plt.ylabel("Passengers (thousands)")
plt.legend()
No description has been provided for this image
factors = pd.Series(hw_mul.season[-12:].to_numpy(),
                    index=train.index[-12:].month_name().str[:3])
factors.plot(kind="bar", color="darkorange")
plt.axhline(1, color="grey")
plt.title("Holt-Winters seasonal factors")
plt.ylabel("Multiple of trend")
plt.show()
print(factors.idxmax(), round(factors.max(), 2))
No description has been provided for this image
Aug 1.19

Common mistake: passing the whole table, date text and all

raw = pd.read_csv("airline_passengers.csv")
ExponentialSmoothing(raw, trend="add")
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[30], line 2
      1 raw = pd.read_csv("data/mining/airline_passengers.csv")
----> 2 ExponentialSmoothing(raw, trend="add")

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\statsmodels\tsa\holtwinters\model.py:227, in ExponentialSmoothing.__init__(self, endog, trend, damped_trend, seasonal, seasonal_periods, initialization_method, initial_level, initial_trend, initial_seasonal, use_boxcox, bounds, dates, freq, missing)
    209 def __init__(
    210     self,
    211     endog,
   (...)
    225     missing="none",
    226 ):
--> 227     super().__init__(endog, None, dates, freq, missing=missing)
    228     self._y = self._data = array_like(
    229         endog, "endog", ndim=1, contiguous=True, order="C"
    230     )
    231     options = ("add", "mul", "additive", "multiplicative")

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\statsmodels\tsa\base\tsa_model.py:477, in TimeSeriesModel.__init__(self, endog, exog, dates, freq, missing, **kwargs)
    474 def __init__(
    475     self, endog, exog=None, dates=None, freq=None, missing="none", **kwargs
    476 ):
--> 477     super().__init__(endog, exog, missing=missing, **kwargs)
    479     # Date handling in indexes
    480     self._init_dates(dates, freq)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\statsmodels\base\model.py:287, in LikelihoodModel.__init__(self, endog, exog, **kwargs)
    286 def __init__(self, endog, exog=None, **kwargs):
--> 287     super().__init__(endog, exog, **kwargs)
    288     self.initialize()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\statsmodels\base\model.py:102, in Model.__init__(self, endog, exog, **kwargs)
    100 missing = kwargs.pop("missing", "none")
    101 hasconst = kwargs.pop("hasconst", None)
--> 102 self.data = self._handle_data(endog, exog, missing, hasconst, **kwargs)
    103 self.k_constant = self.data.k_constant
    104 self.exog = self.data.exog

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\statsmodels\base\model.py:143, in Model._handle_data(self, endog, exog, missing, hasconst, **kwargs)
    142 def _handle_data(self, endog, exog, missing, hasconst, **kwargs):
--> 143     data = handle_data(endog, exog, missing, hasconst, **kwargs)
    144     # kwargs arrays could have changed, easier to just attach here
    145     for key in kwargs:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\statsmodels\base\data.py:747, in handle_data(endog, exog, missing, hasconst, **kwargs)
    744     exog = np.asarray(exog)
    746 klass = handle_data_class_factory(endog, exog)
--> 747 return klass(endog, exog=exog, missing=missing, hasconst=hasconst, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\statsmodels\base\data.py:86, in ModelData.__init__(self, endog, exog, missing, hasconst, **kwargs)
     84     self.orig_endog = endog
     85     self.orig_exog = exog
---> 86     self.endog, self.exog = self._convert_endog_exog(endog, exog)
     88 self.const_idx = None
     89 self.k_constant = 0

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\statsmodels\base\data.py:553, in PandasData._convert_endog_exog(self, endog, exog)
    551 exog = exog if exog is None else np.asarray(exog, dtype=float)
    552 if endog.dtype == object:
--> 553     raise ValueError(
    554         "Pandas data cast to numpy dtype of object. "
    555         "Check input data with np.asarray(data)."
    556     )
    557 return super()._convert_endog_exog(endog, exog)

ValueError: Pandas data cast to numpy dtype of object. Check input data with np.asarray(data).
fixed = raw.set_index(pd.to_datetime(raw["Month"]))["Passengers"]
fixed = fixed.asfreq("MS")
print(fixed.dtype, fixed.index.freqstr, len(fixed))
int64 MS 144

Compare every forecast on the same months#

Problem: we need one table that ranks all methods on the same held-out months.

holt_air = ExponentialSmoothing(train, trend="add",
                              initialization_method="estimated")
holt_air_fc = holt_air.fit().forecast(len(test))
forecasts = {"seasonal naive": snaive, "simple": ses_fc,
             "Holt": holt_air_fc, "HW additive": add_fc,
             "HW multiplicative": mul_fc}
scores = pd.Series({name: mean_absolute_error(test, f)
                    for name, f in forecasts.items()})
scores.sort_values().round(2)
HW multiplicative     28.98
HW additive           31.08
seasonal naive        71.25
Holt                  91.62
simple               115.25
dtype: float64
plt.plot(test.index, test, color="black", linewidth=2, label="actual")
plt.plot(test.index, snaive, "--", label="seasonal naive")
plt.plot(test.index, mul_fc, "--", label="HW multiplicative")
plt.title("Holt-Winters vs seasonal naive, 1959-1960")
plt.ylabel("Passengers (thousands)")
plt.legend()
No description has been provided for this image

Your turn: in which test month did Holt-Winters miss by the most, and by how much?

miss = (test - mul_fc).abs()
print(miss.idxmax().strftime("%Y-%m"), round(miss.max(), 1))
print(round((test - mul_fc).mean(), 1))
1960-04 52.8
28.7
from sklearn.metrics import root_mean_squared_error
rmse = {name: root_mean_squared_error(test, f)
        for name, f in forecasts.items()}
pd.Series(rmse).sort_values().round(2)
HW multiplicative     32.49
HW additive           35.76
seasonal naive        76.99
Holt                 115.70
simple               137.33
dtype: float64
best = scores.min()
gain = 1 - best / scores["seasonal naive"]
print(round(best / test.mean() * 100, 1), round(gain * 100))
6.4 59

Mini-project: forecast 1961#

Refit the winner on all 12 years, forecast 1961, and report the busiest month and the growth.

final = ExponentialSmoothing(y, trend="add", seasonal="mul",
                             seasonal_periods=12,
                             initialization_method="estimated").fit()
f1961 = final.forecast(12)
f1961.index = f1961.index.month_name().str[:3]
f1961.round(0)
Jan    445.0
Feb    418.0
Mar    465.0
Apr    495.0
May    505.0
Jun    573.0
Jul    664.0
Aug    655.0
Sep    547.0
Oct    488.0
Nov    416.0
Dec    460.0
dtype: float64
print(f1961.idxmax(), round(f1961.max()))
total_1960 = y.loc["1960"].sum()
print(total_1960, round(f1961.sum()))
print(round((f1961.sum() / total_1960 - 1) * 100, 1))
Jul 664
5714 6132
7.3
last = y.loc["1960"].to_numpy()
growth = (f1961.to_numpy() / last - 1) * 100
pd.Series(growth, index=f1961.index).round(1).head(6)
Jan     6.8
Feb     7.0
Mar    11.1
Apr     7.4
May     7.1
Jun     7.2
dtype: float64
summer = ["Jun", "Jul", "Aug"]
extra = f1961[summer].sum() - y.loc["1960-06":"1960-08"].sum()
print(round(extra))
129
y.loc["1958":].plot(label="actual", color="black")
final.forecast(12).plot(label="1961 forecast", style="--")
plt.title("Airline passengers: Holt-Winters forecast for 1961")
plt.xlabel("Month")
plt.ylabel("Passengers (thousands)")
plt.legend()
No description has been provided for this image

Finding: the model expects another record year, peaking in summer. Forecasts are guides, not promises.

Recap#

  • Exponential smoothing weights recent values most; alpha sets how fast
  • Simple smoothing gives a flat forecast: fine for level-only series
  • Holt adds a trend; damping stops it overshooting
  • Holt-Winters adds seasons: multiplicative when swings grow
  • Fit first, score on held-out months, then refit on all data

Challenges#

  1. Hold out only 1960 (12 months). Does Holt-Winters still beat seasonal naive?
  2. Try damped_trend=True in the multiplicative Holt-Winters model.
  3. Apply Holt-Winters to monthly Melbourne temperatures from lesson 41.

What's next#

Lesson 43: Forecasting with ARIMA Models.

Data Mining 43: Forecasting with ARIMA Models#

In this lesson you'll be able to:

  • Test whether a series is stationary, and fix it with logs and differencing
  • Read ACF and PACF plots to choose the p, d and q orders
  • Fit ARIMA and seasonal ARIMA with statsmodels and forecast ahead
  • Judge a model on a held-out test period and check its residuals
  • Produce a forecast with a confidence band a planner can use Datasets: airline passengers 1949-1960 (data/mining/airline_passengers.csv), plus shampoo sales for practice

Warm-up: the benchmark from lesson 42#

Lesson 42 forecast with exponential smoothing. Its score is the bar ARIMA must clear.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
path = "airline_passengers.csv"
df = pd.read_csv(path, parse_dates=["Month"], index_col="Month")
y = df["Passengers"].asfreq("MS")
print(y.shape)
y.tail(3)
(144,)
Month
1960-10-01    461
1960-11-01    390
1960-12-01    432
Freq: MS, Name: Passengers, dtype: int64
from statsmodels.tsa.holtwinters import ExponentialSmoothing
train, test = y[:-24], y[-24:]
hw = ExponentialSmoothing(train, trend="add", seasonal="mul",
                          seasonal_periods=12).fit()
hw_pred = hw.forecast(24)
hw_mape = ((test - hw_pred).abs() / test).mean() * 100
print("Holt-Winters MAPE:", round(hw_mape, 1), "%")
Holt-Winters MAPE: 6.4 %

What ARIMA does, and why it needs a steady series#

Problem: ARIMA learns patterns that repeat, so the series must not drift.

  • AR, autoregressive: predict from the last few values
  • I, integrated: model the changes, not the levels
  • MA, moving average: correct using the last few errors The orders p, d and q say how much of each to use.
roll = y.rolling(12)
ax = y.plot(figsize=(9, 4), label="Passengers")
roll.mean().plot(ax=ax, label="12-month mean")
roll.std().plot(ax=ax, label="12-month std")
ax.set_title("Airline passengers: level and spread both grow")
ax.set_xlabel("Month")
ax.set_ylabel("Passengers (thousands)")
ax.legend()
No description has been provided for this image
from statsmodels.tsa.stattools import adfuller
adf = adfuller(y, result_object=True)
print("ADF statistic:", round(adf.statistic, 2))
print("p-value:", round(adf.pvalue, 3))
ADF statistic: 0.82
p-value: 0.992

Quick check#

With that p-value, can we call the raw series stationary?

is_steady = adf.pvalue < 0.05
print("Stationary?", is_steady)
Stationary? False

Make the series steady with logs and differencing#

Problem: remove the growing swings, the trend and the seasons.

log_y = np.log(y)
fig, axes = plt.subplots(1, 2, figsize=(10, 3.5))
y.plot(ax=axes[0], title="Raw passengers")
log_y.plot(ax=axes[1], title="Log of passengers")
axes[0].set_ylabel("Passengers (thousands)")
axes[1].set_ylabel("log(passengers)")
fig.tight_layout()
No description has been provided for this image
diff1 = log_y.diff()
check = log_y - log_y.shift(1)
print("Same as diff?", diff1.equals(check))
diff1.head(3).round(3)
Same as diff? True
Month
1949-01-01      NaN
1949-02-01    0.052
1949-03-01    0.112
Freq: MS, Name: Passengers, dtype: float64

Common mistake: testing a differenced series that still holds its missing first value

adfuller(diff1, result_object=True)
---------------------------------------------------------------------------
MissingDataError                          Traceback (most recent call last)
Cell In[9], line 1
----> 1 adfuller(diff1, result_object=True)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\statsmodels\tsa\stattools\_stattools.py:407, in adfuller(x, maxlag, regression, autolag, store, regresults, result_object)
    401 # 1 for level
    402 # search for lag length with smallest information criteria
    403 # Note: use the same number of observations to have comparable IC
    404 # aic and bic: smaller is better
    406 if not regresults:
--> 407     icbest, bestlag = _autolag(OLS, xdshort, fullRHS, startlag, maxlag, autolag)
    408 else:
    409     icbest, bestlag, alres = _autolag(
    410         OLS,
    411         xdshort,
   (...)
    416         regresults=regresults,
    417     )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\statsmodels\tsa\stattools\_stattools.py:148, in _autolag(mod, endog, exog, startlag, maxlag, method, modargs, fitargs, regresults)
    146 results = {}
    147 for lag in range(startlag, startlag + maxlag + 1):
--> 148     mod_instance = mod(endog, exog[:, :lag], *modargs)
    149     results[lag] = mod_instance.fit()
    151 if method == "aic":

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\statsmodels\regression\linear_model.py:1040, in OLS.__init__(self, endog, exog, missing, hasconst, **kwargs)
   1035     msg = (
   1036         "Weights are not supported in OLS and will be ignored"
   1037         "An exception will be raised in the next version."
   1038     )
   1039     warnings.warn(msg, ValueWarning, stacklevel=2)
-> 1040 super().__init__(endog, exog, missing=missing, hasconst=hasconst, **kwargs)
   1041 if "weights" in self._init_keys:
   1042     self._init_keys.remove("weights")

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\statsmodels\regression\linear_model.py:850, in WLS.__init__(self, endog, exog, weights, missing, hasconst, **kwargs)
    848 else:
    849     weights = weights.squeeze()
--> 850 super().__init__(
    851     endog, exog, missing=missing, weights=weights, hasconst=hasconst, **kwargs
    852 )
    853 nobs = self.exog.shape[0]
    854 weights = self.weights

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\statsmodels\regression\linear_model.py:246, in RegressionModel.__init__(self, endog, exog, **kwargs)
    245 def __init__(self, endog, exog, **kwargs):
--> 246     super().__init__(endog, exog, **kwargs)
    247     self.pinv_wexog: Float64Array | None = None
    248     self._data_attr.extend(["pinv_wexog", "wendog", "wexog", "weights"])

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\statsmodels\base\model.py:287, in LikelihoodModel.__init__(self, endog, exog, **kwargs)
    286 def __init__(self, endog, exog=None, **kwargs):
--> 287     super().__init__(endog, exog, **kwargs)
    288     self.initialize()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\statsmodels\base\model.py:102, in Model.__init__(self, endog, exog, **kwargs)
    100 missing = kwargs.pop("missing", "none")
    101 hasconst = kwargs.pop("hasconst", None)
--> 102 self.data = self._handle_data(endog, exog, missing, hasconst, **kwargs)
    103 self.k_constant = self.data.k_constant
    104 self.exog = self.data.exog

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\statsmodels\base\model.py:143, in Model._handle_data(self, endog, exog, missing, hasconst, **kwargs)
    142 def _handle_data(self, endog, exog, missing, hasconst, **kwargs):
--> 143     data = handle_data(endog, exog, missing, hasconst, **kwargs)
    144     # kwargs arrays could have changed, easier to just attach here
    145     for key in kwargs:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\statsmodels\base\data.py:747, in handle_data(endog, exog, missing, hasconst, **kwargs)
    744     exog = np.asarray(exog)
    746 klass = handle_data_class_factory(endog, exog)
--> 747 return klass(endog, exog=exog, missing=missing, hasconst=hasconst, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\statsmodels\base\data.py:90, in ModelData.__init__(self, endog, exog, missing, hasconst, **kwargs)
     88 self.const_idx = None
     89 self.k_constant = 0
---> 90 self._handle_constant(hasconst)
     91 self._check_integrity()
     92 self._cache = {}

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\statsmodels\base\data.py:139, in ModelData._handle_constant(self, hasconst)
    137 exog_max = np.max(self.exog, axis=0)
    138 if not np.isfinite(exog_max).all():
--> 139     raise MissingDataError("exog contains inf or nans")
    140 exog_min = np.min(self.exog, axis=0)
    141 const_idx = np.where(exog_max == exog_min)[0].squeeze()

MissingDataError: exog contains inf or nans
adf1 = adfuller(diff1.dropna(), result_object=True)
print("p-value after one difference:", round(adf1.pvalue, 3))
p-value after one difference: 0.071
diff12 = diff1.diff(12).dropna()
adf12 = adfuller(diff12, result_object=True)
print("p-value after seasonal difference:", round(adf12.pvalue, 4))
print(len(diff12), "months left")
p-value after seasonal difference: 0.0002
131 months left
ax = diff12.plot(figsize=(9, 3.5))
ax.axhline(0, color="grey", linewidth=1)
ax.set_title("Log passengers after both differences")
ax.set_xlabel("Month")
ax.set_ylabel("Change in log(passengers)")
No description has been provided for this image

Your turn#

Load data/mining/shampoo.csv. Test the Sales column, then its first difference. Is one difference enough?

shampoo = pd.read_csv("shampoo.csv")
sales = shampoo["Sales"]
p_raw = adfuller(sales, result_object=True).pvalue
p_d1 = adfuller(sales.diff().dropna(), result_object=True).pvalue
print("Raw sales p-value:", round(p_raw, 3))
print("Steady after one difference?", p_d1 < 0.05)
Raw sales p-value: 1.0
Steady after one difference? True

Read p and q from the ACF and PACF plots#

Problem: which past months and past errors actually matter?

  • ACF: how strongly each month echoes the month k steps back
  • PACF: the same echo with the in-between months removed A sharp ACF cut-off hints at q; a sharp PACF cut-off hints at p.
from statsmodels.graphics.tsaplots import plot_acf, plot_pacf
fig, axes = plt.subplots(1, 2, figsize=(11, 3.5))
plot_acf(diff12, lags=24, ax=axes[0])
plot_pacf(diff12, lags=24, ax=axes[1], method="ywm")
for ax in axes:
    ax.set_xlabel("Lag (months)")
fig.tight_layout()
No description has been provided for this image
from statsmodels.tsa.stattools import acf
values = pd.Series(acf(diff12, nlags=24))
band = 1.96 / np.sqrt(len(diff12))
strong = values[values.abs() > band].drop(0)
strong.round(3)
1    -0.341
3    -0.202
9     0.176
12   -0.387
23    0.223
dtype: float64

Quick check#

The two biggest spikes are negative, at lag 1 and lag 12. Which orders do they suggest?

  • Big lag 1 spike: q = 1, one past error
  • Lag 12 spike: a seasonal q of 1, last year's error This is the famous airline model: (0, 1, 1) x (0, 1, 1, 12).

Fit a first ARIMA and forecast#

Problem: turn chosen orders into a fitted model and real forecasts.

from statsmodels.tsa.arima.model import ARIMA
log_train = np.log(train)
plain = ARIMA(log_train, order=(1, 1, 1))
plain_fit = plain.fit()
plain_fit.params.round(3)
ar.L1    -0.576
ma.L1     0.835
sigma2    0.010
dtype: float64

Common mistake: asking the model, rather than the fitted result, for a forecast

plain.forecast(24)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[17], line 1
----> 1 plain.forecast(24)

AttributeError: 'ARIMA' object has no attribute 'forecast'
plain_pred = np.exp(plain_fit.forecast(24))
plain_pred.head(3).round(1)
1959-01-01    358.5
1959-02-01    346.0
1959-03-01    353.1
Freq: MS, Name: predicted_mean, dtype: float64
def mape(actual, pred):
    return ((actual - pred).abs() / actual).mean() * 100

snaive = y[-36:-24].to_list() * 2
snaive = pd.Series(snaive, index=test.index)
print("Seasonal naive:", round(mape(test, snaive), 1))
print("ARIMA(1,1,1):  ", round(mape(test, plain_pred), 1))
Seasonal naive: 15.5
ARIMA(1,1,1):   20.6
ax = train[-48:].plot(figsize=(9, 4), label="Train")
test.plot(ax=ax, label="Actual")
plain_pred.plot(ax=ax, label="ARIMA(1,1,1)", style="--")
ax.set_title("Plain ARIMA misses the summer peaks")
ax.set_xlabel("Month")
ax.set_ylabel("Passengers (thousands)")
ax.legend()
No description has been provided for this image

Add the seasons with seasonal ARIMA#

Problem: teach ARIMA that July looks like last July.

sarima = ARIMA(log_train, order=(0, 1, 1),
               seasonal_order=(0, 1, 1, 12))
sar_fit = sarima.fit()
print(sar_fit.params.round(3))
print("AIC:", round(sar_fit.aic, 1))
ma.L1      -0.343
ma.S.L12   -0.541
sigma2      0.001
dtype: float64
AIC: -389.0
sar_pred = np.exp(sar_fit.forecast(24))
print("Seasonal ARIMA:", round(mape(test, sar_pred), 1))
print("Holt-Winters:  ", round(hw_mape, 1))
Seasonal ARIMA: 8.5
Holt-Winters:   6.4
ax = test.plot(figsize=(9, 4), label="Actual", linewidth=2)
sar_pred.plot(ax=ax, label="Seasonal ARIMA", style="--")
hw_pred.plot(ax=ax, label="Holt-Winters", style=":")
ax.set_title("Both seasonal models track the peaks, 1959-1960")
ax.set_xlabel("Month")
ax.set_ylabel("Passengers (thousands)")
ax.legend()
No description has been provided for this image
rows = []
for p, q in [(0, 1), (1, 0), (1, 1)]:
    fit = ARIMA(log_train, order=(p, 1, q),
                seasonal_order=(0, 1, 1, 12)).fit()
    err = mape(test, np.exp(fit.forecast(24)))
    rows.append({"order": (p, 1, q), "aic": fit.aic, "mape": err})
pd.DataFrame(rows).round(1)
order aic mape
0 (0, 1, 1) -389.0 8.5
1 (1, 1, 0) -388.6 9.0
2 (1, 1, 1) -387.3 7.8

Choose with AIC (training data only). Test MAPE is the final check, not a dial to tune.

Your turn#

Fit a seasonal ARIMA with a seasonal p of 1: seasonal_order=(1, 1, 1, 12). Does the AIC improve?

alt = ARIMA(log_train, order=(0, 1, 1),
            seasonal_order=(1, 1, 1, 12)).fit()
print("AIC with seasonal p=1:", round(alt.aic, 1))
print("Airline model AIC:   ", round(sar_fit.aic, 1))
AIC with seasonal p=1: -387.2
Airline model AIC:    -389.0

Check the leftovers with residual diagnostics#

Problem: a good model leaves only random noise behind.

resid = sar_fit.resid[13:]
print("Mean:", round(resid.mean(), 4))
print("Std: ", round(resid.std(), 4))
Mean: 0.0003
Std:  0.0386
fig, axes = plt.subplots(1, 2, figsize=(10, 3.5))
resid.plot(ax=axes[0], title="Residuals over time")
resid.plot.hist(ax=axes[1], bins=20, title="Residual spread")
axes[0].set_ylabel("Error (log units)")
axes[1].set_xlabel("Error (log units)")
fig.tight_layout()
No description has been provided for this image
from statsmodels.stats.diagnostic import acorr_ljungbox
acorr_ljungbox(resid, lags=[6, 12, 24]).round(3)
lb_stat lb_pvalue
6 4.882 0.559
12 6.073 0.912
24 17.985 0.804

Quick check#

All p-values are above 0.05. Is there pattern left for the model to learn?

lb = acorr_ljungbox(resid, lags=[6, 12, 24])
print("Leftover pattern?", (lb["lb_pvalue"] < 0.05).any())
Leftover pattern? False

Mini-project: plan capacity for 1961#

Question: how many passengers should the airline plan for, and how sure can it be?

scores = {
    "Seasonal naive": mape(test, snaive),
    "ARIMA(1,1,1)": mape(test, plain_pred),
    "Seasonal ARIMA": mape(test, sar_pred),
    "Holt-Winters": hw_mape,
}
pd.Series(scores, name="MAPE %").sort_values().round(1)
Holt-Winters       6.4
Seasonal ARIMA     8.5
Seasonal naive    15.5
ARIMA(1,1,1)      20.6
Name: MAPE %, dtype: float64
final = ARIMA(np.log(y), order=(0, 1, 1),
              seasonal_order=(0, 1, 1, 12)).fit()
fc = final.get_forecast(12)
plan = np.exp(fc.conf_int(alpha=0.05))
plan.columns = ["low", "high"]
plan["forecast"] = np.exp(fc.predicted_mean)
plan.round(0).head(6)
low high forecast
1961-01-01 419.0 484.0 450.0
1961-02-01 391.0 463.0 426.0
1961-03-01 436.0 526.0 479.0
1961-04-01 444.0 546.0 492.0
1961-05-01 455.0 570.0 509.0
1961-06-01 517.0 658.0 583.0
ax = y[-36:].plot(figsize=(9, 4), label="Actual")
plan["forecast"].plot(ax=ax, label="1961 forecast", style="--")
ax.fill_between(plan.index, plan["low"], plan["high"], alpha=0.25,
                label="95% interval")
ax.set_title("Airline passengers: forecast for 1961")
ax.set_xlabel("Month")
ax.set_ylabel("Passengers (thousands)")
ax.legend()
No description has been provided for this image
total_1960 = y["1960"].sum()
total_1961 = plan["forecast"].sum()
growth = (total_1961 / total_1960 - 1) * 100
peak = plan["forecast"].idxmax().strftime("%B")
print(round(total_1960), round(total_1961), round(growth, 1), peak)
5714 6240 9.2 July

Finding: plan for about 9% more passengers in 1961, peaking in July, with the band widening each month.

Recap#

  • ARIMA = past values (p) + differencing (d) + past errors (q)
  • Test stationarity with adfuller; fix it with logs, diff and seasonal diff
  • ACF and PACF spikes suggest q and p; AIC compares candidates
  • Seasonal data needs seasonal_order; always beat a naive baseline
  • Check residuals with Ljung-Box, then forecast with get_forecast intervals

Challenges#

  1. Fit ARIMA to the shampoo sales (no seasons) and score the last 6 months.
  2. Refit the airline model on the raw series, without logs. How does MAPE change?
  3. Hold back 12 months instead of 24. Do the model rankings stay the same?

What's next#

Lesson 44: Hands-on Project- Stock Price Prediction Real share prices, a naive benchmark, and an honest look at what can be predicted.

Data Mining 44: Hands-on Project- Stock Price Prediction#

In this lesson you'll be able to:

  • Turn prices into daily returns and test them for patterns
  • Build the naive forecast that every price model must beat
  • Engineer past-only features without leaking the future
  • Compare regression, random forest and ARIMA with walk-forward tests
  • Judge a trading signal honestly against buy-and-hold Dataset: daily adjusted closing prices, 2007-2016, for MSFT, IBM, SBUX, AAPL and the S&P 500 (data/pandas/stocks.csv)

Warm-up: stationarity from lesson 43#

Lesson 43 tested series with adfuller before modelling. Try it on Apple.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
path = "stocks.csv"
s = pd.read_csv(path, parse_dates=["Date"], index_col="Date")
print(s.shape)
s.head(3).round(2)
(2306, 5)
MSFT IBM SBUX AAPL GSPC
Date
2007-01-03 23.95 80.52 16.15 11.09 1416.60
2007-01-04 23.91 81.38 16.17 11.33 1418.34
2007-01-05 23.77 80.64 16.10 11.25 1409.71
from statsmodels.tsa.stattools import adfuller
aapl = s["AAPL"]
p_price = adfuller(aapl, result_object=True).pvalue
p_ret = adfuller(aapl.pct_change().dropna(),
                 result_object=True).pvalue
print("Price p-value: ", round(p_price, 3))
print("Return p-value below 0.001?", p_ret < 0.001)
Price p-value:  0.846
Return p-value below 0.001? True

Look before you predict: nine years of prices#

Problem: compare five series on very different price scales.

growth = s / s.iloc[0] * 100
ax = growth.plot(figsize=(9, 4))
ax.set_title("Growth of 100 invested on 3 January 2007")
ax.set_xlabel("Date")
ax.set_ylabel("Value (start = 100)")
No description has been provided for this image
growth.iloc[-1].sort_values(ascending=False).round(0)
AAPL    907.0
SBUX    372.0
MSFT    220.0
IBM     167.0
GSPC    140.0
Name: 2016-03-01 00:00:00, dtype: float64

Common mistake: computing returns on a table that still holds the Date text column

raw = pd.read_csv(path)
raw.pct_change()
---------------------------------------------------------------------------
ArrowNotImplementedError                  Traceback (most recent call last)
File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\arrow\array.py:1097, in ArrowExtensionArray._evaluate_op_method(self, other, op, arrow_funcs)
   1096 try:
-> 1097     result = pc_func(self._pa_array, other)
   1098 except pa.ArrowNotImplementedError as err:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\arrow\array.py:184, in <lambda>(x, y)
    175         return result
    177     ARROW_ARITHMETIC_FUNCS = {
    178         "add": pc.add_checked,
    179         "radd": lambda x, y: pc.add_checked(y, x),
    180         "sub": pc.subtract_checked,
    181         "rsub": lambda x, y: pc.subtract_checked(y, x),
    182         "mul": pc.multiply_checked,
    183         "rmul": lambda x, y: pc.multiply_checked(y, x),
--> 184         "truediv": lambda x, y: pc.divide(*cast_for_truediv(x, y)),
    185         "rtruediv": lambda x, y: pc.divide(*cast_for_truediv(y, x)),
    186         "floordiv": lambda x, y: floordiv_compat(x, y),
    187         "rfloordiv": lambda x, y: floordiv_compat(y, x),
    188         "mod": NotImplemented,
    189         "rmod": NotImplemented,
    190         "divmod": NotImplemented,
    191         "rdivmod": NotImplemented,
    192         "pow": pc.power_checked,
    193         "rpow": lambda x, y: pc.power_checked(y, x),
    194     }
    196 if TYPE_CHECKING:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pyarrow\compute.py:255, in _make_generic_wrapper.<locals>.wrapper(memory_pool, *args)
    254     return Expression._call(func_name, list(args))
--> 255 return func.call(args, None, memory_pool)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pyarrow\_compute.pyx:405, in pyarrow._compute.Function.call()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pyarrow\error.pxi:155, in pyarrow.lib.pyarrow_internal_check_status()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pyarrow\error.pxi:92, in pyarrow.lib.check_status()

ArrowNotImplementedError: Function 'divide' has no kernel matching input types (large_string, large_string)

The above exception was the direct cause of the following exception:

TypeError                                 Traceback (most recent call last)
Cell In[6], line 2
      1 raw = pd.read_csv(path)
----> 2 raw.pct_change()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\generic.py:11561, in NDFrame.pct_change(self, periods, fill_method, freq, **kwargs)
  11559 shifted = self.shift(periods=periods, freq=freq, axis=axis, **kwargs)
  11560 # Unsupported left operand type for / ("Self")
> 11561 rs = self / shifted - 1  # type: ignore[operator]
  11562 if freq is not None:
  11563     # Shift method is implemented differently when freq is not None
  11564     # We want to restore the original index
  11565     rs = rs.loc[~rs.index.duplicated()]

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\ops\common.py:85, in _unpack_zerodim_and_defer.<locals>.new_method(self, other)
     82     other = sanitize_array(other, None)
     83     other = ensure_wrapped_if_datetimelike(other)
---> 85 return method(self, other)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arraylike.py:214, in OpsMixin.__truediv__(self, other)
    212 @unpack_zerodim_and_defer("__truediv__")
    213 def __truediv__(self, other):
--> 214     return self._arith_method(other, operator.truediv)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\frame.py:9153, in DataFrame._arith_method(self, other, op)
   9150 self, other = self._align_for_op(other, axis, flex=True, level=None)
   9152 with np.errstate(all="ignore"):
-> 9153     new_data = self._dispatch_frame_op(other, op, axis=axis)
   9154 return self._construct_result(new_data, other=other)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\frame.py:9196, in DataFrame._dispatch_frame_op(self, right, func, axis)
   9190     assert self.columns.equals(right.columns)
   9191     # TODO: The previous assertion `assert right._indexed_same(self)`
   9192     #  fails in cases with empty columns reached via
   9193     #  _frame_arith_method_with_reindex
   9194 
   9195     # TODO operate_blockwise expects a manager of the same type
-> 9196     bm = self._mgr.operate_blockwise(
   9197         right._mgr,
   9198         array_op,
   9199     )
   9200     return self._constructor_from_mgr(bm, axes=bm.axes)
   9202 elif isinstance(right, Series) and axis == 1:
   9203     # axis=1 means we want to operate row-by-row

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\managers.py:1704, in BlockManager.operate_blockwise(self, other, array_op)
   1700 def operate_blockwise(self, other: BlockManager, array_op) -> BlockManager:
   1701     """
   1702     Apply array_op blockwise with another (aligned) BlockManager.
   1703     """
-> 1704     return operate_blockwise(self, other, array_op)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\ops.py:65, in operate_blockwise(left, right, array_op)
     63 res_blks: list[Block] = []
     64 for lvals, rvals, locs, left_ea, right_ea, rblk in _iter_block_pairs(left, right):
---> 65     res_values = array_op(lvals, rvals)
     66     if (
     67         left_ea
     68         and not right_ea
     69         and hasattr(res_values, "reshape")
     70         and not is_1d_only_ea_dtype(res_values.dtype)
     71     ):
     72         res_values = res_values.reshape(1, -1)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\ops\array_ops.py:279, in arithmetic_op(left, right, op)
    270 right = ensure_wrapped_if_datetimelike(right)
    272 if (
    273     should_extension_dispatch(left, right)
    274     or isinstance(right, (Timedelta, BaseOffset, Timestamp))
   (...)
    277     # Timedelta/Timestamp and other custom scalars are included in the check
    278     # because numexpr will fail on it, see GH#31457
--> 279     res_values = op(left, right)
    280 else:
    281     # TODO we should handle EAs consistently and move this check before the if/else
    282     # (https://github.com/pandas-dev/pandas/issues/41165)
    283     # error: Argument 2 to "_bool_arith_check" has incompatible type
    284     # "Union[ExtensionArray, ndarray[Any, Any]]"; expected "ndarray[Any, Any]"
    285     _bool_arith_check(op, left, right)  # type: ignore[arg-type]

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\ops\common.py:85, in _unpack_zerodim_and_defer.<locals>.new_method(self, other)
     82     other = sanitize_array(other, None)
     83     other = ensure_wrapped_if_datetimelike(other)
---> 85 return method(self, other)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arraylike.py:214, in OpsMixin.__truediv__(self, other)
    212 @unpack_zerodim_and_defer("__truediv__")
    213 def __truediv__(self, other):
--> 214     return self._arith_method(other, operator.truediv)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\arrow\array.py:1158, in ArrowExtensionArray._arith_method(self, other, op)
   1154 if pa.types.is_string(self._pa_array.type) or pa.types.is_large_string(
   1155     self._pa_array.type
   1156 ):
   1157     try:
-> 1158         result = self._evaluate_op_method(other, op, ARROW_ARITHMETIC_FUNCS)
   1159     except (pa.ArrowInvalid, pa.ArrowTypeError):
   1160         result = self._str_arith_method_object_fallback(other, op)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\arrow\array.py:1099, in ArrowExtensionArray._evaluate_op_method(self, other, op, arrow_funcs)
   1097     result = pc_func(self._pa_array, other)
   1098 except pa.ArrowNotImplementedError as err:
-> 1099     raise TypeError(self._op_method_error_message(other_original, op)) from err
   1100 return self._from_pyarrow_array(result)

TypeError: operation 'truediv' not supported for dtype 'str' with dtype 'str'
rets = s.pct_change().dropna()
rets.head(3).round(4)
MSFT IBM SBUX AAPL GSPC
Date
2007-01-04 -0.0017 0.0107 0.0011 0.0222 0.0012
2007-01-05 -0.0057 -0.0091 -0.0043 -0.0071 -0.0061
2007-01-08 0.0098 0.0152 -0.0037 0.0049 0.0022

Daily returns: what actually changes#

Problem: find out whether yesterday's move tells us anything about today's.

rets.describe().loc[["mean", "std", "min", "max"]].round(4)
MSFT IBM SBUX AAPL GSPC
mean 0.0005 0.0003 0.0008 0.0012 0.0002
std 0.0183 0.0144 0.0213 0.0213 0.0136
min -0.1171 -0.0828 -0.1097 -0.1792 -0.0903
max 0.1860 0.1151 0.1838 0.1390 0.1158
ax = rets["AAPL"].plot.hist(bins=80, figsize=(8, 3.5))
ax.set_title("Apple daily returns, 2007-2016")
ax.set_xlabel("Daily return (0.05 = 5%)")
ax.set_ylabel("Number of days")
No description has been provided for this image
rets["AAPL"].nsmallest(3).round(3)
Date
2008-09-29   -0.179
2013-01-24   -0.124
2008-01-23   -0.106
Name: AAPL, dtype: float64

Quick check#

If Apple rose yesterday, is it more likely to rise today? Guess, then check.

r = rets["AAPL"]
print("Return vs yesterday:", round(r.autocorr(1), 3))
print("Size vs yesterday:  ", round(r.abs().autocorr(1), 3))
Return vs yesterday: -0.003
Size vs yesterday:   0.232
vol = r.rolling(20).std() * 100
ax = vol.plot(figsize=(9, 3.5))
ax.set_title("Apple 20-day volatility: storms come in clusters")
ax.set_xlabel("Date")
ax.set_ylabel("Daily std (%)")
print("Calmest:", round(vol.min(), 2), " Stormiest:", round(vol.max(), 2))
Calmest: 0.46  Stormiest: 7.41
No description has been provided for this image
ax = plt.figure(figsize=(5, 5)).gca()
ax.scatter(r.shift(1), r, s=4, alpha=0.4)
ax.set_title("Apple: today vs yesterday")
ax.set_xlabel("Yesterday's return")
ax.set_ylabel("Today's return")
No description has been provided for this image
rets.corr().round(2)["AAPL"]
MSFT    0.47
IBM     0.49
SBUX    0.43
AAPL    1.00
GSPC    0.61
Name: AAPL, dtype: float64

The naive forecast: tomorrow equals today#

Problem: set the bar any model has to clear.

cut = int(len(aapl) * 0.8)
train, test = aapl[:cut], aapl[cut:]
start = test.index[0]
print(len(train), len(test), start.date())
1844 462 2014-05-01
naive = aapl.shift(1)[cut:]
naive_mae = (test - naive).abs().mean()
naive_mape = ((test - naive).abs() / test).mean() * 100
print("Naive MAE: $", round(naive_mae, 3))
print("Naive MAPE:", round(naive_mape, 2), "%")
Naive MAE: $ 1.274
Naive MAPE: 1.17 %
ax = test[:60].plot(figsize=(9, 4), label="Actual")
naive[:60].plot(ax=ax, label="Naive forecast", style="--")
ax.set_title("A forecast that looks brilliant, and is just a lag")
ax.set_xlabel("Date")
ax.set_ylabel("Price ($)")
ax.legend()
No description has been provided for this image
from sklearn.metrics import r2_score
print("R2 of naive prices:", round(r2_score(test, naive), 3))
R2 of naive prices: 0.983
err = (test - naive).abs()
by_year = err.groupby(err.index.year).mean()
by_year.round(3)
Date
2014    0.890
2015    1.478
2016    1.619
Name: AAPL, dtype: float64

Warning: a forecast that hugs the price line may be yesterday's price in disguise. Always compare with naive.

Your turn#

Compute the naive MAPE for Microsoft (MSFT) over the same test days.

msft = s["MSFT"]
m_naive = msft.shift(1)[cut:]
m_err = (msft[cut:] - m_naive).abs() / msft[cut:]
print("MSFT naive MAPE:", round(m_err.mean() * 100, 2), "%")
MSFT naive MAPE: 1.07 %

Build features from the past only#

Problem: describe each day using only what was known at that day's close.

feat = pd.DataFrame({"price": aapl, "ret": aapl.pct_change()})
for k in range(1, 5):
    feat[f"ret_lag{k}"] = feat["ret"].shift(k)
feat["ma5_gap"] = aapl / aapl.rolling(5).mean() - 1
feat["ma20_gap"] = aapl / aapl.rolling(20).mean() - 1
feat["vol20"] = feat["ret"].rolling(20).std()
feat.isna().sum()
price        0
ret          1
ret_lag1     2
ret_lag2     3
ret_lag3     4
ret_lag4     5
ma5_gap      4
ma20_gap    19
vol20       20
dtype: int64
feat["target"] = feat["ret"].shift(-1)
feat["next_price"] = aapl.shift(-1)
cols = ["ret"] + [c for c in feat.columns if c.startswith(
        ("ret_lag", "ma", "vol"))]
print(cols)
feat[["ret", "target"]].tail(3).round(4)
['ret', 'ret_lag1', 'ret_lag2', 'ret_lag3', 'ret_lag4', 'ma5_gap', 'ma20_gap', 'vol20']
ret target
Date
2016-02-26 0.0016 -0.0023
2016-02-29 -0.0023 0.0397
2016-03-01 0.0397 NaN

Common mistake: fitting on rows that rolling windows and shifts left empty

from sklearn.linear_model import LinearRegression
LinearRegression().fit(feat[cols], feat["target"])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[23], line 2
      1 from sklearn.linear_model import LinearRegression
----> 2 LinearRegression().fit(feat[cols], feat["target"])

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py:1403, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1393     estimator._validate_params()
   1395 with (
   1396     config_context(
   1397         skip_parameter_validation=(
   (...)
   1401     callback_management_context(estimator),
   1402 ):
-> 1403     return fit_method(estimator, *args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\linear_model\_base.py:681, in LinearRegression.fit(self, X, y, sample_weight)
    677 n_jobs_ = self.n_jobs
    679 accept_sparse = False if self.positive else ["csr", "csc", "coo"]
--> 681 X, y = validate_data(
    682     self,
    683     X,
    684     y,
    685     accept_sparse=accept_sparse,
    686     y_numeric=True,
    687     multi_output=True,
    688     force_writeable=True,
    689 )
    691 has_sw = sample_weight is not None
    692 if has_sw:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:3058, in validate_data(_estimator, X, y, reset, validate_separately, skip_check_array, **check_params)
   3056         y = check_array(y, input_name="y", **check_y_params)
   3057     else:
-> 3058         X, y = check_X_y(X, y, **check_params)
   3059     out = X, y
   3061 if not no_val_X and check_params.get("ensure_2d", True):

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:1330, in check_X_y(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, ensure_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, estimator)
   1325         estimator_name = _check_estimator_name(estimator)
   1326     raise ValueError(
   1327         f"{estimator_name} requires y to be passed, but the target y is None"
   1328     )
-> 1330 X = check_array(
   1331     X,
   1332     accept_sparse=accept_sparse,
   1333     accept_large_sparse=accept_large_sparse,
   1334     dtype=dtype,
   1335     order=order,
   1336     copy=copy,
   1337     force_writeable=force_writeable,
   1338     ensure_all_finite=ensure_all_finite,
   1339     ensure_2d=ensure_2d,
   1340     allow_nd=allow_nd,
   1341     ensure_min_samples=ensure_min_samples,
   1342     ensure_min_features=ensure_min_features,
   1343     estimator=estimator,
   1344     input_name="X",
   1345 )
   1347 y = _check_y(y, multi_output=multi_output, y_numeric=y_numeric, estimator=estimator)
   1349 check_consistent_length(X, y)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:1090, in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, ensure_all_finite, ensure_non_negative, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)
   1084     raise ValueError(
   1085         f"Found array with dim {array.ndim},"
   1086         f" while dim <= 2 is required{context}."
   1087     )
   1089 if ensure_all_finite:
-> 1090     _assert_all_finite(
   1091         array,
   1092         input_name=input_name,
   1093         estimator_name=estimator_name,
   1094         allow_nan=ensure_all_finite == "allow-nan",
   1095     )
   1097 if copy:
   1098     if _is_numpy_namespace(xp):
   1099         # only make a copy if `array` and `array_orig` may share memory`

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:137, in _assert_all_finite(X, allow_nan, msg_dtype, estimator_name, input_name)
    134 if first_pass_isfinite:
    135     return
--> 137 _assert_all_finite_element_wise(
    138     X,
    139     xp=xp,
    140     allow_nan=allow_nan,
    141     msg_dtype=msg_dtype,
    142     estimator_name=estimator_name,
    143     input_name=input_name,
    144 )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:186, in _assert_all_finite_element_wise(X, xp, allow_nan, msg_dtype, estimator_name, input_name)
    169 if estimator_name and input_name == "X" and has_nan_error:
    170     # Improve the error message on how to handle missing values in
    171     # scikit-learn.
    172     msg_err += (
    173         f"\n{estimator_name} does not accept missing values"
    174         " encoded as NaN natively. For supervised learning, you might want"
   (...)
    184         "#estimators-that-handle-nan-values"
    185     )
--> 186 raise ValueError(msg_err)

ValueError: Input X contains NaN.
LinearRegression does not accept missing values encoded as NaN natively. For supervised learning, you might want to consider sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept missing values encoded as NaNs natively. Alternatively, it is possible to preprocess the data, for instance by using an imputer transformer in a pipeline or drop samples with missing values. See https://scikit-learn.org/stable/modules/impute.html You can find a list of all estimators that handle NaN values at the following page: https://scikit-learn.org/stable/modules/impute.html#estimators-that-handle-nan-values
data = feat.dropna()
tr = data[data.index < start]
te = data[data.index >= start]
print(len(feat), len(data), len(tr), len(te))
2306 2285 1824 461
links = tr[cols].corrwith(tr["target"])
links.sort_values().round(3)
ret_lag1   -0.041
vol20      -0.029
ma5_gap    -0.013
ret        -0.005
ret_lag2    0.001
ret_lag4    0.006
ma20_gap    0.009
ret_lag3    0.064
dtype: float64

Can a model beat the naive forecast?#

Problem: predict tomorrow's return, then turn it into a price.

lin = LinearRegression().fit(tr[cols], tr["target"])
lin_ret = lin.predict(te[cols])
print("R2 on test:", round(lin.score(te[cols], te["target"]), 3))
print("Mean predicted return:", round(lin_ret.mean(), 5))
R2 on test: -0.015
Mean predicted return: 0.00163
def price_mae(pred_ret):
    pred_price = te["price"] * (1 + pred_ret)
    return (te["next_price"] - pred_price).abs().mean()

base = price_mae(0)
print("Naive:  ", round(base, 4))
print("Linear: ", round(price_mae(lin_ret), 4))
Naive:   1.2762
Linear:  1.2872
from sklearn.ensemble import RandomForestRegressor
rf = RandomForestRegressor(n_estimators=200, min_samples_leaf=20,
                           random_state=42, n_jobs=-1)
rf.fit(tr[cols], tr["target"])
rf_ret = rf.predict(te[cols])
print("Forest: ", round(price_mae(rf_ret), 4))
Forest:  1.3086
imp = pd.Series(rf.feature_importances_, index=cols)
imp.sort_values(ascending=False).round(3)
ma20_gap    0.160
ma5_gap     0.137
ret_lag4    0.137
vol20       0.133
ret_lag3    0.130
ret         0.107
ret_lag2    0.104
ret_lag1    0.094
dtype: float64
board = pd.Series({
    "Naive": base,
    "Linear": price_mae(lin_ret),
    "Random forest": price_mae(rf_ret),
    "Average return": price_mae(tr["target"].mean()),
}, name="MAE $")
board.sort_values().round(4)
Naive             1.2762
Average return    1.2809
Linear            1.2872
Random forest     1.3086
Name: MAE $, dtype: float64

Quick check#

Up days are about half the test period. Is 51% direction accuracy impressive?

up = te["target"] > 0
print("Share of up days:  ", round(up.mean(), 3))
print("Linear direction:  ", round(((lin_ret > 0) == up).mean(), 3))
print("Forest direction:  ", round(((rf_ret > 0) == up).mean(), 3))
Share of up days:   0.508
Linear direction:   0.495
Forest direction:   0.514

ARIMA and walk-forward checks#

Problem: try the lesson 43 tool, and test on more than one period.

from statsmodels.tsa.arima.model import ARIMA
log_p = np.log(aapl).reset_index(drop=True)
arima = ARIMA(log_p[:cut], order=(1, 1, 1)).fit()
arima.params.round(3)
ar.L1    -0.387
ma.L1     0.400
sigma2    0.001
dtype: float64
walk = arima.append(log_p[cut:])
ar_pred = np.exp(walk.fittedvalues[cut:])
ar_mae = (test.to_numpy() - ar_pred.to_numpy())
print("ARIMA MAE:", round(np.abs(ar_mae).mean(), 3))
print("Naive MAE:", round(naive_mae, 3))
ARIMA MAE: 1.275
Naive MAE: 1.274
  • Walk-forward testing: train on the past, test on the next block, then slide forward scikit-learn's TimeSeriesSplit makes the blocks, never shuffling time.
from sklearn.model_selection import TimeSeriesSplit
rows = []
for a, b in TimeSeriesSplit(n_splits=5).split(data):
    f_tr, f_te = data.iloc[a], data.iloc[b]
    m = LinearRegression().fit(f_tr[cols], f_tr["target"])
    pred = f_te["price"] * (1 + m.predict(f_te[cols]))
    err = (f_te["next_price"] - pred).abs().mean()
    nv = (f_te["next_price"] - f_te["price"]).abs().mean()
    rows.append({"start": f_te.index[0].date(), "naive": nv,
                 "model": err})
folds = pd.DataFrame(rows).round(3)
folds
start naive model
0 2008-08-12 0.359 0.357
1 2010-02-16 0.473 0.483
2 2011-08-17 0.957 0.963
3 2013-02-22 0.738 0.738
4 2014-08-26 1.406 1.422

Your turn#

In how many folds does the model beat naive, and by how much at most?

gap = folds["naive"] - folds["model"]
print("Folds won:", (gap > 0).sum(), "of", len(folds))
print("Best margin: $", round(gap.max(), 3))
Folds won: 1 of 5
Best margin: $ 0.002

Mini-project: would the signal have made money?#

Question: hold Apple only on days the linear model predicts a rise. Does that beat buy-and-hold?

signal = pd.Series(lin_ret > 0, index=te.index)
strat = te["target"] * signal
hold = te["target"]
curves = pd.DataFrame({"Signal": (1 + strat).cumprod(),
                       "Buy and hold": (1 + hold).cumprod()})
print("Days invested:", round(signal.mean() * 100, 1), "%")
(curves.iloc[-1] - 1).mul(100).round(1)
Days invested: 91.3 %
Signal          12.6
Buy and hold    23.5
Name: 2016-02-29 00:00:00, dtype: float64
ax = curves.mul(100).plot(figsize=(9, 4))
ax.set_title("Signal vs buy-and-hold on the test period")
ax.set_xlabel("Date")
ax.set_ylabel("Value of 100 invested")
No description has been provided for this image
worst = curves / curves.cummax() - 1
print("Worst drop (%):")
worst.min().mul(100).round(1)
Worst drop (%):
Signal         -40.1
Buy and hold   -28.9
dtype: float64
yearly = curves.groupby(curves.index.year).last()
yearly.round(3)
Signal Buy and hold
Date
2014 1.333 1.313
2015 1.144 1.287
2016 1.126 1.235

Finding: no model beat the naive forecast in a useful way, and the signal earned about half of buy-and-hold.

Recap#

  • Model returns, not raw prices: prices drift, returns are steadier
  • The naive forecast is the bar, and it is very hard to clear
  • Build features from the past only, and split strictly by time
  • Check across several periods with TimeSeriesSplit
  • A chart that hugs the price line can be just a one-day lag

Challenges#

  1. Repeat the model comparison for the S&P 500 (GSPC).
  2. Add yesterday's S&P 500 return as a feature. Does Apple's error improve?
  3. Predict weekly instead of daily returns. Is naive still as hard to beat?

What's next#

Lesson 45: Capstone Project- Customer Churn Prediction End-to-End Pipelines, cross-validation, a cost-based threshold and a saved model.

Data Mining 45: Capstone Project- Customer Churn Prediction End-to-End#

In this lesson you'll be able to:

  • Bundle cleaning, encoding and a model into one scikit-learn Pipeline
  • Compare and tune models with cross-validation on training data only
  • Choose a decision threshold from business costs, not the default 0.5
  • Save the finished model, reload it and score new customers
  • Turn predictions into a costed retention plan Dataset: IBM Telco customer churn, 7,043 customers (data/mining/telco_churn.csv)

Warm-up: the baseline to beat#

Lesson 44 showed that every model must beat a naive baseline. Here is churn's.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
df = pd.read_csv("telco_churn.csv")
print(df.shape)
df["Churn"].value_counts(normalize=True).round(3)
(7043, 21)
Churn
No     0.735
Yes    0.265
Name: proportion, dtype: float64
from sklearn.dummy import DummyClassifier
y_all = (df["Churn"] == "Yes").astype(int)
lazy = DummyClassifier(strategy="most_frequent")
lazy.fit(df[["tenure"]], y_all)
print("Accuracy:", round(lazy.score(df[["tenure"]], y_all), 3))
Accuracy: 0.735

Set the data up once, the right way#

Problem: split before learning anything, and keep some real customers back for the very end.

new_customers = df.tail(5)
known = df.iloc[:-5]
X = known.drop(columns=["customerID", "Churn"])
y = (known["Churn"] == "Yes").astype(int)
print(X.shape, y.mean().round(3))
(7038, 19) 0.265
num_cols = ["tenure", "MonthlyCharges", "TotalCharges"]
cat_cols = [c for c in X.columns if c not in num_cols]
print(len(num_cols), "numeric,", len(cat_cols), "categorical")
print(cat_cols[:4])
3 numeric, 16 categorical
['gender', 'SeniorCitizen', 'Partner', 'Dependents']
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)
print(len(X_train), len(X_test))
print(y_train.mean().round(3), y_test.mean().round(3))
5630 1408
0.265 0.266

The test set now stays locked until the final check. Every choice uses training data only.

Build one pipeline for every preparation step#

Problem: apply the same cleaning to training rows, test rows and next month's customers.

  • ColumnTransformer: different recipes for different columns
  • Numbers: fill gaps with the median, then scale
  • Categories: one-hot encode, ignoring unseen values
  • Pipeline: preparation and model, fitted together
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
numeric = Pipeline([("fill", SimpleImputer(strategy="median")),
                    ("scale", StandardScaler())])
prep = ColumnTransformer([
    ("num", numeric, num_cols),
    ("cat", OneHotEncoder(handle_unknown="ignore"), cat_cols)])

Common mistake: fitting the pipeline while TotalCharges is still text

prep.fit(X_train)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[8], line 1
----> 1 prep.fit(X_train)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\compose\_column_transformer.py:924, in ColumnTransformer.fit(self, X, y, **params)
    921 _raise_for_params(params, self, "fit")
    922 # we use fit_transform to make sure to set sparse_output_ (for which we
    923 # need the transformed data) to have consistent output type in predict
--> 924 self.fit_transform(X, y=y, **params)
    925 return self

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_set_output.py:327, in _wrap_method_output.<locals>.wrapped(self, X, *args, **kwargs)
    325 @wraps(f)
    326 def wrapped(self, X, *args, **kwargs):
--> 327     data_to_wrap = f(self, X, *args, **kwargs)
    328     if isinstance(data_to_wrap, tuple):
    329         # only wrap the first output for cross decomposition
    330         return_tuple = (
    331             _wrap_data_with_container(method, data_to_wrap[0], X, self),
    332             *data_to_wrap[1:],
    333         )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py:1403, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1393     estimator._validate_params()
   1395 with (
   1396     config_context(
   1397         skip_parameter_validation=(
   (...)
   1401     callback_management_context(estimator),
   1402 ):
-> 1403     return fit_method(estimator, *args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\compose\_column_transformer.py:977, in ColumnTransformer.fit_transform(self, X, y, **params)
    972 # ``params`` is empty unless routing is enabled (guaranteed by
    973 # ``_raise_for_params`` above), so the call below produces a properly
    974 # shaped empty routing in the disabled case.
    975 routed_params = process_routing(self, "fit_transform", **params)
--> 977 result = self._call_func_on_transformers(
    978     X,
    979     y,
    980     _fit_transform_one,
    981     column_as_labels=False,
    982     routed_params=routed_params,
    983 )
    985 if not result:
    986     self._update_fitted_transformers([])

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\compose\_column_transformer.py:887, in ColumnTransformer._call_func_on_transformers(self, X, y, func, column_as_labels, routed_params)
    875             extra_args = {}
    876         jobs.append(
    877             delayed(func)(
    878                 transformer=clone(trans) if not fitted else trans,
   (...)
    884             )
    885         )
--> 887     return Parallel(n_jobs=self.n_jobs)(jobs)
    889 except ValueError as e:
    890     if "Expected 2D array, got 1D array instead" in str(e):

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\parallel.py:91, in Parallel.__call__(self, iterable)
     79 warning_filters = (
     80     filters_func() if filters_func is not None else warnings.filters
     81 )
     83 iterable_with_config_and_warning_filters = (
     84     (
     85         _with_config_and_warning_filters(delayed_func, config, warning_filters),
   (...)
     89     for delayed_func, args, kwargs in iterable
     90 )
---> 91 return super().__call__(iterable_with_config_and_warning_filters)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\joblib\parallel.py:2012, in Parallel.__call__(self, iterable)
   2010     output = self._get_sequential_output(iterable)
   2011     next(output)
-> 2012     return output if self.return_generator else list(output)
   2014 # Let's create an ID that uniquely identifies the current call. If the
   2015 # call is interrupted early and that the same instance is immediately
   2016 # reused, this id will be used to prevent workers that were
   2017 # concurrently finalizing a task from the previous call to run the
   2018 # callback.
   2019 with self._lock:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\joblib\parallel.py:1940, in Parallel._get_sequential_output(self, iterable)
   1938 self.n_dispatched_batches += 1
   1939 self.n_dispatched_tasks += 1
-> 1940 res = func(*args, **kwargs)
   1941 self.n_completed_tasks += 1
   1942 self.print_progress()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\parallel.py:184, in _FuncWrapper.__call__(self, *args, **kwargs)
    180                 this_warning_filter_dict[special_key] = this_value.pattern
    182         warnings.filterwarnings(**this_warning_filter_dict, append=True)
--> 184 return self.function(*args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\pipeline.py:1566, in _fit_transform_one(transformer, X, y, weight, message_clsname, message, params)
   1564 with _print_elapsed_time(message_clsname, message):
   1565     if hasattr(transformer, "fit_transform"):
-> 1566         res = transformer.fit_transform(X, y, **params.get("fit_transform", {}))
   1567     else:
   1568         res = transformer.fit(X, y, **params.get("fit", {})).transform(
   1569             X, **params.get("transform", {})
   1570         )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py:1403, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1393     estimator._validate_params()
   1395 with (
   1396     config_context(
   1397         skip_parameter_validation=(
   (...)
   1401     callback_management_context(estimator),
   1402 ):
-> 1403     return fit_method(estimator, *args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\pipeline.py:712, in Pipeline.fit_transform(self, X, y, **params)
    709 callback_ctx.call_on_fit_task_begin(estimator=self, X=X, y=y)
    711 routed_params = self._check_method_params(method="fit_transform", props=params)
--> 712 Xt = self._fit(
    713     X, y, routed_params, raw_params=params, callback_ctx=callback_ctx
    714 )
    716 last_step = self._final_estimator
    717 with _print_elapsed_time("Pipeline", self._log_message(len(self.steps) - 1)):

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\pipeline.py:561, in Pipeline._fit(self, X, y, routed_params, raw_params, callback_ctx)
    554 # Fit or load from cache the current transformer
    555 step_params = self._get_metadata_for_step(
    556     step_idx=step_idx,
    557     step_params=routed_params[name],
    558     all_params=raw_params,
    559 )
--> 561 X, fitted_transformer = fit_transform_one_cached(
    562     cloned_transformer,
    563     X,
    564     y,
    565     weight=None,
    566     message_clsname="Pipeline",
    567     message=self._log_message(step_idx),
    568     params=step_params,
    569     caller=self,
    570     callback_ctx=subcontext,
    571 )
    572 # Replace the transformer of the step with the fitted
    573 # transformer. This is necessary when loading the transformer
    574 # from the cache.
    575 self.steps[step_idx] = (name, fitted_transformer)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\joblib\memory.py:331, in NotMemorizedFunc.__call__(self, *args, **kwargs)
    330 def __call__(self, *args, **kwargs):
--> 331     return self.func(*args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\pipeline.py:1593, in _fit_transform_one_with_callbacks(transformer, X, y, weight, message_clsname, message, params, caller, callback_ctx)
   1590 with callback_ctx.propagate_callback_context(transformer):
   1591     callback_ctx.call_on_fit_task_begin(estimator=caller, X=X, y=y)
-> 1593     Xt, transformer = _fit_transform_one(
   1594         transformer, X, y, weight, message_clsname, message, params
   1595     )
   1597     callback_ctx.call_on_fit_task_end(estimator=caller, X=Xt, y=y)
   1599 return Xt, transformer

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\pipeline.py:1566, in _fit_transform_one(transformer, X, y, weight, message_clsname, message, params)
   1564 with _print_elapsed_time(message_clsname, message):
   1565     if hasattr(transformer, "fit_transform"):
-> 1566         res = transformer.fit_transform(X, y, **params.get("fit_transform", {}))
   1567     else:
   1568         res = transformer.fit(X, y, **params.get("fit", {})).transform(
   1569             X, **params.get("transform", {})
   1570         )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_set_output.py:327, in _wrap_method_output.<locals>.wrapped(self, X, *args, **kwargs)
    325 @wraps(f)
    326 def wrapped(self, X, *args, **kwargs):
--> 327     data_to_wrap = f(self, X, *args, **kwargs)
    328     if isinstance(data_to_wrap, tuple):
    329         # only wrap the first output for cross decomposition
    330         return_tuple = (
    331             _wrap_data_with_container(method, data_to_wrap[0], X, self),
    332             *data_to_wrap[1:],
    333         )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py:969, in TransformerMixin.fit_transform(self, X, y, **fit_params)
    954         warnings.warn(
    955             (
    956                 f"This object ({self.__class__.__name__}) has a `transform`"
   (...)
    964             UserWarning,
    965         )
    967 if y is None:
    968     # fit method of arity 1 (unsupervised transformation)
--> 969     return self.fit(X, **fit_params).transform(X)
    970 else:
    971     # fit method of arity 2 (supervised transformation)
    972     return self.fit(X, y, **fit_params).transform(X)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py:1403, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1393     estimator._validate_params()
   1395 with (
   1396     config_context(
   1397         skip_parameter_validation=(
   (...)
   1401     callback_management_context(estimator),
   1402 ):
-> 1403     return fit_method(estimator, *args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\impute\_base.py:448, in SimpleImputer.fit(self, X, y)
    430 @_fit_context(prefer_skip_nested_validation=True)
    431 def fit(self, X, y=None):
    432     """Fit the imputer on `X`.
    433 
    434     Parameters
   (...)
    446         Fitted estimator.
    447     """
--> 448     X = self._validate_input(X, in_fit=True)
    450     # default fill_value is 0 for numerical input and "missing_value"
    451     # otherwise
    452     if self.fill_value is None:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\impute\_base.py:373, in SimpleImputer._validate_input(self, X, in_fit)
    367 if "could not convert" in str(ve):
    368     new_ve = ValueError(
    369         "Cannot use {} strategy with non-numeric data:\n{}".format(
    370             self.strategy, ve
    371         )
    372     )
--> 373     raise new_ve from None
    374 else:
    375     raise ve

ValueError: Cannot use median strategy with non-numeric data:
could not convert string to float: ' '
for part in (X_train, X_test, new_customers):
    part["TotalCharges"] = pd.to_numeric(
        part["TotalCharges"], errors="coerce")
print(X_train["TotalCharges"].isna().sum(), "gaps in training")
7 gaps in training
prep.fit(X_train)
names = prep.get_feature_names_out()
print(len(names), "model inputs")
print(names[:3], names[-2:])
46 model inputs
['num__tenure' 'num__MonthlyCharges' 'num__TotalCharges'] ['cat__PaymentMethod_Electronic check' 'cat__PaymentMethod_Mailed check']

Quick check#

The imputer fills gaps with a median. Which rows was that median computed from?

fill = prep.named_transformers_["num"]["fill"]
print("Learned median:", fill.statistics_.round(2))
print("Train median:  ", X_train[num_cols].median().to_numpy())
Learned median: [  29.     70.55 1390.85]
Train median:   [  29.     70.55 1390.85]

Compare models with cross-validation#

Problem: one train/test split can be lucky; we need a steadier score.

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
logit = Pipeline([("prep", prep),
                  ("model", LogisticRegression(max_iter=1000))])
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
auc = cross_val_score(logit, X_train, y_train, cv=cv,
                      scoring="roc_auc")
print(auc.round(3), round(auc.mean(), 3))
[0.851 0.849 0.833 0.859 0.836] 0.846

The preparation is refitted inside every fold, so no fold ever learns from the rows it is scored on.

from sklearn.tree import DecisionTreeClassifier
tree = Pipeline([("prep", prep),
                 ("model", DecisionTreeClassifier(
                     max_depth=5, random_state=42))])
t_auc = cross_val_score(tree, X_train, y_train, cv=cv,
                        scoring="roc_auc")
print(t_auc.round(3), round(t_auc.mean(), 3))
[0.83  0.83  0.809 0.84  0.826] 0.827

Your turn#

Cross-validate a RandomForestClassifier pipeline (100 trees, min_samples_leaf=5, random_state=42) on the same folds.

from sklearn.ensemble import RandomForestClassifier
forest = Pipeline([("prep", prep),
                   ("model", RandomForestClassifier(
                       n_estimators=100, min_samples_leaf=5,
                       random_state=42))])
f_auc = cross_val_score(forest, X_train, y_train, cv=cv,
                        scoring="roc_auc")
print(f_auc.round(3), round(f_auc.mean(), 3))
[0.851 0.851 0.834 0.864 0.831] 0.846
board = pd.DataFrame({"mean_auc": [auc.mean(), t_auc.mean(),
                                   f_auc.mean()],
                      "spread": [auc.std(), t_auc.std(),
                                 f_auc.std()]},
                     index=["logistic", "tree", "forest"])
board.sort_values("mean_auc", ascending=False).round(3)
mean_auc spread
forest 0.846 0.012
logistic 0.846 0.010
tree 0.827 0.010
from sklearn.model_selection import cross_validate
scores = cross_validate(logit, X_train, y_train, cv=cv,
                        scoring=["precision", "recall", "f1"])
pd.DataFrame(scores).filter(like="test").mean().round(3)
test_precision    0.652
test_recall       0.552
test_f1           0.597
dtype: float64

Tune the winner with GridSearchCV#

Problem: pick settings fairly, using cross-validation on training rows only.

from sklearn.model_selection import GridSearchCV
grid = {"model__C": np.logspace(-2, 1, 4),
        "model__class_weight": [None, "balanced"]}
search = GridSearchCV(logit, grid, cv=cv, scoring="roc_auc")
search.fit(X_train, y_train)
print(pd.Series(search.best_params_).to_string())
print("Best CV AUC:", round(search.best_score_, 4))
model__C               10.0
model__class_weight     NaN
Best CV AUC: 0.8461
res = pd.DataFrame(search.cv_results_)
res = res[["param_model__C", "param_model__class_weight",
           "mean_test_score"]]
res.sort_values("mean_test_score", ascending=False).head(4).round(4)
param_model__C param_model__class_weight mean_test_score
6 10.0 NaN 0.8461
7 10.0 balanced 0.8459
4 1.0 NaN 0.8457
5 1.0 balanced 0.8455

Quick check#

The top four settings are within 0.001 AUC. Is tuning worth much here?

scores = res["mean_test_score"]
print("Best minus worst:", round(scores.max() - scores.min(), 4))
Best minus worst: 0.0033
best = search.best_estimator_
from sklearn.metrics import roc_auc_score
test_proba = best.predict_proba(X_test)[:, 1]
print("Test AUC:", round(roc_auc_score(y_test, test_proba), 3))
Test AUC: 0.838
coefs = pd.Series(best["model"].coef_[0],
                  index=best["prep"].get_feature_names_out())
coefs = coefs.sort_values()
pd.concat([coefs.head(3), coefs.tail(3)]).round(2)
num__MonthlyCharges                -2.13
num__tenure                        -1.37
cat__InternetService_DSL           -1.17
cat__Contract_Month-to-month        0.57
num__TotalCharges                   0.67
cat__InternetService_Fiber optic    1.37
dtype: float64

Choose the threshold by business cost#

Problem: 0.5 is a default, not a business decision.

Assumed costs for this lesson (swap in your own):

  • Contacting a customer with an offer: $100
  • Missing a customer who then leaves: $500 in lost revenue
from sklearn.model_selection import cross_val_predict
oof = cross_val_predict(best, X_train, y_train, cv=cv,
                        method="predict_proba")[:, 1]

def cost(proba, truth, cut, offer=100, lost=500):
    called = proba >= cut
    missed = (~called) & (truth == 1)
    return called.sum() * offer + missed.sum() * lost
print(len(oof), round(oof.mean(), 3))
5630 0.266
cuts = np.arange(0.05, 0.95, 0.05).round(2)
bill = pd.Series([cost(oof, y_train.to_numpy(), c) for c in cuts],
                 index=cuts)
best_cut = bill.idxmin()
print("Cheapest threshold:", best_cut)
bill.loc[[0.1, best_cut, 0.5, 0.8]]
Cheapest threshold: 0.15
0.10    396100
0.15    374700
0.50    460800
0.80    731300
dtype: int64
ax = (bill / 1000).plot(figsize=(8, 3.5), marker="o")
ax.axvline(best_cut, color="grey", linestyle="--")
ax.set_title("Cost of each threshold, training customers")
ax.set_xlabel("Threshold (call if probability >= this)")
ax.set_ylabel("Total cost ($ thousands)")
No description has been provided for this image
from sklearn.metrics import precision_score, recall_score
for c in (0.5, best_cut):
    flag = oof >= c
    print(c, round(precision_score(y_train, flag), 3),
          round(recall_score(y_train, flag), 3))
0.5 0.65 0.554
0.15 0.44 0.914

Your turn#

Calls get dearer: $200 each. Find the cheapest threshold with the cost helper.

dear = pd.Series([cost(oof, y_train.to_numpy(), c, offer=200)
                  for c in cuts], index=cuts)
print("Cheapest threshold at $200 a call:", dear.idxmin())
Cheapest threshold at $200 a call: 0.45

Quick check#

Why is the cheapest threshold well below 0.5?

A miss costs 5x a call, so calling pays whenever churn risk tops about 100 / 500 = 0.2.

Save the model and reload it#

Problem: hand the finished pipeline and its threshold to someone else.

import os
import joblib
model_path = "m45_churn_model.joblib"
package = {"pipeline": best, "threshold": float(best_cut)}
joblib.dump(package, model_path)
print("Saved:", os.path.exists(model_path))
Saved: True
loaded = joblib.load(model_path)
model, cut = loaded["pipeline"], loaded["threshold"]
same = (model.predict_proba(X_test)[:, 1] == test_proba).all()
print("Threshold:", cut, " Identical:", same)
Threshold: 0.15  Identical: True

Common mistake: sending new rows that are missing a column the pipeline expects

model.predict_proba(new_customers.drop(columns=["tenure"]))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[29], line 1
----> 1 model.predict_proba(new_customers.drop(columns=["tenure"]))

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\pipeline.py:925, in Pipeline.predict_proba(self, X, **params)
    923 if not _routing_enabled():
    924     for _, name, transform in self._iter(with_final=False):
--> 925         Xt = transform.transform(Xt)
    926     return self.steps[-1][1].predict_proba(Xt, **params)
    928 # metadata routing enabled

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_set_output.py:327, in _wrap_method_output.<locals>.wrapped(self, X, *args, **kwargs)
    325 @wraps(f)
    326 def wrapped(self, X, *args, **kwargs):
--> 327     data_to_wrap = f(self, X, *args, **kwargs)
    328     if isinstance(data_to_wrap, tuple):
    329         # only wrap the first output for cross decomposition
    330         return_tuple = (
    331             _wrap_data_with_container(method, data_to_wrap[0], X, self),
    332             *data_to_wrap[1:],
    333         )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\compose\_column_transformer.py:1066, in ColumnTransformer.transform(self, X, **params)
   1064     diff = all_names - set(column_names)
   1065     if diff:
-> 1066         raise ValueError(f"columns are missing: {diff}")
   1067 else:
   1068     # ndarray was used for fitting or transforming, thus we only
   1069     # check that n_features_in_ is consistent
   1070     _check_n_features(self, X, reset=False)

ValueError: columns are missing: {'tenure'}
fresh = new_customers.drop(columns=["customerID", "Churn"])
risk = model.predict_proba(fresh)[:, 1]
result = new_customers[["customerID", "Contract", "tenure"]].copy()
result["risk"] = risk.round(3)
result["call"] = risk >= cut
result["actual"] = new_customers["Churn"]
result
customerID Contract tenure risk call actual
7038 6840-RESVB One year 24 0.117 False No
7039 2234-XADUH One year 72 0.149 False No
7040 4801-JZAZL Month-to-month 11 0.398 True No
7041 8361-LTMKD Month-to-month 4 0.718 True Yes
7042 3186-AJIEK Two year 66 0.048 False No

Mini-project: price the retention plan#

Question: on the locked test customers, what does our plan cost compared with the alternatives?

truth = y_test.to_numpy()
plans = pd.Series({
    "Call nobody": cost(test_proba, truth, 1.01),
    "Call everyone": cost(test_proba, truth, 0.0),
    "Default 0.5": cost(test_proba, truth, 0.5),
    "Cost-based": cost(test_proba, truth, cut),
}, name="cost $")
plans.sort_values()
Cost-based        93900
Default 0.5      116700
Call everyone    140800
Call nobody      187000
Name: cost $, dtype: int64
called = test_proba >= cut
caught = (called & (truth == 1)).sum()
print("Customers called:", called.sum(), "of", len(truth))
print("Churners reached:", caught, "of", truth.sum())
saving = plans["Call nobody"] - plans["Cost-based"]
print("Saving vs doing nothing: $", saving)
Customers called: 769 of 1408
Churners reached: 340 of 374
Saving vs doing nothing: $ 93100
ax = (plans / 1000).sort_values().plot.barh(figsize=(7, 3))
ax.set_title("Cost of each retention plan, test customers")
ax.set_xlabel("Total cost ($ thousands)")
ax.set_ylabel("Plan")
No description has been provided for this image

Finding: the cost-based threshold is the cheapest plan, reaching most churners for far less than doing nothing.

Recap#

  • Split first, and lock the test set until the final check
  • ColumnTransformer + Pipeline = preparation that cannot leak
  • Cross-validate to compare models and tune settings
  • Pick the threshold from business costs, using out-of-fold predictions
  • Save the pipeline and threshold together with joblib, and reload to check

Challenges#

  1. Change the costs to $50 per call and $800 per lost customer. Where does the threshold move?
  2. Add the random forest to the grid search and compare its tuned AUC.
  3. Add a new feature inside the pipeline: average monthly spend = TotalCharges / tenure.

What's next#

Lesson 46: Capstone Project- Market Segmentation with Clustering RFM customer segments from real online retail transactions.

Data Mining 46: Capstone Project- Market Segmentation with Clustering#

In this lesson you'll be able to:

  • Clean a raw transaction log for customer analysis
  • Turn transactions into RFM features: recency, frequency, monetary
  • Tame skewed features and choose k with elbow and silhouette
  • Profile, name and act on customer segments
  • Package the segmenter in a pipeline for new customers Dataset: Online Retail, a UK online gift shop, Dec 2010 to Dec 2011 (data/pandas/online_retail.csv.gz)

Warm-up: from a churn label to no label at all#

Lesson 45 predicted churn end to end with a pipeline. Today there is no answer column.

import pandas as pd
df = pd.read_csv("online_retail.csv.gz")
print(df.shape)
(541909, 8)
df.head(3)
InvoiceNo StockCode Description Quantity InvoiceDate UnitPrice CustomerID Country
0 536365 85123A WHITE HANGING HEART T-LIGHT HOLDER 6 12/1/2010 8:26 2.55 17850.0 United Kingdom
1 536365 71053 WHITE METAL LANTERN 6 12/1/2010 8:26 3.39 17850.0 United Kingdom
2 536365 84406B CREAM CUPID HEARTS COAT HANGER 8 12/1/2010 8:26 2.75 17850.0 United Kingdom
print(df.columns.to_list())
['InvoiceNo', 'StockCode', 'Description', 'Quantity', 'InvoiceDate', 'UnitPrice', 'CustomerID', 'Country']

Cleaning the transaction log#

Problem: segments built on anonymous sales, refunds or dates stored as text are wrong.

df.isna().sum()
InvoiceNo           0
StockCode           0
Description      1454
Quantity            0
InvoiceDate         0
UnitPrice           0
CustomerID     135080
Country             0
dtype: int64

Quick check: what percentage of lines have no customer ID?

missing = df["CustomerID"].isna().mean() * 100
print(round(missing, 1))
24.9
cancel = df["InvoiceNo"].str.startswith("C")
print(cancel.sum(), (df["Quantity"] <= 0).sum())
df.loc[cancel, ["InvoiceNo", "Quantity", "UnitPrice"]].head(3)
9288 10624
InvoiceNo Quantity UnitPrice
141 C536379 -1 27.50
154 C536383 -1 4.65
235 C536391 -12 1.65
print((df["UnitPrice"] <= 0).sum())
2517
shop = df.dropna(subset=["CustomerID"])
keep = (shop["Quantity"] > 0) & (shop["UnitPrice"] > 0)
shop = shop[keep].copy()
print(len(shop), shop["CustomerID"].nunique())
397884 4338

Common mistake: doing date maths on dates stored as text

latest = shop["InvoiceDate"].max()
print(latest)
gap = latest - shop["InvoiceDate"]
9/9/2011 9:52
---------------------------------------------------------------------------
ArrowNotImplementedError                  Traceback (most recent call last)
File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\arrow\array.py:1097, in ArrowExtensionArray._evaluate_op_method(self, other, op, arrow_funcs)
   1096 try:
-> 1097     result = pc_func(self._pa_array, other)
   1098 except pa.ArrowNotImplementedError as err:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\arrow\array.py:181, in <lambda>(x, y)
    175         return result
    177     ARROW_ARITHMETIC_FUNCS = {
    178         "add": pc.add_checked,
    179         "radd": lambda x, y: pc.add_checked(y, x),
    180         "sub": pc.subtract_checked,
--> 181         "rsub": lambda x, y: pc.subtract_checked(y, x),
    182         "mul": pc.multiply_checked,
    183         "rmul": lambda x, y: pc.multiply_checked(y, x),
    184         "truediv": lambda x, y: pc.divide(*cast_for_truediv(x, y)),
    185         "rtruediv": lambda x, y: pc.divide(*cast_for_truediv(y, x)),
    186         "floordiv": lambda x, y: floordiv_compat(x, y),
    187         "rfloordiv": lambda x, y: floordiv_compat(y, x),
    188         "mod": NotImplemented,
    189         "rmod": NotImplemented,
    190         "divmod": NotImplemented,
    191         "rdivmod": NotImplemented,
    192         "pow": pc.power_checked,
    193         "rpow": lambda x, y: pc.power_checked(y, x),
    194     }
    196 if TYPE_CHECKING:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pyarrow\compute.py:255, in _make_generic_wrapper.<locals>.wrapper(memory_pool, *args)
    254     return Expression._call(func_name, list(args))
--> 255 return func.call(args, None, memory_pool)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pyarrow\_compute.pyx:405, in pyarrow._compute.Function.call()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pyarrow\error.pxi:155, in pyarrow.lib.pyarrow_internal_check_status()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pyarrow\error.pxi:92, in pyarrow.lib.check_status()

ArrowNotImplementedError: Function 'subtract_checked' has no kernel matching input types (large_string, large_string)

The above exception was the direct cause of the following exception:

TypeError                                 Traceback (most recent call last)
Cell In[10], line 3
      1 latest = shop["InvoiceDate"].max()
      2 print(latest)
----> 3 gap = latest - shop["InvoiceDate"]

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\ops\common.py:85, in _unpack_zerodim_and_defer.<locals>.new_method(self, other)
     82     other = sanitize_array(other, None)
     83     other = ensure_wrapped_if_datetimelike(other)
---> 85 return method(self, other)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arraylike.py:202, in OpsMixin.__rsub__(self, other)
    200 @unpack_zerodim_and_defer("__rsub__")
    201 def __rsub__(self, other):
--> 202     return self._arith_method(other, roperator.rsub)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\series.py:6751, in Series._arith_method(self, other, op)
   6749 def _arith_method(self, other, op):
   6750     self, other = self._align_for_op(other)
-> 6751     return base.IndexOpsMixin._arith_method(self, other, op)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\base.py:1644, in IndexOpsMixin._arith_method(self, other, op)
   1641     rvalues = np.arange(rvalues.start, rvalues.stop, rvalues.step)
   1643 with np.errstate(all="ignore"):
-> 1644     result = ops.arithmetic_op(lvalues, rvalues, op)
   1646 return self._construct_result(result, name=res_name, other=other)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\ops\array_ops.py:279, in arithmetic_op(left, right, op)
    270 right = ensure_wrapped_if_datetimelike(right)
    272 if (
    273     should_extension_dispatch(left, right)
    274     or isinstance(right, (Timedelta, BaseOffset, Timestamp))
   (...)
    277     # Timedelta/Timestamp and other custom scalars are included in the check
    278     # because numexpr will fail on it, see GH#31457
--> 279     res_values = op(left, right)
    280 else:
    281     # TODO we should handle EAs consistently and move this check before the if/else
    282     # (https://github.com/pandas-dev/pandas/issues/41165)
    283     # error: Argument 2 to "_bool_arith_check" has incompatible type
    284     # "Union[ExtensionArray, ndarray[Any, Any]]"; expected "ndarray[Any, Any]"
    285     _bool_arith_check(op, left, right)  # type: ignore[arg-type]

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\roperator.py:16, in rsub(left, right)
     15 def rsub(left, right):
---> 16     return right - left

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\ops\common.py:85, in _unpack_zerodim_and_defer.<locals>.new_method(self, other)
     82     other = sanitize_array(other, None)
     83     other = ensure_wrapped_if_datetimelike(other)
---> 85 return method(self, other)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arraylike.py:202, in OpsMixin.__rsub__(self, other)
    200 @unpack_zerodim_and_defer("__rsub__")
    201 def __rsub__(self, other):
--> 202     return self._arith_method(other, roperator.rsub)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\arrow\array.py:1158, in ArrowExtensionArray._arith_method(self, other, op)
   1154 if pa.types.is_string(self._pa_array.type) or pa.types.is_large_string(
   1155     self._pa_array.type
   1156 ):
   1157     try:
-> 1158         result = self._evaluate_op_method(other, op, ARROW_ARITHMETIC_FUNCS)
   1159     except (pa.ArrowInvalid, pa.ArrowTypeError):
   1160         result = self._str_arith_method_object_fallback(other, op)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\arrow\array.py:1099, in ArrowExtensionArray._evaluate_op_method(self, other, op, arrow_funcs)
   1097     result = pc_func(self._pa_array, other)
   1098 except pa.ArrowNotImplementedError as err:
-> 1099     raise TypeError(self._op_method_error_message(other_original, op)) from err
   1100 return self._from_pyarrow_array(result)

TypeError: operation 'rsub' not supported for dtype 'str' with object of type <class 'str'>
fmt = "%m/%d/%Y %H:%M"
shop["InvoiceDate"] = pd.to_datetime(shop["InvoiceDate"], format=fmt)
print(shop["InvoiceDate"].min(), shop["InvoiceDate"].max())
2010-12-01 08:26:00 2011-12-09 12:50:00
shop["CustomerID"] = shop["CustomerID"].astype(int)
shop["Revenue"] = shop["Quantity"] * shop["UnitPrice"]
shop[["InvoiceNo", "CustomerID", "Revenue"]].head(3)
InvoiceNo CustomerID Revenue
0 536365 17850 15.30
1 536365 17850 20.34
2 536365 17850 22.00

Turning transactions into RFM features#

Problem: clustering needs one row per customer, not one per sale line.

  • Recency: days since the last purchase (lower is better)
  • Frequency: number of separate orders
  • Monetary: total spend in pounds
snapshot = shop["InvoiceDate"].max() + pd.Timedelta(days=1)
print(snapshot)
2011-12-10 12:50:00
rfm = shop.groupby("CustomerID").agg(
    last_buy=("InvoiceDate", "max"),
    Frequency=("InvoiceNo", "nunique"),
    Monetary=("Revenue", "sum"))
rfm["Recency"] = (snapshot - rfm["last_buy"]).dt.days
features = ["Recency", "Frequency", "Monetary"]
rfm = rfm[features]
print(rfm.shape)
(4338, 3)
rfm.head().round(1)
Recency Frequency Monetary
CustomerID
12346 326 1 77183.6
12347 2 7 4310.0
12348 75 4 1797.2
12349 19 1 1757.6
12350 310 1 334.4

Quick check: a customer has 100 lines in the log. Is their frequency 100?

lines = shop.groupby("CustomerID").size()
top = lines.idxmax()
print(top, lines[top], rfm.loc[top, "Frequency"])
17841 7847 124
rfm.describe().round(1)
Recency Frequency Monetary
count 4338.0 4338.0 4338.0
mean 92.5 4.3 2054.3
std 100.0 7.7 8989.2
min 1.0 1.0 3.8
25% 18.0 1.0 307.4
50% 51.0 2.0 674.5
75% 142.0 5.0 1661.7
max 374.0 209.0 280206.0

Your turn: find the 5 biggest spenders and their share of all revenue.

top5 = rfm["Monetary"].nlargest(5)
print(top5.round(0))
print(round(top5.sum() / rfm["Monetary"].sum() * 100, 1))
CustomerID
14646    280206.0
18102    259657.0
17450    194551.0
16446    168472.0
14911    143825.0
Name: Monetary, dtype: float64
11.7

Taming skewed features before clustering#

Problem: a few huge spenders and pounds-versus-days scales would swamp the distances.

import numpy as np
import matplotlib.pyplot as plt
fig, axes = plt.subplots(1, 2, figsize=(10, 4))
rfm["Monetary"].plot.hist(bins=50, ax=axes[0])
axes[0].set_title("Monetary, raw")
np.log1p(rfm["Monetary"]).plot.hist(bins=50, ax=axes[1])
axes[1].set_title("Monetary, log1p")
axes[0].set_xlabel("Pounds")
axes[1].set_xlabel("log(1 + pounds)")
plt.tight_layout()
plt.show()
No description has been provided for this image
print(rfm.skew().round(2))
print(np.log1p(rfm).skew().round(2))
Recency       1.25
Frequency    12.07
Monetary     19.32
dtype: float64
Recency     -0.38
Frequency    1.21
Monetary     0.39
dtype: float64

Common mistake: feeding k-means a table that still has a text column

country = shop.groupby("CustomerID")["Country"].first()
rfm_c = rfm.join(country)
from sklearn.cluster import KMeans
KMeans(n_clusters=4, n_init=10, random_state=42).fit(rfm_c)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_26684\3726890348.py in ?()
      1 country = shop.groupby("CustomerID")["Country"].first()
      2 rfm_c = rfm.join(country)
      3 from sklearn.cluster import KMeans
----> 4 KMeans(n_clusters=4, n_init=10, random_state=42).fit(rfm_c)

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py in ?(estimator, *args, **kwargs)
   1399                     )
   1400                 ),
   1401                 callback_management_context(estimator),
   1402             ):
-> 1403                 return fit_method(estimator, *args, **kwargs)

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\cluster\_kmeans.py in ?(self, X, y, sample_weight)
   1455         -------
   1456         self : object
   1457             Fitted estimator.
   1458         """
-> 1459         X = validate_data(
   1460             self,
   1461             X,
   1462             accept_sparse="csr",

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py in ?(_estimator, X, y, reset, validate_separately, skip_check_array, **check_params)
   3037             out = y
   3038         else:
   3039             out = X, y
   3040     elif not no_val_X and no_val_y:
-> 3041         out = check_array(X, input_name="X", **check_params)
   3042     elif no_val_X and not no_val_y:
   3043         out = _check_y(y, **check_params)
   3044     else:

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py in ?(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, ensure_all_finite, ensure_non_negative, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)
   1035                         )
   1036                     array = xp.astype(array, dtype, copy=False)
   1037                 else:
   1038                     array = _asarray_with_order(array, order=order, dtype=dtype, xp=xp)
-> 1039             except ComplexWarning as complex_warning:
   1040                 raise ValueError(
   1041                     "Complex data not supported\n{}\n".format(array)
   1042                 ) from complex_warning

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_array_api.py in ?(array, dtype, order, copy, xp, device)
    979         # Use NumPy API to support order
    980         if copy is True:
    981             array = numpy.array(array, order=order, dtype=dtype)
    982         else:
--> 983             array = numpy.asarray(array, order=order, dtype=dtype)
    984 
    985         # At this point array is a NumPy ndarray. We convert it to an array
    986         # container that is consistent with the input's namespace.

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\generic.py in ?(self, dtype, copy)
   2023             )
   2024         values = self._values
   2025         if copy is None:
   2026             # Note: branch avoids `copy=None` for NumPy 1.x support
-> 2027             arr = np.asarray(values, dtype=dtype)
   2028         else:
   2029             arr = np.array(values, dtype=dtype, copy=copy)
   2030 

ValueError: could not convert string to float: 'United Kingdom'
from sklearn.preprocessing import StandardScaler
X_log = np.log1p(rfm_c[features])
scaler = StandardScaler()
X = scaler.fit_transform(X_log)
print(abs(X.mean(axis=0)).round(2), X.std(axis=0).round(2))
[0. 0. 0.] [1. 1. 1.]

Quick check: what happens if we cluster the raw numbers instead?

raw = KMeans(n_clusters=4, n_init=10, random_state=42)
raw.fit(rfm[features])
print(pd.Series(raw.labels_).value_counts().to_list())
[4300, 31, 5, 2]

Choosing the number of segments#

Problem: k-means needs k up front. Elbow and silhouette advise; the business decides.

K-means in plain words:

  • Drop k centres among the customers
  • Give each customer to the nearest centre
  • Move each centre to the middle of its customers, and repeat
from sklearn.metrics import silhouette_score
ks = range(2, 8)
inertia, silhouette = [], []
for k in ks:
    km = KMeans(n_clusters=k, n_init=10, random_state=42).fit(X)
    inertia.append(round(km.inertia_))
    silhouette.append(round(silhouette_score(X, km.labels_), 3))
print(inertia)
print(silhouette)
[6481, 4868, 3939, 3296, 2855, 2549]
[0.433, 0.337, 0.337, 0.316, 0.313, 0.31]
fig, axes = plt.subplots(1, 2, figsize=(10, 4))
axes[0].plot(ks, inertia, marker="o")
axes[0].set_title("Elbow: inertia by k")
axes[1].plot(ks, silhouette, marker="o", color="tab:orange")
axes[1].set_title("Silhouette by k")
for ax in axes:
    ax.set_xlabel("Number of segments, k")
plt.show()
No description has been provided for this image

k = 2 only splits good from bad customers. k = 3 and 4 tie on silhouette, so we take 4: more to act on.

km = KMeans(n_clusters=4, n_init=10, random_state=42).fit(X)
rfm["cluster"] = km.labels_
print(sorted(rfm["cluster"].value_counts().to_list()))
[716, 837, 1173, 1612]

Profiling and naming the segments#

Problem: cluster 0, 1, 2, 3 means nothing to a marketer, and the numbers are arbitrary.

centres = scaler.inverse_transform(km.cluster_centers_)
centres = pd.DataFrame(np.expm1(centres), columns=features)
centres.sort_values("Monetary").reset_index(drop=True).round(1)
Recency Frequency Monetary
0 150.9 1.3 267.3
1 13.8 2.0 440.1
2 54.6 3.8 1405.9
3 7.7 10.9 4257.1
med = rfm.groupby("cluster")["Monetary"].median()
rank = {c: i for i, c in enumerate(med.sort_values().index)}
rfm["segment"] = rfm["cluster"].map(rank)
profile = rfm.groupby("segment")[features].median()
profile["customers"] = rfm["segment"].value_counts()
profile.round(1)
Recency Frequency Monetary customers
segment
0 177.0 1.0 298.3 1612
1 17.0 2.0 471.7 837
2 56.0 4.0 1345.6 1173
3 8.0 10.0 3733.9 716
  • 0 Lost: last bought months ago, one small order
  • 1 Newcomers: recent, but only a couple of orders
  • 2 Regulars: steady buyers, cooling a little
  • 3 Champions: recent, frequent, high spend
names = {0: "Lost", 1: "Newcomers", 2: "Regulars", 3: "Champions"}
rfm["name"] = rfm["segment"].map(names)
rfm["name"].value_counts()
name
Lost         1612
Regulars     1173
Newcomers     837
Champions     716
Name: count, dtype: int64
import seaborn as sns
sns.scatterplot(data=rfm, x="Recency", y="Monetary", hue="name",
                s=10, alpha=0.6)
plt.yscale("log")
plt.title("Customer segments: recency vs spend")
plt.xlabel("Days since last purchase")
plt.ylabel("Total spend (pounds, log scale)")
plt.show()
No description has been provided for this image

Quick check: which segment brings in the most revenue, and how much?

total = rfm["Monetary"].sum()
rev = rfm.groupby("name")["Monetary"].sum() / total * 100
cust = rfm["name"].value_counts(normalize=True) * 100
share = pd.DataFrame({"customers %": cust, "revenue %": rev})
share.sort_values("revenue %").round(1)
customers % revenue %
name
Newcomers 19.3 5.2
Lost 37.2 6.2
Regulars 27.0 23.7
Champions 16.5 64.9
share.sort_values("revenue %").plot.barh(figsize=(8, 4))
plt.title("Share of customers vs share of revenue")
plt.xlabel("Percent")
plt.ylabel("Segment")
plt.tight_layout()
plt.show()
No description has been provided for this image

Your turn: what percentage of each segment lives outside the UK?

abroad = rfm_c["Country"] != "United Kingdom"
pct_abroad = abroad.groupby(rfm["name"]).mean() * 100
print(pct_abroad.sort_values().round(1))
name
Newcomers     7.9
Lost          8.4
Champions    11.3
Regulars     11.6
Name: Country, dtype: float64

Packaging the segmenter for new customers#

Problem: new customers arrive every day. Log, scale and k-means must run as one unit.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import FunctionTransformer
segmenter = make_pipeline(
    FunctionTransformer(np.log1p),
    StandardScaler(),
    KMeans(n_clusters=4, n_init=10, random_state=42))
print(list(segmenter.named_steps))
['functiontransformer', 'standardscaler', 'kmeans']
from sklearn.model_selection import train_test_split
old_ids, new_ids = train_test_split(rfm.index, test_size=0.2,
                                    random_state=42)
segmenter.fit(rfm.loc[old_ids, features])
new_labels = segmenter.predict(rfm.loc[new_ids, features])
print(len(old_ids), len(new_ids))
3470 868
from sklearn.metrics import adjusted_rand_score
full = rfm.loc[new_ids, "segment"]
print(round(adjusted_rand_score(full, new_labels), 3))
0.974

Your turn: refit k-means on X with random_state=7. Do the segments survive?

km7 = KMeans(n_clusters=4, n_init=10, random_state=7).fit(X)
print(round(adjusted_rand_score(rfm["segment"], km7.labels_), 3))
0.984

Mini-project: who should get the win-back campaign?#

Valuable customers (Regulars or Champions) who haven't bought for 90+ days. How many are there, and how much have they spent?

valuable = rfm["name"].isin(["Regulars", "Champions"])
quiet = rfm["Recency"] > 90
at_risk = rfm[valuable & quiet]
print(len(at_risk), round(at_risk["Monetary"].sum()))
287 630657
at_risk["name"].value_counts()
name
Regulars     286
Champions      1
Name: count, dtype: int64
at_risk.join(country)["Country"].value_counts().head(3)
Country
United Kingdom    248
Germany             9
Switzerland         4
Name: count, dtype: int64
cols = features + ["name"]
at_risk.sort_values("Monetary", ascending=False)[cols].head()
Recency Frequency Monetary name
CustomerID
12346 326 1 77183.60 Regulars
15749 235 3 44534.30 Regulars
15098 182 3 39916.50 Regulars
16180 100 8 10254.18 Regulars
12590 211 2 9864.26 Regulars

Quick check: the top name spent 77,184 pounds in one order, then vanished. Suspicious?

first = at_risk["Monetary"].idxmax()
cols_raw = ["InvoiceNo", "Quantity", "UnitPrice"]
df.loc[df["CustomerID"] == first, cols_raw]
InvoiceNo Quantity UnitPrice
61619 541431 74215 1.04
61624 C541433 -74215 1.04
at_risk = at_risk.drop(index=first)
print(len(at_risk), round(at_risk["Monetary"].sum()))
286 553474
import os
path = "m46_win_back.csv"
at_risk[cols].to_csv(path)
back = pd.read_csv(path, index_col="CustomerID")
print(back.shape)
(286, 4)

pct = at_risk["Monetary"].sum() / total * 100
print(round(pct, 1), round(len(at_risk) / len(rfm) * 100, 1))
6.2 6.6

Recap#

  • Clean first: drop anonymous lines, refunds and zero prices; parse dates; check odd results in the raw log
  • RFM turns a transaction log into one row per customer
  • Log and scale skewed features before k-means
  • Elbow and silhouette advise on k; the business decides
  • Rank clusters, name them, and package the steps in a pipeline

Challenges#

  1. Segment UK customers only. Do the same four segments appear?
  2. Try k = 5 and profile it. Which segment splits in two?
  3. Subtract each customer's refunds from Monetary. Who leaves the win-back list?

What's next#

Lesson 47: Capstone Project- Fraud Detection in Financial Transactions Cost-based thresholds, and anomaly detection vs supervised learning.

Data Mining 47: Capstone Project- Fraud Detection in Financial Transactions#

In this lesson you'll be able to:

  • Put a money value on missed frauds and on false alarms
  • Choose a decision threshold by cost, using a validation set
  • Detect fraud without labels using Isolation Forest
  • Compare anomaly detection with supervised learning honestly
  • Test the idea on a second problem: forged banknotes Datasets: European card transactions, Sept 2013 (data/mining/creditcard.csv.gz) and banknotes (data/mining/banknote.csv)

Warm-up: back to the card transactions#

Lesson 46 found groups with no labels. Fraud hunting often starts the same way.

import numpy as np
import pandas as pd
cc = pd.read_csv("creditcard.csv.gz")
print(cc.shape)
(284807, 31)
print(cc["Class"].value_counts().to_list())
print(round(cc["Class"].mean() * 100, 3))
[284315, 492]
0.173
cc[["Time", "V1", "V2", "V28", "Amount", "Class"]].head(3)
Time V1 V2 V28 Amount Class
0 0.0 -1.35981 -0.07278 -0.02105 149.62 0
1 0.0 1.19186 0.26615 0.01472 2.69 0
2 1.0 -1.35835 -1.34016 -0.05975 378.66 0

Putting a price on mistakes#

Problem: precision and recall treat all frauds alike. A missed 2,000 fraud is not a missed 1.

  • Missed fraud: the bank loses the transaction amount
  • False alarm: an analyst checks it; we assume 10 euros each
cc.groupby("Class")["Amount"].describe().round(1)
count mean std min 25% 50% 75% max
Class
0 284315.0 88.3 250.1 0.0 5.6 22.0 77.0 25691.2
1 492.0 122.2 256.7 0.0 1.0 9.2 105.9 2125.9

Quick check: how much money did all the frauds move in total?

fraud_total = cc.loc[cc["Class"] == 1, "Amount"].sum()
print(round(fraud_total, 2))
60127.97

Common mistake: combining two true/false columns with and

y_true = cc["Class"]
flag = np.zeros(len(cc))
missed = (y_true == 1) and (flag == 0)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_7548\3608078553.py in ?()
      1 y_true = cc["Class"]
      2 flag = np.zeros(len(cc))
----> 3 missed = (y_true == 1) and (flag == 0)

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\generic.py in ?(self)
   1499     @final
   1500     def __bool__(self) -> NoReturn:
-> 1501         raise ValueError(
   1502             f"The truth value of a {type(self).__name__} is ambiguous. "
   1503             "Use a.empty, a.bool(), a.item(), a.any() or a.all()."
   1504         )

ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
def cost(y_true, flag, amount, alarm_cost=10):
    missed = (y_true == 1) & (flag == 0)
    false_alarm = (y_true == 0) & (flag == 1)
    total = amount[missed].sum() + alarm_cost * false_alarm.sum()
    return round(float(total), 2)
amount = cc["Amount"]
print(cost(y_true, np.zeros(len(cc)), amount))
print(cost(y_true, np.ones(len(cc)), amount))
60127.97
2843150.0

Splitting honestly and training a supervised model#

Problem: if we choose a threshold on the test set, the test score is no longer honest.

  • Train 60%: the model learns
  • Validation 20%: we choose the threshold
  • Test 20%: we report the result, once
from sklearn.model_selection import train_test_split
X = cc.drop(columns=["Time", "Class"])
y = cc["Class"]
X_train, X_rest, y_train, y_rest = train_test_split(
    X, y, test_size=0.4, stratify=y, random_state=42)
X_val, X_test, y_val, y_test = train_test_split(
    X_rest, y_rest, test_size=0.5, stratify=y_rest, random_state=42)
print(len(X_train), len(X_val), len(X_test))
print(y_train.sum(), y_val.sum(), y_test.sum())
170884 56961 56962
295 98 99
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
model = make_pipeline(StandardScaler(),
                      LogisticRegression(max_iter=1000))
model.fit(X_train, y_train)
p_val = model.predict_proba(X_val)[:, 1]
print((p_val >= 0.5).sum())
76
print(round(np.median(p_val[y_val == 1]), 3))
print(round(np.median(p_val[y_val == 0]), 5))
0.912
0.00016

Common mistake: scoring new rows that have different columns

new_rows = cc.drop(columns="Class").head(3)
model.predict_proba(new_rows)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[13], line 2
      1 new_rows = cc.drop(columns="Class").head(3)
----> 2 model.predict_proba(new_rows)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\pipeline.py:925, in Pipeline.predict_proba(self, X, **params)
    923 if not _routing_enabled():
    924     for _, name, transform in self._iter(with_final=False):
--> 925         Xt = transform.transform(Xt)
    926     return self.steps[-1][1].predict_proba(Xt, **params)
    928 # metadata routing enabled

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_set_output.py:327, in _wrap_method_output.<locals>.wrapped(self, X, *args, **kwargs)
    325 @wraps(f)
    326 def wrapped(self, X, *args, **kwargs):
--> 327     data_to_wrap = f(self, X, *args, **kwargs)
    328     if isinstance(data_to_wrap, tuple):
    329         # only wrap the first output for cross decomposition
    330         return_tuple = (
    331             _wrap_data_with_container(method, data_to_wrap[0], X, self),
    332             *data_to_wrap[1:],
    333         )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\preprocessing\_data.py:1111, in StandardScaler.transform(self, X, copy)
   1108 check_is_fitted(self)
   1110 copy = copy if copy is not None else self.copy
-> 1111 X = validate_data(
   1112     self,
   1113     X,
   1114     reset=False,
   1115     accept_sparse="csr",
   1116     copy=copy,
   1117     dtype=supported_float_dtypes(xp, X_device),
   1118     force_writeable=True,
   1119     ensure_all_finite="allow-nan",
   1120 )
   1122 if sparse.issparse(X):
   1123     if self.with_mean:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:3016, in validate_data(_estimator, X, y, reset, validate_separately, skip_check_array, **check_params)
   2932 def validate_data(
   2933     _estimator,
   2934     /,
   (...)
   2940     **check_params,
   2941 ):
   2942     """Validate input data and set or check feature names and counts of the input.
   2943 
   2944     This helper function should be used in an estimator that requires input
   (...)
   3014         validated.
   3015     """
-> 3016     _check_feature_names(_estimator, X, reset=reset)
   3017     tags = get_tags(_estimator)
   3018     if y is None and tags.target_tags.required:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:2868, in _check_feature_names(estimator, X, reset)
   2865 if not missing_names and not unexpected_names:
   2866     message += "Feature names must be in the same order as they were in fit.\n"
-> 2868 raise ValueError(message)

ValueError: The feature names should match those that were passed during fit.
Feature names unseen at fit time:
- Time
new_rows = new_rows[X_train.columns]
print(model.predict_proba(new_rows)[:, 1].round(4))
[0.0006 0.0001 0.0006]

Quick check: at the default 0.5 cut-off, does the model beat flagging nothing?

amt_val = X_val["Amount"]
print(cost(y_val, np.zeros(len(y_val)), amt_val))
print(cost(y_val, (p_val >= 0.5).astype(int), amt_val))
12529.6
8124.08

Choosing the threshold by cost#

Problem: 0.5 assumes both mistakes cost the same. Here they clearly don't.

thresholds = np.linspace(0.01, 0.99, 99).round(2)
costs = [cost(y_val, (p_val >= t).astype(int), amt_val)
         for t in thresholds]
best = thresholds[int(np.argmin(costs))]
print(best, min(costs))
0.05 4937.11
import matplotlib.pyplot as plt
plt.plot(thresholds, costs)
plt.axvline(best, color="red", linestyle="--", label="cheapest")
plt.axvline(0.5, color="grey", linestyle=":", label="default 0.5")
plt.title("Validation cost by threshold")
plt.xlabel("Threshold")
plt.ylabel("Cost (euros)")
plt.legend()
plt.show()
No description has been provided for this image
p_test = model.predict_proba(X_test)[:, 1]
amt_test = X_test["Amount"]
flag_05 = (p_test >= 0.5).astype(int)
flag_best = (p_test >= best).astype(int)
print(cost(y_test, flag_05, amt_test))
print(cost(y_test, flag_best, amt_test))
3336.23
2862.65
from sklearn.metrics import confusion_matrix
print(confusion_matrix(y_test, flag_05))
print(confusion_matrix(y_test, flag_best))
[[56853    10]
 [   31    68]]
[[56830    33]
 [   17    82]]
lost = amt_test[(y_test == 1) & (flag_best == 0)].sum()
alarms = 10 * ((y_test == 0) & (flag_best == 1)).sum()
print(round(lost, 2), alarms)
2532.65 330

Your turn: a false alarm now means a blocked card and a phone call, 200 euros. Find the best threshold.

costs200 = [cost(y_val, (p_val >= t).astype(int), amt_val, 200)
            for t in thresholds]
print(thresholds[int(np.argmin(costs200))])
0.52

Catching fraud without labels: Isolation Forest#

Problem: fraud labels arrive weeks late, and new tricks have no labels at all.

  • Build random trees that split the data at random cut points
  • Odd transactions get isolated in very few cuts
  • Fewer cuts means a higher anomaly score
from sklearn.ensemble import IsolationForest
iso = IsolationForest(n_estimators=200, random_state=42)
iso.fit(X_train)
print(iso.max_samples_)
256

Quick check: Isolation Forest's predict gives two values. Are they 0 and 1?

values, counts = np.unique(iso.predict(X_val), return_counts=True)
print(values, counts)
[-1  1] [ 2299 54662]
s_val = -iso.score_samples(X_val)
s_test = -iso.score_samples(X_test)
print(round(s_val[y_val == 1].mean(), 3))
print(round(s_val[y_val == 0].mean(), 3))
0.579
0.404
plt.hist(s_val[y_val == 0], bins=50, density=True, alpha=0.6,
         label="normal")
plt.hist(s_val[y_val == 1], bins=50, density=True, alpha=0.6,
         label="fraud")
plt.title("Isolation Forest scores (validation)")
plt.xlabel("Anomaly score (higher = stranger)")
plt.ylabel("Density")
plt.legend()
plt.show()
No description has been provided for this image
shares = [0.001, 0.002, 0.005, 0.01, 0.02]
iso_costs = []
for share in shares:
    cut = np.quantile(s_val, 1 - share)
    iso_flag = (s_val >= cut).astype(int)
    iso_costs.append(cost(y_val, iso_flag, amt_val))
print(iso_costs)
[11918.67, 11458.24, 10932.48, 11543.62, 15747.56]
best_share = shares[int(np.argmin(iso_costs))]
cut = np.quantile(s_val, 1 - best_share)
iso_flag = (s_test >= cut).astype(int)
print(best_share, cost(y_test, iso_flag, amt_test))
0.005 7342.71

Head to head: anomaly detection vs supervised learning#

Problem: which approach should the fraud team run, and what does each one buy them?

from sklearn.metrics import average_precision_score
print(round(average_precision_score(y_test, p_test), 3))
print(round(average_precision_score(y_test, s_test), 3))
0.775
0.143
def summary(flag):
    caught = int(((y_test == 1) & (flag == 1)).sum())
    alarms = int(((y_test == 0) & (flag == 1)).sum())
    return {"caught": caught, "false alarms": alarms,
            "cost": cost(y_test, flag, amt_test)}
results = pd.DataFrame.from_dict({
    "flag nothing": summary(np.zeros(len(y_test))),
    "logistic 0.5": summary(flag_05),
    "logistic tuned": summary(flag_best),
    "isolation forest": summary(iso_flag)}, orient="index")
results
caught false alarms cost
flag nothing 0 0 9874.35
logistic 0.5 68 10 3336.23
logistic tuned 82 33 2862.65
isolation forest 34 226 7342.71
results["cost"].plot.barh(color="tab:red")
plt.title("Test-set cost by approach")
plt.xlabel("Cost (euros)")
plt.ylabel("Approach")
plt.tight_layout()
plt.show()
No description has been provided for this image

Quick check: does Isolation Forest catch any frauds that the tuned model misses?

only_iso = (y_test == 1) & (iso_flag == 1) & (flag_best == 0)
print(only_iso.sum(), round(amt_test[only_iso].sum(), 2))
0 0.0

So when is anomaly detection still worth running?

  • Before labels exist, for example on a new product or market
  • To spot new fraud patterns the supervised model never saw
  • To decide which transactions are worth labelling first

A second case: forged banknotes#

Problem: anomaly detection assumes the odd cases are rare. What if they aren't?

bn = pd.read_csv("banknote.csv")
print(bn.shape)
print(bn["forged"].value_counts().to_list())
(1372, 5)
[762, 610]
Xb = bn.drop(columns="forged")
yb = bn["forged"]
Xb_train, Xb_test, yb_train, yb_test = train_test_split(
    Xb, yb, test_size=0.3, stratify=yb, random_state=42)
print(len(Xb_train), len(Xb_test))
960 412
import seaborn as sns
sns.scatterplot(data=bn, x="variance", y="skewness", hue="forged",
                s=12, alpha=0.6)
plt.title("Banknotes: genuine (0) vs forged (1)")
plt.xlabel("Variance of wavelet image")
plt.ylabel("Skewness of wavelet image")
plt.show()
No description has been provided for this image
from sklearn.metrics import roc_auc_score
lr_b = make_pipeline(StandardScaler(), LogisticRegression())
lr_b.fit(Xb_train, yb_train)
p_b = lr_b.predict_proba(Xb_test)[:, 1]
print(round(roc_auc_score(yb_test, p_b), 4))
0.9999

Quick check: 44% of these notes are forged. Can Isolation Forest spot them as anomalies?

iso_all = IsolationForest(n_estimators=200, random_state=42)
iso_all.fit(Xb_train)
s_all = -iso_all.score_samples(Xb_test)
print(round(roc_auc_score(yb_test, s_all), 3))
0.557
genuine = Xb_train[yb_train == 0]
iso_gen = IsolationForest(n_estimators=200, random_state=42)
iso_gen.fit(genuine)
s_gen = -iso_gen.score_samples(Xb_test)
print(round(roc_auc_score(yb_test, s_gen), 3))
0.939

Your turn: try Local Outlier Factor (lesson 38) with novelty=True, scaled, on genuine notes only.

from sklearn.neighbors import LocalOutlierFactor
lof = make_pipeline(StandardScaler(),
                    LocalOutlierFactor(novelty=True))
lof.fit(genuine)
s_lof = -lof.score_samples(Xb_test)
print(round(roc_auc_score(yb_test, s_lof), 3))
0.997

Mini-project: the business case for the fraud team#

Using the test set: how much does the tuned model save, how many alerts does it raise, and what slips through?

nothing = results.loc["flag nothing", "cost"]
tuned = results.loc["logistic tuned", "cost"]
print(round(nothing - tuned, 2))
print(round((nothing - tuned) / nothing * 100, 1))
7011.7
71.0
alerts = flag_best.sum()
print(alerts, round(alerts / len(y_test) * 10000, 1))
115 20.2
print(round((nothing - tuned) / alerts, 2))
60.97
missed = (y_test == 1) & (flag_best == 0)
print(missed.sum())
amt_test[missed].sort_values(ascending=False).head(3)
17
191074    720.80
275992    634.30
100623    549.06
Name: Amount, dtype: float64
import os
path = "m47_results.csv"
results.to_csv(path, index_label="method")
back = pd.read_csv(path, index_col="method")
back["cost"]
method
flag nothing        9874.35
logistic 0.5        3336.23
logistic tuned      2862.65
isolation forest    7342.71
Name: cost, dtype: float64

Recap#

  • Price both mistakes: missed fraud loses the amount, false alarms cost reviews
  • Train, validate, test: choose the threshold on validation, report on test once
  • The cheapest threshold is rarely 0.5, and moves with the price of an alarm
  • Isolation Forest needs no labels but trails a supervised model
  • Anomaly detection works when anomalies are rare, or when you fit on normal data

Challenges#

  1. Add class_weight="balanced" to the logistic model. Does the tuned test cost improve?
  2. Add the Isolation Forest score as an extra feature for the logistic model.
  3. Make the false-alarm cost depend on the amount (say 1% of it) and redo the threshold search.

What's next#

Lesson 48: Capstone Project- Sentiment Analysis using Text Mining Turn real text into numbers and classify it as positive or negative.

Data Mining 48: Capstone Project- Sentiment Analysis using Text Mining#

In this lesson you'll be able to:

  • Explore a collection of real text reviews
  • Score sentiment with a ready-made dictionary (VADER)
  • Turn text into numbers with bag of words and TF-IDF
  • Train, tune and test a sentiment classifier
  • Read which words drive the model, and pick the right tool Dataset: 2,000 movie reviews (data/mining/movie_reviews.csv) and the VADER lexicon

Warm-up: is this data balanced?#

Lesson 47 showed that rare classes make accuracy misleading. Check the balance first.

import pandas as pd
df = pd.read_csv("movie_reviews.csv")
print(df.shape)
df["sentiment"].value_counts()
(2000, 3)
sentiment
pos    1000
neg    1000
Name: count, dtype: int64

Meet the reviews#

Problem: before mining text, learn how long and how clean it is.

first = df.loc[0, "text"]
print(len(first))
print(first[:250])
4225
films adapted from comic books have had plenty of success , whether they're about superheroes ( batman , superman , spawn ) , or geared toward kids ( casper ) or the arthouse crowd ( ghost world ) , but there's never really been a comic book like fro
df["words"] = df["text"].str.split().str.len()
df.groupby("sentiment")["words"].describe().round(1)
count mean std min 25% 50% 75% max
sentiment
neg 1000.0 705.6 296.7 17.0 510.0 668.5 848.2 2181.0
pos 1000.0 787.1 352.8 130.0 543.5 731.5 957.2 2678.0

Quick check: in this data, do critics write more when they like a film, or less?

import matplotlib.pyplot as plt
for label in ["neg", "pos"]:
    part = df.loc[df["sentiment"] == label, "words"]
    plt.hist(part, bins=40, alpha=0.5, label=label)
plt.title("Review length by sentiment")
plt.xlabel("Words in review")
plt.ylabel("Number of reviews")
plt.legend()
plt.show()
No description has been provided for this image
has_capitals = df["text"].str.contains("[A-Z]")
print(has_capitals.sum())
0

A ready-made dictionary: VADER#

Problem: score sentiment today, with no labelled data and no training.

  • VADER: about 7,500 words, each rated by people from -4 to +4
  • Plus rules for 'not', 'very', capitals and exclamation marks
  • It adds everything up into one compound score from -1 to +1
lex = pd.read_csv("vader_lexicon.txt", sep="\t",
                  header=None, usecols=[0, 1],
                  names=["token", "score"])
print(lex.shape)
(7517, 2)
picks = ["great", "fun", "boring", "waste", "horrendous"]
lex[lex["token"].isin(picks)]
token score
1167 boring -1.3
3200 fun 2.3
3418 great 3.1
3736 horrendous -2.8
7246 waste -1.8

Your turn: how many lexicon words are positive, and how many are negative?

print((lex["score"] > 0).sum())
print((lex["score"] < 0).sum())
3344
4173
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
sia = SentimentIntensityAnalyzer()
short = df.loc[df["words"].idxmin(), "text"]
print(short)
print(sia.polarity_scores(short))
this film is extraordinarily horrendous and i'm not going to waste any more words on it .
{'neg': 0.18, 'neu': 0.71, 'pos': 0.11, 'compound': -0.3544}

Quick check: VADER rates 'good' as positive. What happens to 'not good'?

for phrase in ["good", "not good"]:
    print(phrase, sia.polarity_scores(phrase)["compound"])
good 0.4404
not good -0.3412

Common mistake: comparing the whole score dictionary with 0

scores = sia.polarity_scores(short)
scores > 0
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[12], line 2
      1 scores = sia.polarity_scores(short)
----> 2 scores > 0

TypeError: '>' not supported between instances of 'dict' and 'int'
print(scores["compound"])
print(scores["compound"] > 0)
-0.3544
False

A fair test: hold out unseen reviews#

Problem: the dictionary and a trained model must face the same unseen reviews.

from sklearn.model_selection import train_test_split
y = (df["sentiment"] == "pos").astype(int)
X_train, X_test, y_train, y_test = train_test_split(
    df["text"], y, test_size=0.2, random_state=42, stratify=y)
print(len(X_train), len(X_test), y_test.mean())
1600 400 0.5

Scoring the test reviews with VADER#

Problem: does a general word list understand film critics?

def compound(text):
    return sia.polarity_scores(text)["compound"]

vader = X_test.apply(compound)
vader.describe().round(3)
count    400.000
mean       0.364
std        0.854
min       -0.999
25%       -0.777
50%        0.970
75%        0.995
max        1.000
Name: text, dtype: float64
from sklearn.metrics import accuracy_score
vader_pred = (vader >= 0).astype(int)
print(accuracy_score(y_test, vader_pred))
0.645
pd.crosstab(y_test, vader_pred,
            rownames=["actual"], colnames=["vader"])
vader 0 1
actual
0 93 107
1 35 165
vader.groupby(y_test).plot.hist(bins=30, alpha=0.5, legend=True)
plt.title("VADER compound score by real sentiment")
plt.xlabel("Compound score (0 = negative, 1 = positive label)")
plt.ylabel("Number of reviews")
plt.show()
No description has been provided for this image

Why? Long reviews retell the plot, and even a pan praises the actors or the music. Every positive word pushes the total up.

Let's test that idea: score one misjudged review sentence by sentence.

missed = X_test[(y_test == 0) & (vader_pred == 1)]
example = missed.iloc[0]
sentences = pd.Series(example.split(" . "))
sent_scores = sentences.apply(compound)
print(len(sentences), (sent_scores > 0).sum(), (sent_scores < 0).sum())
print(round(vader[missed.index[0]], 3))
28 21 6
0.996

Your turn: what share of the negative test reviews did VADER call positive?

print(vader_pred[y_test == 0].mean().round(3))
0.535

Turning text into numbers#

Problem: a model can't read sentences, so each review becomes a row of word counts.

  • Bag of words: one column per word, counting how often it appears
  • Word order is lost; stop words like 'the' are dropped
from sklearn.feature_extraction.text import CountVectorizer
counter = CountVectorizer(stop_words="english", min_df=3)
counts = counter.fit_transform(X_train)
print(counts.shape)
(1600, 16216)
totals = pd.Series(counts.sum(axis=0).A1,
                   index=counter.get_feature_names_out())
totals.nlargest(8)
film         7543
movie        4629
like         2944
just         2295
time         1968
good         1936
story        1775
character    1594
dtype: int64

Quick check: 'not' flipped VADER's score. Is 'not' still in our vocabulary?

print("not" in counter.vocabulary_)
print(len(counter.vocabulary_))
False
16216
  • TF-IDF: like counts, but words found in almost every review weigh less
  • Rare, specific words like 'boring' stand out more
from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer(stop_words="english", min_df=3)
tfidf.fit(X_train)
idf = pd.Series(tfidf.idf_, index=tfidf.get_feature_names_out())
print(idf[["film", "movie", "great", "boring"]].round(2))
film      1.13
movie     1.26
great     2.08
boring    3.25
dtype: float64

Training and tuning a sentiment classifier#

Problem: learn from 1,600 labelled reviews which words signal praise or a pan.

  • Pipeline: vectoriser + model, so each cross-validation fold builds its own vocabulary
  • Baseline: Naive Bayes (lesson 16). Challenger: logistic regression (lesson 17)
from sklearn.pipeline import make_pipeline
from sklearn.naive_bayes import MultinomialNB
from sklearn.model_selection import cross_val_score
nb_model = make_pipeline(
    TfidfVectorizer(stop_words="english", min_df=3), MultinomialNB())
nb_cv = cross_val_score(nb_model, X_train, y_train, cv=5)
print(nb_cv.mean().round(3))
0.815

Quick check: dropping stop words also dropped 'not'. Does keeping them help?

keep_all = make_pipeline(TfidfVectorizer(min_df=3), MultinomialNB())
keep_cv = cross_val_score(keep_all, X_train, y_train, cv=5)
print(keep_cv.mean().round(3))
0.818
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV
lr_model = make_pipeline(
    TfidfVectorizer(stop_words="english", min_df=3),
    LogisticRegression(max_iter=1000))
grid = GridSearchCV(lr_model, {"logisticregression__C": [1, 10, 100]},
                    cv=5)
grid.fit(X_train, y_train)
print(grid.best_params_, round(grid.best_score_, 3))
{'logisticregression__C': 100} 0.838

Only now, once, do we touch the test set.

model = grid.best_estimator_
nb_model.fit(X_train, y_train)
print("Naive Bayes:", nb_model.score(X_test, y_test))
print("Logistic:", model.score(X_test, y_test))
Naive Bayes: 0.8225
Logistic: 0.835
from sklearn.metrics import classification_report
model_pred = model.predict(X_test)
print(classification_report(y_test, model_pred,
                            target_names=["neg", "pos"]))
              precision    recall  f1-score   support

         neg       0.86      0.80      0.83       200
         pos       0.81      0.88      0.84       200

    accuracy                           0.83       400
   macro avg       0.84      0.83      0.83       400
weighted avg       0.84      0.83      0.83       400

Common mistake: building a fresh vectoriser on the test reviews

lr = model[-1]
wrong = TfidfVectorizer(stop_words="english", min_df=3)
X_wrong = wrong.fit_transform(X_test)
lr.predict(X_wrong)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[30], line 4
      2 wrong = TfidfVectorizer(stop_words="english", min_df=3)
      3 X_wrong = wrong.fit_transform(X_test)
----> 4 lr.predict(X_wrong)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\linear_model\_base.py:414, in LinearClassifierMixin.predict(self, X)
    412 check_same_namespace(X, self, attribute="coef_", method="predict")
    413 xp, _, device_ = get_namespace_and_device(X)
--> 414 scores = self.decision_function(X)
    415 if len(scores.shape) == 1:
    416     indices = xp.astype(scores > 0, indexing_dtype(xp))

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\linear_model\_base.py:389, in LinearClassifierMixin.decision_function(self, X)
    386 xp, _ = get_namespace(X)
    387 check_same_namespace(X, self, attribute="coef_", method="decision_function")
--> 389 X = validate_data(self, X, accept_sparse="csr", reset=False)
    390 coef_T = self.coef_.T if self.coef_.ndim == 2 else self.coef_
    391 scores = safe_sparse_dot(X, coef_T, dense_output=True) + self.intercept_

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:3062, in validate_data(_estimator, X, y, reset, validate_separately, skip_check_array, **check_params)
   3059     out = X, y
   3061 if not no_val_X and check_params.get("ensure_2d", True):
-> 3062     _check_n_features(_estimator, X, reset=reset)
   3064 return out

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:2926, in _check_n_features(estimator, X, reset)
   2923     return
   2925 if n_features != estimator.n_features_in_:
-> 2926     raise ValueError(
   2927         f"X has {n_features} features, but {estimator.__class__.__name__} "
   2928         f"is expecting {estimator.n_features_in_} features as input."
   2929     )

ValueError: X has 6827 features, but LogisticRegression is expecting 16216 features as input.
vocab = model[0]
X_right = vocab.transform(X_test)
print(X_right.shape)
print((lr.predict(X_right) == model_pred).all())
(400, 16216)
True

What did the model learn?#

Problem: a business wants to know why, not just how accurate.

weights = pd.Series(lr.coef_[0],
                    index=vocab.get_feature_names_out())
weights = weights.sort_values()
print(weights.head(6).round(2))
print(weights.tail(6).round(2))
bad             -14.10
plot             -8.75
worst            -8.22
supposed         -7.46
boring           -6.76
unfortunately    -6.20
dtype: float64
perfect      5.59
overall      5.60
matrix       5.65
excellent    5.70
fun          6.46
great        8.23
dtype: float64
top = pd.concat([weights.head(8), weights.tail(8)])
colours = ["tab:red" if w < 0 else "tab:blue" for w in top]
top.plot.barh(color=colours)
plt.title("Words that move the sentiment model")
plt.xlabel("Logistic regression weight")
plt.ylabel("Word")
plt.show()
No description has been provided for this image

Notice 'plot' and 'script': critics mention them mostly when complaining. And 'matrix' is a film title: the model also learns which films critics liked.

Your turn: how sure is the model about the shortest review?

print(model.predict_proba([short]).round(3))
[[0.976 0.024]]

Mini-project: dictionary or trained model?#

A studio wants to track reviews automatically. Which tool should it use, and where does each one fail?

results = pd.DataFrame({
    "actual": y_test, "vader": vader_pred, "model": model_pred,
    "words": X_test.str.split().str.len()})
results["vader_ok"] = results["vader"] == results["actual"]
results["model_ok"] = results["model"] == results["actual"]
results[["vader_ok", "model_ok"]].mean().round(3)
vader_ok    0.645
model_ok    0.835
dtype: float64
labels = ["short", "medium", "long"]
results["length"] = pd.qcut(results["words"], 3, labels=labels)
by_len = results.groupby("length", observed=True)
acc = by_len[["vader_ok", "model_ok"]].mean().round(3)
acc
vader_ok model_ok
length
short 0.642 0.806
medium 0.647 0.872
long 0.647 0.827
acc.plot.bar(rot=0)
plt.title("Accuracy by review length")
plt.xlabel("Review length (thirds of the test set)")
plt.ylabel("Accuracy")
plt.show()
No description has been provided for this image
agree = results["vader"] == results["model"]
print(agree.mean().round(3))
print(results.loc[agree, "model_ok"].mean().round(3))
print(results.loc[~agree, "model_ok"].mean().round(3))
0.645
0.872
0.768

Finding: the trained model wins at every length. VADER is a quick start when no labels exist. When both tools agree, the verdict is right more often: a cheap confidence check.

Recap#

  • Check text first: length, balance and how it was cleaned
  • VADER scores sentiment with no training, but leans positive on film reviews
  • Bag of words and TF-IDF turn each review into a row of numbers
  • Fit the vectoriser on training text only; a pipeline does it for you
  • Read the weights: they explain the model in plain words

Challenges#

  1. Add ngram_range=(1, 2) to the TF-IDF step. Does 'not good' help accuracy?
  2. Find VADER's best cut-off on 200 training reviews, then test it once.
  3. Train the same pipeline on data/mining/sms_spam.csv. Which words mark spam?

What's next#

Lesson 49: Capstone Project- Comprehensive Data Mining Case Study The whole KDD process, start to finish, on one new dataset.

Data Mining 49: Capstone Project- Comprehensive Data Mining Case Study#

In this lesson you'll be able to:

  • Run every step of the KDD process on one dataset
  • Check, clean and scale data before mining it
  • Find natural groups with k-means and test them against the truth
  • Compare classifiers fairly and test the winner once
  • Turn the results into a clear, practical finding Dataset: chemical analysis of 178 Italian wines (data/mining/wine.csv)

Warm-up: numbers first#

Lesson 48 had to turn text into numbers. Here the numbers come ready-made.

import pandas as pd
df = pd.read_csv("wine.csv")
print(df.shape)
df.iloc[:3, [0, 6, 9, 12, 13]]
(178, 14)
alcohol flavanoids color_intensity proline cultivar
0 14.23 3.06 5.64 1065.0 class_0
1 13.20 2.76 4.38 1050.0 class_0
2 13.16 3.24 5.68 1185.0 class_0

The KDD process: our map for today#

  1. Selection: pick the data and the question
  2. Preprocessing: check and clean
  3. Transformation: scale, reduce, reshape
  4. Data mining: clustering and classification
  5. Interpretation: turn patterns into a finding

Step 1: Selection: what question can this data answer?#

Problem: a lab wants to confirm a wine's grape cultivar from its chemistry alone.

df["cultivar"].value_counts()
cultivar
class_1    71
class_0    59
class_2    48
Name: count, dtype: int64
print(df.columns[:-1].to_list())
['alcohol', 'malic_acid', 'ash', 'alcalinity_of_ash', 'magnesium', 'total_phenols', 'flavanoids', 'nonflavanoid_phenols', 'proanthocyanins', 'color_intensity', 'hue', 'od280/od315_of_diluted_wines', 'proline']

The cultivars are coded class_0, class_1 and class_2 in the original data.

Step 2: Preprocessing: check before you trust#

Problem: hidden gaps, duplicates or odd values would spoil every later step.

Quick check: how many missing values and duplicate rows do you expect in lab data?

print(df.isna().sum().sum())
print(df.duplicated().sum())
0
0
summary = df.describe().T[["mean", "std", "min", "max"]]
summary.round(2)
mean std min max
alcohol 13.00 0.81 11.03 14.83
malic_acid 2.34 1.12 0.74 5.80
ash 2.37 0.27 1.36 3.23
alcalinity_of_ash 19.49 3.34 10.60 30.00
magnesium 99.74 14.28 70.00 162.00
total_phenols 2.30 0.63 0.98 3.88
flavanoids 2.03 1.00 0.34 5.08
nonflavanoid_phenols 0.36 0.12 0.13 0.66
proanthocyanins 1.59 0.57 0.41 3.58
color_intensity 5.06 2.32 1.28 13.00
hue 0.96 0.23 0.48 1.71
od280/od315_of_diluted_wines 2.61 0.71 1.27 4.00
proline 746.89 314.91 278.00 1680.00
picks = ["alcohol", "flavanoids", "color_intensity", "proline"]
df.groupby("cultivar")[picks].mean().round(2)
alcohol flavanoids color_intensity proline
cultivar
class_0 13.74 2.98 5.53 1115.71
class_1 12.28 2.08 3.09 519.51
class_2 13.15 0.78 7.40 629.90
import matplotlib.pyplot as plt
df.boxplot(column="flavanoids", by="cultivar")
plt.title("Flavanoids by cultivar")
plt.suptitle("")
plt.xlabel("Cultivar")
plt.ylabel("Flavanoids")
plt.show()
No description has been provided for this image

Quick check: are there extreme values? Count readings more than 3 standard deviations from the mean.

chem = df.drop(columns="cultivar")
z = (chem - chem.mean()) / chem.std()
extreme = (z.abs() > 3).sum()
print(extreme[extreme > 0])
print(extreme.sum(), z.size)
malic_acid           1
ash                  3
alcalinity_of_ash    1
magnesium            2
flavanoids           1
proanthocyanins      1
color_intensity      1
hue                  1
dtype: int64
11 2314

A handful of extremes in real lab readings: we keep them, and scaling handles the rest.

Common mistake: scaling the whole table, label column included

from sklearn.preprocessing import StandardScaler
StandardScaler().fit_transform(df)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_21704\1731558933.py in ?()
      1 from sklearn.preprocessing import StandardScaler
----> 2 StandardScaler().fit_transform(df)

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_set_output.py in ?(self, X, *args, **kwargs)
    325     @wraps(f)
    326     def wrapped(self, X, *args, **kwargs):
--> 327         data_to_wrap = f(self, X, *args, **kwargs)
    328         if isinstance(data_to_wrap, tuple):
    329             # only wrap the first output for cross decomposition
    330             return_tuple = (

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py in ?(self, X, y, **fit_params)
    965                 )
    966 
    967         if y is None:
    968             # fit method of arity 1 (unsupervised transformation)
--> 969             return self.fit(X, **fit_params).transform(X)
    970         else:
    971             # fit method of arity 2 (supervised transformation)
    972             return self.fit(X, y, **fit_params).transform(X)

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\preprocessing\_data.py in ?(self, X, y, sample_weight)
    924             Fitted scaler.
    925         """
    926         # Reset internal state before fitting
    927         self._reset()
--> 928         return self.partial_fit(X, y, sample_weight)

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py in ?(estimator, *args, **kwargs)
   1399                     )
   1400                 ),
   1401                 callback_management_context(estimator),
   1402             ):
-> 1403                 return fit_method(estimator, *args, **kwargs)

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\preprocessing\_data.py in ?(self, X, y, sample_weight)
    961             Fitted scaler.
    962         """
    963         xp, _, X_device = get_namespace_and_device(X)
    964         first_call = not hasattr(self, "n_samples_seen_")
--> 965         X = validate_data(
    966             self,
    967             X,
    968             accept_sparse=("csr", "csc"),

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py in ?(_estimator, X, y, reset, validate_separately, skip_check_array, **check_params)
   3037             out = y
   3038         else:
   3039             out = X, y
   3040     elif not no_val_X and no_val_y:
-> 3041         out = check_array(X, input_name="X", **check_params)
   3042     elif no_val_X and not no_val_y:
   3043         out = _check_y(y, **check_params)
   3044     else:

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py in ?(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, ensure_all_finite, ensure_non_negative, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)
   1035                         )
   1036                     array = xp.astype(array, dtype, copy=False)
   1037                 else:
   1038                     array = _asarray_with_order(array, order=order, dtype=dtype, xp=xp)
-> 1039             except ComplexWarning as complex_warning:
   1040                 raise ValueError(
   1041                     "Complex data not supported\n{}\n".format(array)
   1042                 ) from complex_warning

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_array_api.py in ?(array, dtype, order, copy, xp, device)
    979         # Use NumPy API to support order
    980         if copy is True:
    981             array = numpy.array(array, order=order, dtype=dtype)
    982         else:
--> 983             array = numpy.asarray(array, order=order, dtype=dtype)
    984 
    985         # At this point array is a NumPy ndarray. We convert it to an array
    986         # container that is consistent with the input's namespace.

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\generic.py in ?(self, dtype, copy)
   2023             )
   2024         values = self._values
   2025         if copy is None:
   2026             # Note: branch avoids `copy=None` for NumPy 1.x support
-> 2027             arr = np.asarray(values, dtype=dtype)
   2028         else:
   2029             arr = np.array(values, dtype=dtype, copy=copy)
   2030 

ValueError: could not convert string to float: 'class_0'
X = df.drop(columns="cultivar")
y = df["cultivar"]
print(X.shape, y.shape)
(178, 13) (178,)

Step 3: Transformation: put every chemical on the same scale#

Problem: proline runs into the hundreds, hue stays near 1, so distances are lopsided.

X_scaled = StandardScaler().fit_transform(X)
scaled = pd.DataFrame(X_scaled, columns=X.columns)
scaled[["hue", "proline"]].describe().round(2)
hue proline
count 178.00 178.00
mean 0.00 -0.00
std 1.00 1.00
min -2.09 -1.49
25% -0.77 -0.78
50% 0.03 -0.23
75% 0.71 0.76
max 3.30 2.97
  • PCA squeezes 13 measurements into 2 new axes that keep the most spread
  • A picture only: the models still use all 13 columns
from sklearn.decomposition import PCA
pca = PCA(n_components=2)
coords = pca.fit_transform(X_scaled)
print(pca.explained_variance_ratio_.round(3))
[0.362 0.192]
for name in sorted(y.unique()):
    mask = (y == name).to_numpy()
    plt.scatter(coords[mask, 0], coords[mask, 1], label=name,
                alpha=0.7)
plt.title("178 wines on two PCA axes")
plt.xlabel("PC1")
plt.ylabel("PC2")
plt.legend()
plt.show()
No description has been provided for this image
loadings = pd.Series(pca.components_[0], index=X.columns)
loadings.sort_values().round(2).iloc[[0, 1, -2, -1]]
nonflavanoid_phenols   -0.30
malic_acid             -0.25
total_phenols           0.39
flavanoids              0.42
dtype: float64

Step 4a: Mining without labels: do the wines group themselves?#

Problem: if nobody had labelled the wines, would the chemistry still reveal three groups?

from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
for k in range(2, 7):
    km = KMeans(n_clusters=k, n_init=10, random_state=42)
    labels = km.fit_predict(X_scaled)
    print(k, round(silhouette_score(X_scaled, labels), 3))
2 0.259
3 0.285
4 0.26
5 0.202
6 0.237
ks = range(2, 7)
sil = [silhouette_score(X_scaled, KMeans(n_clusters=k, n_init=10,
         random_state=42).fit_predict(X_scaled)) for k in ks]
plt.plot(list(ks), sil, marker="o")
plt.title("Silhouette score by number of clusters")
plt.xlabel("Number of clusters (k)")
plt.ylabel("Silhouette score")
plt.show()
No description has been provided for this image

Your turn: with k = 2, where do the three cultivars end up?

km2 = KMeans(n_clusters=2, n_init=10, random_state=42)
two = km2.fit_predict(X_scaled)
pd.crosstab(two, df["cultivar"], rownames=["cluster"])
cultivar class_0 class_1 class_2
cluster
0 59 28 0
1 0 43 48
km = KMeans(n_clusters=3, n_init=10, random_state=42)
df["cluster"] = km.fit_predict(X_scaled)
order = df.groupby("cluster")["proline"].mean().rank()
df["cluster"] = df["cluster"].map(order.astype(int) - 1)
df.groupby("cluster")[picks].mean().round(2)
alcohol flavanoids color_intensity proline
cluster
0 12.25 2.05 2.97 510.17
1 13.13 0.82 7.23 619.06
2 13.68 3.00 5.45 1100.23

Now, and only now, compare the clusters with the real cultivars.

pd.crosstab(df["cluster"], df["cultivar"])
cultivar class_0 class_1 class_2
cluster
0 0 65 0
1 0 3 48
2 59 3 0
from sklearn.metrics import adjusted_rand_score
print(round(adjusted_rand_score(y, df["cluster"]), 3))
0.897

Quick check: would k-means find the cultivars as well without scaling?

raw = KMeans(n_clusters=3, n_init=10, random_state=42).fit(X)
print(round(adjusted_rand_score(y, raw.labels_), 3))
0.371

Step 4b: Mining with labels: predict the cultivar#

Problem: the lab wants a model that names the cultivar of a new wine, with an honest score.

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)
print(len(X_train), len(X_test))
133 45
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
models = {
    "logistic": make_pipeline(StandardScaler(),
                              LogisticRegression(max_iter=1000)),
    "knn": make_pipeline(StandardScaler(), KNeighborsClassifier()),
    "tree": DecisionTreeClassifier(random_state=42),
    "forest": RandomForestClassifier(random_state=42),
}
from sklearn.model_selection import cross_val_score
for name, model in models.items():
    s = cross_val_score(model, X_train, y_train, cv=5)
    print(f"{name:9} {s.mean():.3f} +/- {s.std():.3f}")
logistic  0.985 +/- 0.018
knn       0.955 +/- 0.036
tree      0.895 +/- 0.076
forest    0.970 +/- 0.015

Your turn: score k-nearest neighbours again, without the scaler.

knn_raw = KNeighborsClassifier()
s = cross_val_score(knn_raw, X_train, y_train, cv=5)
print(round(s.mean(), 3))
0.677

Logistic regression wins on training data. Only now do we open the test set, once.

best = models["logistic"].fit(X_train, y_train)
print(best.score(X_test, y_test))
1.0
test_pred = best.predict(X_test)
pd.crosstab(y_test, test_pred, rownames=["actual"],
            colnames=["predicted"])
predicted class_0 class_1 class_2
actual
class_0 15 0 0
class_1 0 18 0
class_2 0 0 12

A new bottle arrives: one real wine held back in the test set.

Common mistake: passing one row as a flat Series

bottle = X_test.iloc[0]
best.predict(bottle)
C:\Users\makmw\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:2830: UserWarning: X does not have valid feature names, but StandardScaler was fitted with feature names
  warnings.warn(
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[30], line 2
      1 bottle = X_test.iloc[0]
----> 2 best.predict(bottle)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\pipeline.py:794, in Pipeline.predict(self, X, **params)
    792 if not _routing_enabled():
    793     for _, name, transform in self._iter(with_final=False):
--> 794         Xt = transform.transform(Xt)
    795     return self.steps[-1][1].predict(Xt, **params)
    797 # metadata routing enabled

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_set_output.py:327, in _wrap_method_output.<locals>.wrapped(self, X, *args, **kwargs)
    325 @wraps(f)
    326 def wrapped(self, X, *args, **kwargs):
--> 327     data_to_wrap = f(self, X, *args, **kwargs)
    328     if isinstance(data_to_wrap, tuple):
    329         # only wrap the first output for cross decomposition
    330         return_tuple = (
    331             _wrap_data_with_container(method, data_to_wrap[0], X, self),
    332             *data_to_wrap[1:],
    333         )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\preprocessing\_data.py:1111, in StandardScaler.transform(self, X, copy)
   1108 check_is_fitted(self)
   1110 copy = copy if copy is not None else self.copy
-> 1111 X = validate_data(
   1112     self,
   1113     X,
   1114     reset=False,
   1115     accept_sparse="csr",
   1116     copy=copy,
   1117     dtype=supported_float_dtypes(xp, X_device),
   1118     force_writeable=True,
   1119     ensure_all_finite="allow-nan",
   1120 )
   1122 if sparse.issparse(X):
   1123     if self.with_mean:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:3041, in validate_data(_estimator, X, y, reset, validate_separately, skip_check_array, **check_params)
   3039         out = X, y
   3040 elif not no_val_X and no_val_y:
-> 3041     out = check_array(X, input_name="X", **check_params)
   3042 elif no_val_X and not no_val_y:
   3043     out = _check_y(y, **check_params)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:1076, in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, ensure_all_finite, ensure_non_negative, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)
   1069         else:
   1070             msg = (
   1071                 f"Expected 2D array, got 1D array instead:\narray={array}.\n"
   1072                 "Reshape your data either using array.reshape(-1, 1) if "
   1073                 "your data has a single feature or array.reshape(1, -1) "
   1074                 "if it contains a single sample."
   1075             )
-> 1076         raise ValueError(msg)
   1078 if dtype_numeric and hasattr(array.dtype, "kind") and array.dtype.kind in "USV":
   1079     raise ValueError(
   1080         "dtype='numeric' is not compatible with arrays of bytes/strings."
   1081         "Convert your data to numeric values explicitly instead."
   1082     )

ValueError: Expected a 2-dimensional container but got <class 'pandas.Series'> instead. Pass a DataFrame containing a single row (i.e. single sample) or a single column (i.e. single feature) instead.
bottle = X_test.iloc[[0]]
print(best.predict(bottle), y_test.iloc[0])
print(best.predict_proba(bottle).round(3))
['class_0'] class_0
[[0.926 0.072 0.002]]

Step 5: Interpretation: which chemistry matters?#

Problem: the lab needs to know which measurements carry the signal, not just a score.

forest = models["forest"].fit(X_train, y_train)
imp = pd.Series(forest.feature_importances_, index=X.columns)
imp = imp.sort_values(ascending=False)
imp.head(6).round(3)
color_intensity                 0.176
flavanoids                      0.174
alcohol                         0.138
proline                         0.136
hue                             0.105
od280/od315_of_diluted_wines    0.085
dtype: float64
imp.sort_values().plot.barh()
plt.title("What tells the cultivars apart")
plt.xlabel("Random forest importance")
plt.ylabel("Measurement")
plt.show()
No description has been provided for this image
lr = best[-1]
coef = pd.DataFrame(lr.coef_, index=lr.classes_, columns=X.columns)
coef.loc["class_2"].sort_values().round(2).iloc[[0, 1, -2, -1]]
flavanoids        -0.89
hue               -0.77
malic_acid         0.35
color_intensity    0.83
Name: class_2, dtype: float64

Your turn: save the cultivar profiles for the lab, then read them back.

import os
top4 = imp.index[:4].to_list()
profile = df.groupby("cultivar")[top4].mean().round(2)
profile.to_csv("m49_cultivar_profiles.csv")
pd.read_csv("m49_cultivar_profiles.csv")
cultivar color_intensity flavanoids alcohol proline
0 class_0 5.53 2.98 13.74 1115.71
1 class_1 3.09 2.08 12.28 519.51
2 class_2 7.40 0.78 13.15 629.90

Mini-project: which two tests are enough?#

Each lab test costs money. Can two measurements identify the cultivar almost as well as 13?

one = make_pipeline(StandardScaler(),
                    LogisticRegression(max_iter=1000))
for col in top4:
    s = cross_val_score(one, X_train[[col]], y_train, cv=5)
    print(f"{col:16} {s.mean():.3f}")
color_intensity  0.737
flavanoids       0.789
alcohol          0.685
proline          0.677
from itertools import combinations
rows = []
for pair in combinations(top4, 2):
    pipe = make_pipeline(StandardScaler(),
                         LogisticRegression(max_iter=1000))
    s = cross_val_score(pipe, X_train[list(pair)], y_train, cv=5)
    rows.append({"pair": " + ".join(pair), "cv": s.mean()})
pairs = pd.DataFrame(rows).sort_values("cv", ascending=False)
pairs.round(3)
pair cv
0 color_intensity + flavanoids 0.925
3 flavanoids + alcohol 0.909
2 color_intensity + proline 0.902
4 flavanoids + proline 0.873
1 color_intensity + alcohol 0.850
5 alcohol + proline 0.798
best_pair = pairs.iloc[0]["pair"].split(" + ")
pipe = make_pipeline(StandardScaler(),
                     LogisticRegression(max_iter=1000))
pipe.fit(X_train[best_pair], y_train)
print(best_pair, round(pipe.score(X_test[best_pair], y_test), 3))
['color_intensity', 'flavanoids'] 0.867
for name in sorted(y.unique()):
    part = df[df["cultivar"] == name]
    plt.scatter(part[best_pair[0]], part[best_pair[1]],
                label=name, alpha=0.7)
plt.title("Two lab tests, three cultivars")
plt.xlabel(best_pair[0])
plt.ylabel(best_pair[1])
plt.legend()
plt.show()
No description has been provided for this image

Finding: colour intensity + flavanoids name about 87% of new wines; all 13 tests named all 45. Two tests make a cheap first screen; the full panel is worth it when a wrong label is costly.

Recap#

  • Start with the question, then check the data before trusting it
  • Scale features before distance-based methods like k-means and kNN
  • Clustering finds groups without labels; check them when labels exist
  • Compare models with cross-validation; open the test set once
  • Finish with a finding someone can act on

Challenges#

  1. Run the same five KDD steps on data/mining/breast_cancer.csv.
  2. Tune C for the logistic regression with GridSearchCV. Does cross-validation improve?
  3. Try three tests instead of two. How close do you get to all 13?

What's next#

This is the end of the Data Mining series. Congratulations! Keep going with the Pandas series to sharpen your data wrangling. Or explore the channel's other series for new projects.

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.