Mathew K Analytics

Lesson 1 · Data Mining Full Course

Data Mining Full Course Part 1 (3 Hours): Foundations & Exploratory Data Analysis in Python

All 8 lesson notebooks from Part 1 of the Data Mining Full Course in Python, in teaching order. Each lesson starts with a heading; the timestamps below jump…

📓 Full notebook

Download .ipynb

Data Mining Full Course, Part 1: Foundations and Exploratory Data Analysis#

All 8 lesson notebooks from Part 1 of the Data Mining Full Course in Python, in teaching order. Each lesson starts with a heading; the timestamps below jump to it in the video.

  • 0:00 Lesson 1: Introduction to Data Mining and KDD Process
  • 0:22:55 Lesson 2: Data Mining vs Data Science vs Machine Learning
  • 0:45:41 Lesson 3: Applications of Data Mining in Business, Healthcare, and Security
  • 1:08:37 Lesson 4: Getting Started with Python, Pandas, and scikit-learn
  • 1:33:16 Lesson 5: Introduction to Exploratory Data Mining
  • 1:57:22 Lesson 6: Summarization and Descriptive Statistics
  • 2:21:41 Lesson 7: Visualization- Histograms, Scatterplots, Heatmaps
  • 2:45:25 Lesson 8: Exploratory Data Analysis and Visualization Basics

Data files: save the datasets from the lesson page in the same folder as this notebook. Files the lessons create are written to that folder too.

Data Mining 01: Introduction to Data Mining and KDD Process#

In this lesson you'll be able to:

  • Explain what data mining is, and how data differs from knowledge
  • Name the five steps of the KDD process
  • Select, clean and transform a real table ready for mining
  • Train a first model and judge it on passengers it has never seen
  • Reuse the same five steps to mine shopping baskets Datasets: Titanic passengers (data/pandas/titanic.csv) and real grocery baskets (data/mining/groceries_baskets.csv)

Warm-up: a table you already know#

If you followed the Pandas series, loading and peeking will feel familiar.

import pandas as pd
df = pd.read_csv("titanic.csv")
print(df.shape)
df.head(3)
(891, 12)
PassengerId Survived Pclass Name Sex Age SibSp Parch Ticket Fare Cabin Embarked
0 1 0 3 Braund, Mr. Owen Harris male 22.0 1 0 A/5 21171 7.2500 NaN S
1 2 1 1 Cumings, Mrs. John Bradley (Florence Briggs Th... female 38.0 1 0 PC 17599 71.2833 C85 C
2 3 1 3 Heikkinen, Miss. Laina female 26.0 0 0 STON/O2. 3101282 7.9250 NaN S
print(df["Survived"].mean().round(3))
df.groupby("Sex")["Survived"].mean().round(3)
0.384
Sex
female    0.742
male      0.189
Name: Survived, dtype: float64

From data to knowledge#

Problem: a file of facts is not yet an answer. What turns it into one?

  • Data: recorded facts, one row at a time
  • Knowledge: a pattern that holds across many rows and helps a decision
  • Data mining: finding those patterns automatically, in large data
df.loc[0, ["Name", "Pclass", "Fare", "Survived"]]
Name        Braund, Mr. Owen Harris
Pclass                            3
Fare                           7.25
Survived                          0
Name: 0, dtype: object

Quick check: is "third class passengers paid less" data or knowledge?

fares = df.groupby("Pclass")["Fare"].median()
fares.round(2)
Pclass
1    60.29
2    14.25
3     8.05
Name: Fare, dtype: float64

The KDD process: five steps from file to finding#

Problem: mining goes wrong when steps are skipped. KDD gives the order.

KDD = Knowledge Discovery in Databases

  1. Selection: pick the question, the rows and the columns
  2. Preprocessing: find and fix gaps and errors
  3. Transformation: reshape columns into a form a method can use
  4. Data mining: run a method that finds the pattern
  5. Interpretation / evaluation: is the pattern true and useful?

Our question: which passengers were most likely to survive, and why?

Step 1, Selection: choose the target and the evidence#

Problem: which column is the answer, and which columns could explain it?

print(list(df.columns))
['PassengerId', 'Survived', 'Pclass', 'Name', 'Sex', 'Age', 'SibSp', 'Parch', 'Ticket', 'Fare', 'Cabin', 'Embarked']
id_cols = ["PassengerId", "Name", "Ticket"]
df[id_cols].nunique()
PassengerId    891
Name           891
Ticket         681
dtype: int64
target = "Survived"
features = ["Pclass", "Sex", "Age", "SibSp", "Parch", "Fare",
            "Embarked"]
data = df[features + [target]].copy()
print(data.shape)
(891, 8)

Common mistake: selecting several columns with single brackets.

df["Pclass", "Sex"]
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\indexes\base.py:3641, in Index.get_loc(self, key)
   3640 try:
-> 3641     return self._engine.get_loc(casted_key)
   3642 except KeyError as err:

File pandas/_libs/index.pyx:168, in pandas._libs.index.IndexEngine.get_loc()

File pandas/_libs/index.pyx:176, in pandas._libs.index.IndexEngine.get_loc()

File pandas/_libs/index.pyx:583, in pandas._libs.index.StringObjectEngine._check_type()

KeyError: ('Pclass', 'Sex')

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
Cell In[9], line 1
----> 1 df["Pclass", "Sex"]

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\frame.py:4378, in DataFrame.__getitem__(self, key)
   4376 if self.columns.nlevels > 1:
   4377     return self._getitem_multilevel(key)
-> 4378 indexer = self.columns.get_loc(key)
   4379 if is_integer(indexer):
   4380     indexer = [indexer]

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\indexes\base.py:3648, in Index.get_loc(self, key)
   3643     if isinstance(casted_key, slice) or (
   3644         isinstance(casted_key, abc.Iterable)
   3645         and any(isinstance(x, slice) for x in casted_key)
   3646     ):
   3647         raise InvalidIndexError(key) from err
-> 3648     raise KeyError(key) from err
   3649 except TypeError:
   3650     # If we have a listlike key, _check_indexing_error will raise
   3651     #  InvalidIndexError. Otherwise we fall through and re-raise
   3652     #  the TypeError.
   3653     self._check_indexing_error(key)

KeyError: ('Pclass', 'Sex')
df[["Pclass", "Sex"]].head(3)
Pclass Sex
0 3 male
1 1 female
2 3 female

Step 2, Preprocessing: find and handle the gaps#

Problem: real data has holes, and most methods refuse to work with them.

data.isna().sum()
Pclass        0
Sex           0
Age         177
SibSp         0
Parch         0
Fare          0
Embarked      2
Survived      0
dtype: int64

Quick check: if we drop every row of the FULL table that has any gap, how many passengers are left?

print(len(df), len(df.dropna()))
891 183
print(len(data.dropna()))
712
top_port = data["Embarked"].mode()[0]
data["Embarked"] = data["Embarked"].fillna(top_port)
print(top_port, data["Embarked"].isna().sum())
S 0

Age has too many gaps to drop. We'll fill them with a typical age, the median. But which passengers should that median come from? Step 3 answers that.

Step 3, Transformation: numbers a model can learn from#

Problem: models need numbers, and they must be tested on data they never saw.

Common mistake: handing text columns straight to a model.

from sklearn.tree import DecisionTreeClassifier
model = DecisionTreeClassifier(random_state=42)
model.fit(data[features], data[target])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_27264\1960485069.py in ?()
      1 from sklearn.tree import DecisionTreeClassifier
      2 model = DecisionTreeClassifier(random_state=42)
----> 3 model.fit(data[features], data[target])

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py in ?(estimator, *args, **kwargs)
   1399                     )
   1400                 ),
   1401                 callback_management_context(estimator),
   1402             ):
-> 1403                 return fit_method(estimator, *args, **kwargs)

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\tree\_classes.py in ?(self, X, y, sample_weight, check_input)
   1011         self : DecisionTreeClassifier
   1012             Fitted estimator.
   1013         """
   1014 
-> 1015         super()._fit(
   1016             X,
   1017             y,
   1018             sample_weight=sample_weight,

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\tree\_classes.py in ?(self, X, y, sample_weight, check_input, missing_values_in_feature_mask)
    243             check_X_params = dict(
    244                 dtype=np.float32, accept_sparse="csc", ensure_all_finite=False
    245             )
    246             check_y_params = dict(ensure_2d=False, dtype=None)
--> 247             X, y = validate_data(
    248                 self, X, y, validate_separately=(check_X_params, check_y_params)
    249             )
    250 

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py in ?(_estimator, X, y, reset, validate_separately, skip_check_array, **check_params)
   3049             # :(
   3050             check_X_params, check_y_params = validate_separately
   3051             if "estimator" not in check_X_params:
   3052                 check_X_params = {**default_check_params, **check_X_params}
-> 3053             X = check_array(X, input_name="X", **check_X_params)
   3054             if "estimator" not in check_y_params:
   3055                 check_y_params = {**default_check_params, **check_y_params}
   3056             y = check_array(y, input_name="y", **check_y_params)

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py in ?(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, ensure_all_finite, ensure_non_negative, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)
   1035                         )
   1036                     array = xp.astype(array, dtype, copy=False)
   1037                 else:
   1038                     array = _asarray_with_order(array, order=order, dtype=dtype, xp=xp)
-> 1039             except ComplexWarning as complex_warning:
   1040                 raise ValueError(
   1041                     "Complex data not supported\n{}\n".format(array)
   1042                 ) from complex_warning

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_array_api.py in ?(array, dtype, order, copy, xp, device)
    979         # Use NumPy API to support order
    980         if copy is True:
    981             array = numpy.array(array, order=order, dtype=dtype)
    982         else:
--> 983             array = numpy.asarray(array, order=order, dtype=dtype)
    984 
    985         # At this point array is a NumPy ndarray. We convert it to an array
    986         # container that is consistent with the input's namespace.

~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\generic.py in ?(self, dtype, copy)
   2023             )
   2024         values = self._values
   2025         if copy is None:
   2026             # Note: branch avoids `copy=None` for NumPy 1.x support
-> 2027             arr = np.asarray(values, dtype=dtype)
   2028         else:
   2029             arr = np.array(values, dtype=dtype, copy=copy)
   2030 

ValueError: could not convert string to float: 'male'
X = pd.get_dummies(data[features], columns=["Sex", "Embarked"],
                   dtype=int)
y = data[target]
print(list(X.columns))
['Pclass', 'Age', 'SibSp', 'Parch', 'Fare', 'Sex_female', 'Sex_male', 'Embarked_C', 'Embarked_Q', 'Embarked_S']
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)
print(len(X_train), len(X_test))
712 179
age_median = X_train["Age"].median()
X_train["Age"] = X_train["Age"].fillna(age_median)
X_test["Age"] = X_test["Age"].fillna(age_median)
print(age_median, X_train["Age"].isna().sum(), X_test["Age"].isna().sum())
28.5 0 0

Your turn#

Add a column family to X_train and X_test: SibSp + Parch + 1 (the passenger plus relatives aboard).

for part in (X_train, X_test):
    part["family"] = part["SibSp"] + part["Parch"] + 1
X_train["family"].value_counts().sort_index().head(6)
family
1    434
2    129
3     77
4     22
5     14
6     16
Name: count, dtype: int64

Step 4, Data mining: let a model find the pattern#

Problem: hundreds of rows, many columns. Which combination predicts survival?

A decision tree learns a flowchart of yes/no questions, like "Is the passenger female?" Each question splits the passengers into purer groups of survivors and non-survivors.

baseline = (y_test == 0).mean()
print(f"Guess everyone died: {baseline:.1%}")
Guess everyone died: 61.5%
tree = DecisionTreeClassifier(max_depth=3, random_state=42)
tree.fit(X_train, y_train)
test_acc = tree.score(X_test, y_test)
print(f"Tree accuracy on unseen passengers: {test_acc:.1%}")
Tree accuracy on unseen passengers: 80.4%
from sklearn.tree import export_text
rules = export_text(tree, feature_names=list(X_train.columns),
                    decimals=1)
print(rules)
|--- Sex_male <= 0.5
|   |--- Pclass <= 2.5
|   |   |--- Fare <= 28.9
|   |   |   |--- class: 1
|   |   |--- Fare >  28.9
|   |   |   |--- class: 1
|   |--- Pclass >  2.5
|   |   |--- family <= 4.5
|   |   |   |--- class: 1
|   |   |--- family >  4.5
|   |   |   |--- class: 0
|--- Sex_male >  0.5
|   |--- Age <= 3.5
|   |   |--- family <= 4.5
|   |   |   |--- class: 1
|   |   |--- family >  4.5
|   |   |   |--- class: 0
|   |--- Age >  3.5
|   |   |--- Pclass <= 1.5
|   |   |   |--- class: 0
|   |   |--- Pclass >  1.5
|   |   |   |--- class: 0

sample = X_test.head(4)
check = sample[["Pclass", "Sex_female", "Age"]].copy()
check["predicted"] = tree.predict(sample)
check["actual"] = y_test.head(4)
check
Pclass Sex_female Age predicted actual
565 3 0 24.0 0 0
160 3 0 44.0 0 0
553 3 0 22.0 0 1
860 3 0 41.0 0 0

Quick check: which single column does the tree rely on most?

importance = pd.Series(tree.feature_importances_,
                       index=X_train.columns)
importance.sort_values(ascending=False).round(3).head(5)
Sex_male    0.633
Pclass      0.215
Age         0.090
family      0.058
Fare        0.004
dtype: float64

Step 5, Interpretation and evaluation: is the pattern real?#

Problem: a pattern can fit the past perfectly and still fail on new cases.

Quick check: a tree with no depth limit will fit the training passengers better. Will it also win on the test passengers?

deep = DecisionTreeClassifier(random_state=42)
deep.fit(X_train, y_train)
for name, m in [("depth 3", tree), ("no limit", deep)]:
    train_acc = m.score(X_train, y_train)
    print(name, round(train_acc, 3), round(m.score(X_test, y_test), 3))
depth 3 0.829 0.804
no limit 0.983 0.777

Rule: never judge a model on the rows it learned from. Always use held-out data.

pred = tree.predict(X_test)
pd.crosstab(y_test, pred, rownames=["actual"], colnames=["predicted"])
predicted 0 1
actual
0 99 11
1 24 45
rates = df.pivot_table(index="Sex", columns="Pclass",
                       values="Survived", aggfunc="mean")
rates.round(2)
Pclass 1 2 3
Sex
female 0.97 0.92 0.50
male 0.37 0.16 0.14
import matplotlib.pyplot as plt
ax = rates.T.plot(kind="bar", figsize=(7, 4))
ax.set_title("Titanic survival rate by class and sex")
ax.set_xlabel("Passenger class")
ax.set_ylabel("Share who survived")
plt.show()
No description has been provided for this image

Finding: sex and class drive survival. Women in 1st and 2nd class mostly lived; men in 2nd and 3rd mostly did not. This matches the "women and children first" policy of 1912: the pattern makes sense.

Mini-project: the same five steps on shopping baskets#

Question: when a basket holds whole milk, which other items are unusually likely?

path = "groceries_baskets.csv"
lines = pd.read_csv(path, header=None, sep="|")[0]
print(len(lines))
lines.head(3)
9835
0    citrus fruit,semi-finished bread,margarine,rea...
1                         tropical fruit,yogurt,coffee
2                                           whole milk
Name: 0, dtype: str
onehot = lines.str.get_dummies(sep=",")
print(onehot.shape)
onehot.sum().sort_values(ascending=False).head(5)
(9835, 169)
whole milk          2513
other vegetables    1903
rolls/buns          1809
soda                1715
yogurt              1372
dtype: int64
milk = onehot["whole milk"] == 1
with_milk = onehot[milk].mean()
overall = onehot.mean()
lift = (with_milk / overall).drop("whole milk")
common = overall[overall >= 0.03].index
lift[lift.index.isin(common)].sort_values(ascending=False).round(2).head(5)
butter                1.95
curd                  1.92
domestic eggs         1.85
whipped/sour cream    1.76
root vegetables       1.76
dtype: float64
top = lift[lift.index.isin(common)].nlargest(8)
ax = top.sort_values().plot(kind="barh", figsize=(7, 4))
ax.axvline(1, color="grey", linestyle="--")
ax.set_title("Items bought with whole milk: lift")
ax.set_xlabel("Lift (1 = no link)")
ax.set_ylabel("Item")
No description has been provided for this image
best = lift[lift.index.isin(common)].idxmax()
print(f"Milk shoppers buy {best} {lift[best]:.2f} times as often")
Milk shoppers buy butter 1.95 times as often

Recap#

  • Data is facts; knowledge is a pattern across many rows that helps a decision
  • KDD: selection, preprocessing, transformation, mining, interpretation
  • Fill gaps with values learned from the training rows only
  • Beat a baseline, and always score on held-out data
  • The same five steps work for prediction and for pattern discovery

Challenges#

  1. Refit the depth-3 tree without Fare and Embarked. Does test accuracy change?
  2. Try max_depth from 1 to 8 and print train and test accuracy for each.
  3. Repeat the basket mini-project for "yogurt" instead of whole milk.

What's next#

Lesson 02: Data Mining vs Data Science vs Machine Learning Three words people mix up, shown side by side on one real dataset.

Data Mining 02: Data Mining vs Data Science vs Machine Learning#

In this lesson you'll be able to:

  • Tell the three fields apart by the question each one asks
  • Answer a data science question and communicate it with a chart
  • Mine hidden groups from data that has no labels
  • Train a machine learning model and test it on unseen cases Dataset: Palmer penguins, 344 real birds from Antarctica (data/pandas/penguins.csv)

Warm-up: the first KDD steps on new data#

Lesson 01 walked the KDD process. Selection and preprocessing come first, whatever the field.

import pandas as pd
df = pd.read_csv("penguins.csv")
print(df.shape)
df.head(3)
(344, 7)
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g sex
0 Adelie Torgersen 39.1 18.7 181.0 3750.0 MALE
1 Adelie Torgersen 39.5 17.4 186.0 3800.0 FEMALE
2 Adelie Torgersen 40.3 18.0 195.0 3250.0 FEMALE
df.isna().sum()
species               0
island                0
bill_length_mm        2
bill_depth_mm         2
flipper_length_mm     2
body_mass_g           2
sex                  11
dtype: int64

Three fields, three kinds of question#

Problem: the words overlap, so define each one by what it is for.

  • Data science: the whole job, from a question to a communicated answer
  • Data mining: discovering patterns nobody told us to look for, in large data
  • Machine learning: algorithms that learn from examples to predict new cases Data science contains the other two; mining borrows many ML algorithms.

Quick check: which field fits each question? A. How much heavier are Gentoo penguins, and how do we show it? B. Are there natural groups of penguins, if nobody tells us the species? C. Which species is a newly measured penguin?

Answers: A is data science, B is data mining, C is machine learning. We'll now answer all three, one section each.

Data science: answer a question, then show it#

Question A: how much heavier are Gentoo penguins, and does sex matter?

Common mistake: averaging a whole table that still holds text columns.

df.groupby("species").mean()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[4], line 1
----> 1 df.groupby("species").mean()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\groupby\groupby.py:2285, in GroupBy.mean(self, numeric_only, skipna, engine, engine_kwargs)
   2277     return self._numba_agg_general(
   2278         grouped_mean,
   2279         executor.float_dtype_mapping,
   (...)
   2282         skipna=skipna,
   2283     )
   2284 else:
-> 2285     result = self._cython_agg_general(
   2286         "mean",
   2287         alt=lambda x: Series(x, copy=False).mean(
   2288             numeric_only=numeric_only, skipna=skipna
   2289         ),
   2290         numeric_only=numeric_only,
   2291         skipna=skipna,
   2292     )
   2293     return result.__finalize__(self.obj, method="groupby")

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\groupby\groupby.py:1808, in GroupBy._cython_agg_general(self, how, alt, numeric_only, min_count, **kwargs)
   1805     result = self._agg_py_fallback(how, values, ndim=data.ndim, alt=alt)
   1806     return result
-> 1808 new_mgr = data.grouped_reduce(array_func)
   1809 res = self._wrap_agged_manager(new_mgr)
   1810 if how in ["idxmin", "idxmax"]:
   1811     # mypy expects how to be Literal["idxmin", "idxmax"].

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\managers.py:1667, in BlockManager.grouped_reduce(self, func)
   1665             result_blocks = extend_blocks(applied, result_blocks)
   1666     else:
-> 1667         applied = blk.apply(func)
   1668         result_blocks = extend_blocks(applied, result_blocks)
   1670 if len(result_blocks) == 0:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\blocks.py:347, in Block.apply(self, func, **kwargs)
    341 @final
    342 def apply(self, func, **kwargs) -> list[Block]:
    343     """
    344     apply the function to my values; return a block if we are not
    345     one
    346     """
--> 347     result = func(self.values, **kwargs)
    349     result = maybe_coerce_values(result)
    350     return self._split_op_result(result)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\groupby\groupby.py:1783, in GroupBy._cython_agg_general.<locals>.array_func(values)
   1781 def array_func(values: ArrayLike) -> ArrayLike:
   1782     try:
-> 1783         result = self._grouper._cython_operation(
   1784             "aggregate",
   1785             values,
   1786             how,
   1787             axis=data.ndim - 1,
   1788             min_count=min_count,
   1789             **kwargs,
   1790         )
   1791     except NotImplementedError:
   1792         # generally if we have numeric_only=False
   1793         # and non-applicable functions
   1794         # try to python agg
   1795         # TODO: shouldn't min_count matter?
   1796         # TODO: avoid special casing SparseArray here
   1797         if how in ["any", "all"] and isinstance(values, SparseArray):

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\groupby\ops.py:963, in BaseGrouper._cython_operation(self, kind, values, how, axis, min_count, **kwargs)
    959 assert kind in ["transform", "aggregate"]
    961 cy_op = WrappedCythonOp(kind=kind, how=how, has_dropped_na=self.has_dropped_na)
--> 963 return cy_op.cython_operation(
    964     values=values,
    965     axis=axis,
    966     min_count=min_count,
    967     comp_ids=self.ids,
    968     ngroups=self.ngroups,
    969     **kwargs,
    970 )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\groupby\ops.py:553, in WrappedCythonOp.cython_operation(self, values, axis, min_count, comp_ids, ngroups, **kwargs)
    549 self._validate_axis(axis, values)
    551 if not isinstance(values, np.ndarray):
    552     # i.e. ExtensionArray
--> 553     return values._groupby_op(
    554         how=self.how,
    555         has_dropped_na=self.has_dropped_na,
    556         min_count=min_count,
    557         ngroups=ngroups,
    558         ids=comp_ids,
    559         **kwargs,
    560     )
    562 return self._cython_op_ndim_compat(
    563     values,
    564     min_count=min_count,
   (...)
    568     **kwargs,
    569 )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\arrow\array.py:2748, in ArrowExtensionArray._groupby_op(self, how, has_dropped_na, min_count, ngroups, ids, **kwargs)
   2736 if isinstance(self.dtype, StringDtype):
   2737     if how in [
   2738         "prod",
   2739         "mean",
   (...)
   2746         "skew",
   2747     ]:
-> 2748         raise TypeError(
   2749             f"dtype '{self.dtype}' does not support operation '{how}'"
   2750         )
   2751     return super()._groupby_op(
   2752         how=how,
   2753         has_dropped_na=has_dropped_na,
   (...)
   2757         **kwargs,
   2758     )
   2760 # maybe convert to a compatible dtype optimized for groupby

TypeError: dtype 'str' does not support operation 'mean'
cols = ["bill_length_mm", "flipper_length_mm", "body_mass_g"]
df.groupby("species")[cols].mean().round(1)
bill_length_mm flipper_length_mm body_mass_g
species
Adelie 38.8 190.0 3700.7
Chinstrap 48.8 195.8 3733.1
Gentoo 47.5 217.2 5076.0
mass = df.pivot_table(index="species", columns="sex",
                      values="body_mass_g", aggfunc="mean")
mass.round(0)
sex FEMALE MALE
species
Adelie 3369.0 4043.0
Chinstrap 3527.0 3939.0
Gentoo 4680.0 5485.0
import matplotlib.pyplot as plt
ax = mass.plot(kind="bar", figsize=(7, 4), rot=0)
ax.set_title("Average body mass of Palmer penguins")
ax.set_xlabel("Species")
ax.set_ylabel("Body mass (g)")
plt.show()
No description has been provided for this image
gap = mass.loc["Gentoo"].mean() - mass.loc["Adelie"].mean()
print(f"Gentoo penguins are about {gap:.0f} g heavier than Adelie")
Gentoo penguins are about 1376 g heavier than Adelie

Your turn#

Which island has the heaviest penguins on average?

df.groupby("island")["body_mass_g"].mean().round(0)
island
Biscoe       4716.0
Dream        3713.0
Torgersen    3706.0
Name: body_mass_g, dtype: float64
pd.crosstab(df["island"], df["species"])
species Adelie Chinstrap Gentoo
island
Biscoe 44 0 124
Dream 56 68 0
Torgersen 52 0 0

That's data science: a clear question, a summary, a chart and a plain sentence.

Data mining: discover groups nobody labelled#

Question B: hide the species. Do the measurements alone fall into natural groups?

k-means places k centre points and gives each bird to its nearest centre. It then moves each centre to the middle of its birds, and repeats until nothing changes.

num = ["bill_length_mm", "bill_depth_mm", "flipper_length_mm",
       "body_mass_g"]
from sklearn.cluster import KMeans
km = KMeans(n_clusters=3, n_init=10, random_state=42)

Common mistake: clustering data that still has gaps.

km.fit(df[num])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[12], line 1
----> 1 km.fit(df[num])

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py:1403, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1393     estimator._validate_params()
   1395 with (
   1396     config_context(
   1397         skip_parameter_validation=(
   (...)
   1401     callback_management_context(estimator),
   1402 ):
-> 1403     return fit_method(estimator, *args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\cluster\_kmeans.py:1459, in KMeans.fit(self, X, y, sample_weight)
   1431 @_fit_context(prefer_skip_nested_validation=True)
   1432 def fit(self, X, y=None, sample_weight=None):
   1433     """Compute k-means clustering.
   1434 
   1435     Parameters
   (...)
   1457         Fitted estimator.
   1458     """
-> 1459     X = validate_data(
   1460         self,
   1461         X,
   1462         accept_sparse="csr",
   1463         dtype=[np.float64, np.float32],
   1464         order="C",
   1465         copy=self.copy_x,
   1466         accept_large_sparse=False,
   1467     )
   1469     self._check_params_vs_input(X)
   1471     random_state = check_random_state(self.random_state)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:3041, in validate_data(_estimator, X, y, reset, validate_separately, skip_check_array, **check_params)
   3039         out = X, y
   3040 elif not no_val_X and no_val_y:
-> 3041     out = check_array(X, input_name="X", **check_params)
   3042 elif no_val_X and not no_val_y:
   3043     out = _check_y(y, **check_params)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:1090, in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, ensure_all_finite, ensure_non_negative, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)
   1084     raise ValueError(
   1085         f"Found array with dim {array.ndim},"
   1086         f" while dim <= 2 is required{context}."
   1087     )
   1089 if ensure_all_finite:
-> 1090     _assert_all_finite(
   1091         array,
   1092         input_name=input_name,
   1093         estimator_name=estimator_name,
   1094         allow_nan=ensure_all_finite == "allow-nan",
   1095     )
   1097 if copy:
   1098     if _is_numpy_namespace(xp):
   1099         # only make a copy if `array` and `array_orig` may share memory`

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:137, in _assert_all_finite(X, allow_nan, msg_dtype, estimator_name, input_name)
    134 if first_pass_isfinite:
    135     return
--> 137 _assert_all_finite_element_wise(
    138     X,
    139     xp=xp,
    140     allow_nan=allow_nan,
    141     msg_dtype=msg_dtype,
    142     estimator_name=estimator_name,
    143     input_name=input_name,
    144 )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:186, in _assert_all_finite_element_wise(X, xp, allow_nan, msg_dtype, estimator_name, input_name)
    169 if estimator_name and input_name == "X" and has_nan_error:
    170     # Improve the error message on how to handle missing values in
    171     # scikit-learn.
    172     msg_err += (
    173         f"\n{estimator_name} does not accept missing values"
    174         " encoded as NaN natively. For supervised learning, you might want"
   (...)
    184         "#estimators-that-handle-nan-values"
    185     )
--> 186 raise ValueError(msg_err)

ValueError: Input X contains NaN.
KMeans does not accept missing values encoded as NaN natively. For supervised learning, you might want to consider sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept missing values encoded as NaNs natively. Alternatively, it is possible to preprocess the data, for instance by using an imputer transformer in a pipeline or drop samples with missing values. See https://scikit-learn.org/stable/modules/impute.html You can find a list of all estimators that handle NaN values at the following page: https://scikit-learn.org/stable/modules/impute.html#estimators-that-handle-nan-values
birds = df.dropna(subset=num).copy()
print(len(birds))
birds[num].std().round(1)
342
bill_length_mm         5.5
bill_depth_mm          2.0
flipper_length_mm     14.1
body_mass_g          802.0
dtype: float64

Quick check: body mass spreads hundreds of times more than bill depth. What will k-means care about most?

km.fit(birds[num])
pd.crosstab(km.labels_, birds["species"], rownames=["group"])
species Adelie Chinstrap Gentoo
group
0 112 52 1
1 0 0 68
2 39 16 54
from sklearn.preprocessing import StandardScaler
scaled = StandardScaler().fit_transform(birds[num])
km.fit(scaled)
birds["raw_group"] = km.labels_
order = birds.groupby("raw_group")["flipper_length_mm"].mean()
names = {g: i for i, g in enumerate(order.sort_values().index)}
birds["group"] = birds["raw_group"].map(names)
pd.crosstab(birds["group"], birds["species"])
species Adelie Chinstrap Gentoo
group
0 127 5 0
1 24 63 0
2 0 0 123
birds.groupby("group")[num].mean().round(1)
bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
group
0 38.2 18.1 188.4 3584.7
1 47.5 18.8 196.9 3902.0
2 47.5 15.0 217.2 5076.0
ax = birds.plot.scatter(x="flipper_length_mm", y="bill_length_mm",
                        c="group", cmap="viridis", figsize=(7, 4))
ax.set_title("Penguin groups found by k-means (no labels used)")
ax.set_xlabel("Flipper length (mm)")
ax.set_ylabel("Bill length (mm)")
No description has been provided for this image
from sklearn.metrics import silhouette_score
for k in range(2, 6):
    test_km = KMeans(n_clusters=k, n_init=10, random_state=42)
    labels = test_km.fit_predict(scaled)
    print(k, round(silhouette_score(scaled, labels), 3))
2 0.532
3 0.447
4 0.4
5 0.379

Mining finds structure; a person decides whether it is useful.

Machine learning: predict penguins it has never seen#

Question C: given a new bird's measurements, which species is it?

k-nearest neighbours finds the 5 most similar known birds and takes a vote. Learning from labelled examples like this is called supervised learning.

from sklearn.model_selection import train_test_split
X, y = birds[num], birds["species"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)
print(len(X_train), len(X_test))
256 86
from sklearn.pipeline import make_pipeline
from sklearn.neighbors import KNeighborsClassifier
knn = make_pipeline(StandardScaler(), KNeighborsClassifier(5))
knn.fit(X_train, y_train)
print(f"Test accuracy: {knn.score(X_test, y_test):.1%}")
Test accuracy: 100.0%

Quick check: what happens to kNN accuracy if we skip the scaler?

raw_knn = KNeighborsClassifier(5).fit(X_train, y_train)
print(f"Without scaling: {raw_knn.score(X_test, y_test):.1%}")
Without scaling: 79.1%
new_birds = X_test.head(4)
result = new_birds.copy()
result["predicted"] = knn.predict(new_birds)
result["actual"] = y_test.head(4)
result[["flipper_length_mm", "body_mass_g", "predicted", "actual"]]
flipper_length_mm body_mass_g predicted actual
53 200.0 4050.0 Adelie Adelie
49 191.0 4150.0 Adelie Adelie
221 230.0 5700.0 Gentoo Gentoo
317 222.0 4875.0 Gentoo Gentoo

Common mistake: predicting with different columns from training.

knn.predict(result)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[24], line 1
----> 1 knn.predict(result)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\pipeline.py:794, in Pipeline.predict(self, X, **params)
    792 if not _routing_enabled():
    793     for _, name, transform in self._iter(with_final=False):
--> 794         Xt = transform.transform(Xt)
    795     return self.steps[-1][1].predict(Xt, **params)
    797 # metadata routing enabled

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_set_output.py:327, in _wrap_method_output.<locals>.wrapped(self, X, *args, **kwargs)
    325 @wraps(f)
    326 def wrapped(self, X, *args, **kwargs):
--> 327     data_to_wrap = f(self, X, *args, **kwargs)
    328     if isinstance(data_to_wrap, tuple):
    329         # only wrap the first output for cross decomposition
    330         return_tuple = (
    331             _wrap_data_with_container(method, data_to_wrap[0], X, self),
    332             *data_to_wrap[1:],
    333         )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\preprocessing\_data.py:1111, in StandardScaler.transform(self, X, copy)
   1108 check_is_fitted(self)
   1110 copy = copy if copy is not None else self.copy
-> 1111 X = validate_data(
   1112     self,
   1113     X,
   1114     reset=False,
   1115     accept_sparse="csr",
   1116     copy=copy,
   1117     dtype=supported_float_dtypes(xp, X_device),
   1118     force_writeable=True,
   1119     ensure_all_finite="allow-nan",
   1120 )
   1122 if sparse.issparse(X):
   1123     if self.with_mean:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:3016, in validate_data(_estimator, X, y, reset, validate_separately, skip_check_array, **check_params)
   2932 def validate_data(
   2933     _estimator,
   2934     /,
   (...)
   2940     **check_params,
   2941 ):
   2942     """Validate input data and set or check feature names and counts of the input.
   2943 
   2944     This helper function should be used in an estimator that requires input
   (...)
   3014         validated.
   3015     """
-> 3016     _check_feature_names(_estimator, X, reset=reset)
   3017     tags = get_tags(_estimator)
   3018     if y is None and tags.target_tags.required:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:2868, in _check_feature_names(estimator, X, reset)
   2865 if not missing_names and not unexpected_names:
   2866     message += "Feature names must be in the same order as they were in fit.\n"
-> 2868 raise ValueError(message)

ValueError: The feature names should match those that were passed during fit.
Feature names unseen at fit time:
- actual
- predicted
print(list(knn.feature_names_in_))
knn.predict(result[knn.feature_names_in_])
['bill_length_mm', 'bill_depth_mm', 'flipper_length_mm', 'body_mass_g']
array(['Adelie', 'Adelie', 'Gentoo', 'Gentoo'], dtype=object)

Your turn#

Train the same pipeline using only the two bill columns. How much accuracy is lost?

bill = ["bill_length_mm", "bill_depth_mm"]
knn_bill = make_pipeline(StandardScaler(), KNeighborsClassifier(5))
knn_bill.fit(X_train[bill], y_train)
print(f"{knn_bill.score(X_test[bill], y_test):.1%}")
95.3%

Side by side: how the three fields differ#

Problem: knowing which field you're in tells you how to judge success.

Data science Data mining Machine learning
Question What happened, and why? What patterns are hidden? What will this new case be?
Labels needed? Sometimes Often not Yes, for supervised
Judged by Clear, correct communication Useful, new patterns Accuracy on unseen data

Quick check: our k-means groups matched species well. Is that a machine learning accuracy score?

match = pd.crosstab(birds["group"], birds["species"]).max(axis=1)
print(f"{match.sum() / len(birds):.1%} of birds sit with their kind")
91.5% of birds sit with their kind

Mini-project: can measurements tell male from female?#

A machine learning question, judged against a baseline, with a data science answer.

sexed = birds.dropna(subset=["sex"])
Xs = pd.get_dummies(sexed[num + ["species"]], columns=["species"],
                    dtype=int)
ys = sexed["sex"]
print(len(sexed), ys.value_counts().to_dict())
333 {'MALE': 168, 'FEMALE': 165}
from sklearn.linear_model import LogisticRegression
Xs_tr, Xs_te, ys_tr, ys_te = train_test_split(
    Xs, ys, test_size=0.25, random_state=42, stratify=ys)
sex_model = make_pipeline(StandardScaler(), LogisticRegression())
sex_model.fit(Xs_tr, ys_tr)
print("Trained on", len(Xs_tr), "birds")
Trained on 249 birds
baseline = (ys_te == ys_tr.mode()[0]).mean()
acc = sex_model.score(Xs_te, ys_te)
print(f"Baseline {baseline:.1%}, model {acc:.1%}")
Baseline 50.0%, model 92.9%
no_species = make_pipeline(StandardScaler(), LogisticRegression())
no_species.fit(Xs_tr[num], ys_tr)
print(f"{no_species.score(Xs_te[num], ys_te):.1%}")
89.3%
coefs = sex_model[-1].coef_[0]
weights = pd.Series(coefs, index=Xs.columns).round(2)
weights.sort_values(ascending=False)
body_mass_g          2.79
bill_depth_mm        2.68
bill_length_mm       2.07
species_Adelie       1.25
flipper_length_mm    0.25
species_Gentoo      -0.64
species_Chinstrap   -0.79
dtype: float64
probs = sex_model.predict_proba(Xs_te.head(3)).round(2)
print(list(sex_model.classes_))
print(probs, list(ys_te.head(3)))
['FEMALE', 'MALE']
[[0.88 0.12]
 [0.26 0.74]
 [0.   1.  ]] ['FEMALE', 'MALE', 'MALE']
right = pd.Series(sex_model.predict(Xs_te) == ys_te, index=ys_te.index)
kind = sexed.loc[ys_te.index, "species"]
right.groupby(kind).agg(["mean", "size"]).round(3)
mean size
species
Adelie 0.969 32
Chinstrap 0.875 16
Gentoo 0.917 36

Finding: measurements identify sex far better than guessing, and knowing the species helps.

Recap#

  • Data science runs from a question to a communicated answer
  • Data mining discovers patterns, often without labels
  • Machine learning learns from examples to predict new cases
  • Scale features before distance-based methods like k-means and kNN
  • Judge each by its own yardstick: clarity, usefulness, or test accuracy

Challenges#

  1. Average flipper length by island and species. Which island holds which species?
  2. Run k-means with 2 groups. Which species end up together?
  3. Try KNeighborsClassifier with 1, 3, 7 and 15 neighbours. Does accuracy change?

What's next#

Lesson 03: Applications of Data Mining in Business, Healthcare, and Security Real shopping baskets, tumour scans and spam messages.

Data Mining 03: Applications of Data Mining in Business, Healthcare, and Security#

In this lesson you'll be able to:

  • Describe any application as question, data, method, decision and cost of error
  • Mine customer records for the segment most likely to leave
  • Build a tumour classifier and tune how cautious it is
  • Build a spam filter and check what it wrongly blocks Datasets: telco_churn.csv, breast_cancer.csv and sms_spam.csv (all in data/mining)

Warm-up: which field is this?#

Lesson 02 split work into data science, data mining and machine learning.

import pandas as pd
telco = pd.read_csv("telco_churn.csv")
print(telco.shape)
telco["Churn"].value_counts()
(7043, 21)
Churn
No     5174
Yes    1869
Name: count, dtype: int64
telco["left"] = (telco["Churn"] == "Yes").astype(int)
print(round(telco["left"].mean(), 3))
0.265

Every application has the same shape#

Problem: fields look different, but the same five questions decide every project.

Field Question Decision Worst mistake
Business Who will leave? Offer a deal Missing a leaver
Healthcare Is this tumour malignant? Order a biopsy Missing a cancer
Security Is this message spam? Block it Blocking a real message

Quick check: in which field is a false alarm cheapest?

Healthcare: an extra check on a benign tumour costs far less than a missed cancer. In security it's the opposite: a blocked real message is the costly error.

Business: find the customers about to leave#

Problem: keeping a customer is cheaper than winning a new one. Who is at risk?

Common mistake: trusting a column that only looks numeric.

telco["TotalCharges"].astype(float)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[4], line 1
----> 1 telco["TotalCharges"].astype(float)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\generic.py:6529, in NDFrame.astype(self, dtype, copy, errors)
   6525     results = [ser.astype(dtype, errors=errors) for _, ser in self.items()]
   6527 else:
   6528     # else, only a single dtype is given
-> 6529     new_data = self._mgr.astype(dtype=dtype, errors=errors)
   6530     res = self._constructor_from_mgr(new_data, axes=new_data.axes)
   6531     return res.__finalize__(self, method="astype")

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\managers.py:627, in BaseBlockManager.astype(self, dtype, errors)
    626 def astype(self, dtype, errors: str = "raise") -> Self:
--> 627     return self.apply("astype", dtype=dtype, errors=errors)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\managers.py:445, in BaseBlockManager.apply(self, f, align_keys, **kwargs)
    443         applied = b.apply(f, **kwargs)
    444     else:
--> 445         applied = getattr(b, f)(**kwargs)
    446     result_blocks = extend_blocks(applied, result_blocks)
    448 out = type(self).from_blocks(result_blocks, [ax.view() for ax in self.axes])

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\blocks.py:607, in Block.astype(self, dtype, errors, squeeze)
    604         raise ValueError("Can not squeeze with more than one column.")
    605     values = values[0, :]  # type: ignore[call-overload]
--> 607 new_values = astype_array_safe(values, dtype, errors=errors)
    609 new_values = maybe_coerce_values(new_values)
    611 refs = None

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\dtypes\astype.py:240, in astype_array_safe(values, dtype, copy, errors)
    237     dtype = dtype.numpy_dtype
    239 try:
--> 240     new_values = astype_array(values, dtype, copy=copy)
    241 except (ValueError, TypeError):
    242     # e.g. _astype_nansafe can fail on object-dtype of strings
    243     #  trying to convert to float
    244     if errors == "ignore":

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\dtypes\astype.py:182, in astype_array(values, dtype, copy)
    178     return values
    180 if not isinstance(values, np.ndarray):
    181     # i.e. ExtensionArray
--> 182     values = values.astype(dtype, copy=copy)
    184 else:
    185     values = _astype_nansafe(values, dtype, copy=copy)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\string_arrow.py:336, in ArrowStringArray.astype(self, dtype, copy)
    334     return dtype.__from_arrow__(data)
    335 elif isinstance(dtype, np.dtype) and np.issubdtype(dtype, np.floating):
--> 336     return self.to_numpy(dtype=dtype, na_value=np.nan)
    338 return super().astype(dtype, copy=copy)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\arrow\array.py:1803, in ArrowExtensionArray.to_numpy(self, dtype, copy, na_value)
   1801 result = data._pa_array.to_numpy()
   1802 if dtype is not None:
-> 1803     result = result.astype(dtype, copy=False)
   1804 if copy:
   1805     result = result.copy()

ValueError: could not convert string to float: ' '
total = pd.to_numeric(telco["TotalCharges"], errors="coerce")
telco["TotalCharges"] = total
telco.loc[total.isna(), ["tenure", "MonthlyCharges"]].head()
tenure MonthlyCharges
488 0 52.55
753 0 20.25
936 0 80.85
1082 0 25.75
1340 0 56.05
by_contract = telco.groupby("Contract")["left"].agg(["mean", "size"])
by_contract.round(3)
mean size
Contract
Month-to-month 0.427 3875
One year 0.113 1473
Two year 0.028 1695
import matplotlib.pyplot as plt
ax = by_contract["mean"].plot(kind="bar", rot=0, figsize=(7, 4))
ax.set_title("Share of customers who left, by contract")
ax.set_xlabel("Contract type")
ax.set_ylabel("Churn rate")
No description has been provided for this image
edges = [0, 12, 24, 48, 72]
bands = ["0-12", "13-24", "25-48", "49-72"]
telco["tenure_band"] = pd.cut(telco["tenure"], bins=edges,
                              labels=bands, include_lowest=True)
telco.groupby("tenure_band", observed=True)["left"].mean().round(3)
tenure_band
0-12     0.474
13-24    0.287
25-48    0.204
49-72    0.095
Name: left, dtype: float64

Quick check: which contract and tenure combination do you expect to be riskiest?

combo = telco.pivot_table(index="Contract", columns="tenure_band",
                          values="left", aggfunc="mean",
                          observed=True)
combo.round(2)
tenure_band 0-12 13-24 25-48 49-72
Contract
Month-to-month 0.51 0.38 0.33 0.26
One year 0.10 0.08 0.11 0.13
Two year 0.00 0.00 0.02 0.03

Your turn#

Find the churn rate for each PaymentMethod. Which one stands out?

pay = telco.groupby("PaymentMethod")["left"].mean()
pay.sort_values(ascending=False).round(3)
PaymentMethod
Electronic check             0.453
Mailed check                 0.191
Bank transfer (automatic)    0.167
Credit card (automatic)      0.152
Name: left, dtype: float64
new_monthly = (telco["Contract"] == "Month-to-month") & (
    telco["tenure"] <= 12)
seg = telco[new_monthly]
share = seg["left"].sum() / telco["left"].sum()
print(len(seg), round(seg["left"].mean(), 3), round(share, 3))
1994 0.514 0.548

Business finding: new month-to-month customers are the place to spend the retention budget.

Healthcare: flag tumours that need a closer look#

Problem: doctors need help deciding which tumours are likely malignant.

cancer = pd.read_csv("breast_cancer.csv")
print(cancer.shape)
cancer["diagnosis"].value_counts()
(569, 31)
diagnosis
benign       357
malignant    212
Name: count, dtype: int64
cols = ["mean radius", "worst concave points"]
cancer.groupby("diagnosis")[cols].mean().round(3)
mean radius worst concave points
diagnosis
benign 12.147 0.074
malignant 17.463 0.182
fig, ax = plt.subplots(figsize=(7, 4))
for label, part in cancer.groupby("diagnosis"):
    part["mean radius"].plot(kind="hist", bins=30, alpha=0.6,
                             label=label, ax=ax)
ax.set_title("Nucleus radius by diagnosis")
ax.set_xlabel("Mean radius")
ax.set_ylabel("Number of tumours")
ax.legend()
No description has been provided for this image

Logistic regression weighs every measurement and turns the total into a probability.

from sklearn.model_selection import train_test_split
X = cancer.drop(columns="diagnosis")
y = (cancer["diagnosis"] == "malignant").astype(int)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)
print(len(X_train), len(X_test))
426 143
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
clf = make_pipeline(StandardScaler(), LogisticRegression())
clf.fit(X_train, y_train)
print(f"Accuracy: {clf.score(X_test, y_test):.1%}")
Accuracy: 96.5%

Quick check: accuracy is high. Is that enough for a hospital?

pred = clf.predict(X_test)
pd.crosstab(y_test, pred, rownames=["actual"], colnames=["predicted"])
predicted 0 1
actual
0 89 1
1 4 49
from sklearn.metrics import recall_score
print(f"Recall: {recall_score(y_test, pred):.1%}")
Recall: 92.5%

Screening policy, set in advance: refer any tumour with at least a 30% chance.

prob = clf.predict_proba(X_test)[:, 1]
cautious = (prob >= 0.3).astype(int)
print(pd.crosstab(y_test, cautious, rownames=["actual"],
                  colnames=["flagged"]))
print(f"Recall: {recall_score(y_test, cautious):.1%}")
flagged   0   1
actual         
0        89   1
1         2  51
Recall: 96.2%
view = y_test.head(5).to_frame("actual")
view["chance"] = prob[:5].round(3)
view["flagged"] = cautious[:5]
view
actual chance flagged
12 1 0.999 1
506 0 0.004 0
321 1 1.000 1
71 0 0.000 0
375 0 0.035 0

Your turn#

How many test patients does the 30% policy send for a biopsy, and what share is that?

referred = cautious.sum()
print(referred, f"{referred / len(cautious):.1%}")
52 36.4%

Security: filter spam before it reaches people#

Problem: spam texts carry scams and phishing links. Block them, but never real messages.

sms = pd.read_csv("sms_spam.csv")
print(sms["label"].value_counts())
print(sms["message"].iloc[2][:70])
label
ham     4825
spam     747
Name: count, dtype: int64
Free entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Te
sms["length"] = sms["message"].str.len()
sms.groupby("label")["length"].median()
label
ham      52.0
spam    149.0
Name: length, dtype: float64
fig, ax = plt.subplots(figsize=(7, 4))
for label, part in sms.groupby("label"):
    part["length"].clip(upper=300).plot(kind="hist", bins=30,
                                        alpha=0.6, label=label, ax=ax)
ax.set_title("Text message length: ham vs spam")
ax.set_xlabel("Characters (capped at 300)")
ax.set_ylabel("Number of messages")
ax.legend()
No description has been provided for this image

Naive Bayes counts how often each word appears in spam and in normal messages. A new message is scored by how spammy its words are, taken together.

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
spam_filter = make_pipeline(CountVectorizer(), MultinomialNB())

Common mistake: giving a text model a table instead of a column of text.

spam_filter.fit(sms[["message"]], sms["label"])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[26], line 1
----> 1 spam_filter.fit(sms[["message"]], sms["label"])

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py:1403, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1393     estimator._validate_params()
   1395 with (
   1396     config_context(
   1397         skip_parameter_validation=(
   (...)
   1401     callback_management_context(estimator),
   1402 ):
-> 1403     return fit_method(estimator, *args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\pipeline.py:645, in Pipeline.fit(self, X, y, **params)
    639         subcontext.call_on_fit_task_begin(estimator=self, X=Xt, y=y)
    640         last_step_params = self._get_metadata_for_step(
    641             step_idx=len(self) - 1,
    642             step_params=routed_params[self.steps[-1][0]],
    643             all_params=params,
    644         )
--> 645         self._final_estimator.fit(Xt, y, **last_step_params["fit"])
    646         subcontext.call_on_fit_task_end(estimator=self, X=Xt, y=y)
    647 else:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py:1403, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1393     estimator._validate_params()
   1395 with (
   1396     config_context(
   1397         skip_parameter_validation=(
   (...)
   1401     callback_management_context(estimator),
   1402 ):
-> 1403     return fit_method(estimator, *args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\naive_bayes.py:767, in _BaseDiscreteNB.fit(self, X, y, sample_weight)
    746 @_fit_context(prefer_skip_nested_validation=True)
    747 def fit(self, X, y, sample_weight=None):
    748     """Fit Naive Bayes classifier according to X, y.
    749 
    750     Parameters
   (...)
    765         Returns the instance itself.
    766     """
--> 767     X, y = self._check_X_y(X, y)
    768     _, n_features = X.shape
    770     labelbin = LabelBinarizer()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\naive_bayes.py:613, in _BaseDiscreteNB._check_X_y(self, X, y, reset)
    611 def _check_X_y(self, X, y, reset=True):
    612     """Validate X and y in fit methods."""
--> 613     return validate_data(self, X, y, accept_sparse="csr", reset=reset)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:3058, in validate_data(_estimator, X, y, reset, validate_separately, skip_check_array, **check_params)
   3056         y = check_array(y, input_name="y", **check_y_params)
   3057     else:
-> 3058         X, y = check_X_y(X, y, **check_params)
   3059     out = X, y
   3061 if not no_val_X and check_params.get("ensure_2d", True):

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:1349, in check_X_y(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, ensure_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, estimator)
   1330 X = check_array(
   1331     X,
   1332     accept_sparse=accept_sparse,
   (...)
   1344     input_name="X",
   1345 )
   1347 y = _check_y(y, multi_output=multi_output, y_numeric=y_numeric, estimator=estimator)
-> 1349 check_consistent_length(X, y)
   1351 return X, y

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:458, in check_consistent_length(*arrays)
    456 lengths = [_num_samples(X) for X in arrays if X is not None]
    457 if len(set(lengths)) > 1:
--> 458     raise ValueError(
    459         "Found input variables with inconsistent numbers of samples: %r"
    460         % [int(l) for l in lengths]
    461     )

ValueError: Found input variables with inconsistent numbers of samples: [1, 5572]
m_train, m_test, l_train, l_test = train_test_split(
    sms["message"], sms["label"], test_size=0.25, random_state=42,
    stratify=sms["label"])
spam_filter.fit(m_train, l_train)
print(f"Accuracy: {spam_filter.score(m_test, l_test):.1%}")
Accuracy: 98.6%
guess = spam_filter.predict(m_test)
pd.crosstab(l_test, guess, rownames=["actual"], colnames=["predicted"])
predicted ham spam
actual
ham 1204 2
spam 17 170

Your turn#

Run the filter on the first 4 held-back messages and compare with their labels.

sample = m_test.head(4)
verdicts = spam_filter.predict(sample)
for text, v, truth in zip(sample, verdicts, l_test.head(4)):
    print(v, truth, "|", text[:50])
ham ham | Blank is Blank. But wat is blank? Lol
ham spam | Hi babe its Jordan, how r u? Im home from abroad a
ham ham | How long has it been since you screamed, princess?
ham ham | Lol no. Just trying to make your day a little more
nb = spam_filter[-1]
words = spam_filter[0].get_feature_names_out()
spamminess = nb.feature_log_prob_[1] - nb.feature_log_prob_[0]
pd.Series(spamminess, index=words).nlargest(8).round(2)
claim         5.43
prize         5.13
150p          4.92
tone          4.71
18            4.58
500           4.58
cs            4.58
guaranteed    4.58
dtype: float64

Same tools, different costs#

Problem: one score can't serve every field. Choose the metric that matches the worst mistake.

  • Recall: of the real positives, how many did we catch?
  • Precision: of everything we flagged, how much was really positive?

Quick check: which model lets more of its real positives slip through?

from sklearn.metrics import precision_score
for name, truth, flags, pos in [
        ("tumours", y_test, cautious, 1),
        ("spam", l_test, guess, "spam")]:
    p = precision_score(truth, flags, pos_label=pos)
    r = recall_score(truth, flags, pos_label=pos)
    print(f"{name}: precision {p:.3f}, recall {r:.3f}")
tumours: precision 0.981, recall 0.962
spam: precision 0.988, recall 0.909

The spam filter is built to be sure before it blocks; the tumour model to catch nearly every cancer.

Mini-project: would you switch this spam filter on?#

Check exactly what it gets wrong, then decide.

blocked_real = m_test[(guess == "spam") & (l_test == "ham")]
print(len(blocked_real), "of", (l_test == "ham").sum())
for text in blocked_real:
    print("-", text[:75])
2 of 1206
- We have sent JD for Customer Service cum Accounts Executive to ur mail id, 
- Can u get pic msgs to your phone?
missed = (guess == "ham") & (l_test == "spam")
print(missed.sum(), "of", (l_test == "spam").sum())
17 of 187
per_1000 = 1000 / len(m_test)
wrong_block = len(blocked_real) * per_1000
let_through = missed.sum() * per_1000
print(f"Per 1000 texts: {wrong_block:.1f} real blocked, "
      f"{let_through:.1f} spam let through")
Per 1000 texts: 1.4 real blocked, 12.2 spam let through

Decision: switch it on, but let users check a spam folder for the rare wrongly blocked message.

Recap#

  • Every application: question, data, method, decision, cost of error
  • Business: simple group rates can reveal the riskiest customers
  • Healthcare: favour recall, and set the threshold as a policy
  • Security: favour precision, and read the actual mistakes
  • Clean real messy values first, such as blank text in a numeric column

Challenges#

  1. Find the churn rate by InternetService. Which service loses the most customers?
  2. Refit the tumour model on only the 10 'mean' columns. How does recall change?
  3. Add CountVectorizer(lowercase=False) to the spam filter. Better or worse?

What's next#

Lesson 04: Getting Started with Python, Pandas, and scikit-learn A tour of the toolkit behind every lesson in this series.

Data Mining 04: Getting Started with Python, Pandas, and scikit-learn#

In this lesson you'll be able to:

  • Use lists, dictionaries and functions on real data values
  • Load, inspect, select and summarise a table with pandas
  • Check data for gaps and duplicates before mining it
  • Train and test a classifier with scikit-learn's fit, predict, score
  • Find groups without labels using k-means clustering Datasets: Iris flowers (data/pandas/iris.csv) and the Titanic passenger list (data/pandas/titanic.csv)

Warm-up: every application starts with a table#

Lesson 03 toured data mining in business, healthcare and security. Each one began with rows (records) and columns (features).

import pandas as pd
iris = pd.read_csv("iris.csv")
print(iris.shape)
iris.head()
(150, 5)
sepal_length sepal_width petal_length petal_width species
0 5.1 3.5 1.4 0.2 setosa
1 4.9 3.0 1.4 0.2 setosa
2 4.7 3.2 1.3 0.2 setosa
3 4.6 3.1 1.5 0.2 setosa
4 5.0 3.6 1.4 0.2 setosa

Python building blocks on real values#

Problem: hold, count and transform values before pandas takes over.

lengths = iris["petal_length"].head(5).to_list()
print(lengths)
print(type(lengths), len(lengths))
[1.4, 1.4, 1.3, 1.5, 1.4]
<class 'list'> 5
total = sum(lengths)
average = total / len(lengths)
print(round(total, 2), round(average, 2))
7.0 1.4
counts = iris["species"].value_counts().to_dict()
for name, n in counts.items():
    print(f"{name}: {n} flowers")
setosa: 50 flowers
versicolor: 50 flowers
virginica: 50 flowers
def cm_to_mm(values):
    """Convert a list of centimetre values to millimetres."""
    return [round(v * 10, 1) for v in values]

print(cm_to_mm(lengths))
[14.0, 14.0, 13.0, 15.0, 14.0]
all_petals = iris["petal_length"].to_list()
big = [v for v in all_petals if v > 6]
print(len(big), max(big))
if len(big) > 5:
    print("Several giant petals")
else:
    print("Only a handful of giant petals")
9 6.9
Several giant petals

Common mistake: expecting a plain list to behave like a pandas column

lengths.mean()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[8], line 1
----> 1 lengths.mean()

AttributeError: 'list' object has no attribute 'mean'
petals = pd.Series(lengths)
print(petals.mean(), petals.max())
1.4 1.5

Inspect, select and summarise with pandas#

Problem: understand a table's columns before asking it questions.

iris.dtypes
sepal_length    float64
sepal_width     float64
petal_length    float64
petal_width     float64
species             str
dtype: object
features = ["sepal_length", "sepal_width",
            "petal_length", "petal_width"]
iris[features].describe().round(2)
sepal_length sepal_width petal_length petal_width
count 150.00 150.00 150.00 150.00
mean 5.84 3.06 3.76 1.20
std 0.83 0.44 1.77 0.76
min 4.30 2.00 1.00 0.10
25% 5.10 2.80 1.60 0.30
50% 5.80 3.00 4.35 1.30
75% 6.40 3.30 5.10 1.80
max 7.90 4.40 6.90 2.50
long_petals = iris[iris["petal_length"] > 5]
long_petals["species"].value_counts()
species
virginica     41
versicolor     1
Name: count, dtype: int64
iris.loc[[0, 50, 100], features]
sepal_length sepal_width petal_length petal_width
0 5.1 3.5 1.4 0.2
50 7.0 3.2 4.7 1.4
100 6.3 3.3 6.0 2.5
iris["petal_area"] = iris["petal_length"] * iris["petal_width"]
top = iris.sort_values("petal_area", ascending=False)
top[["petal_area", "species"]].head(3).round(2)
petal_area species
118 15.87 virginica
109 15.25 virginica
100 15.00 virginica

Quick check: which species has the longest petals on average?

by_species = iris.groupby("species")[features].mean()
by_species.round(2)
sepal_length sepal_width petal_length petal_width
species
setosa 5.01 3.43 1.46 0.25
versicolor 5.94 2.77 4.26 1.33
virginica 6.59 2.97 5.55 2.03

Common mistake: guessing a column name

iris["Petal Length"].mean()
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\indexes\base.py:3641, in Index.get_loc(self, key)
   3640 try:
-> 3641     return self._engine.get_loc(casted_key)
   3642 except KeyError as err:

File pandas/_libs/index.pyx:168, in pandas._libs.index.IndexEngine.get_loc()

File pandas/_libs/index.pyx:197, in pandas._libs.index.IndexEngine.get_loc()

File pandas/_libs/hashtable_class_helper.pxi:7668, in pandas._libs.hashtable.PyObjectHashTable.get_item()

File pandas/_libs/hashtable_class_helper.pxi:7676, in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'Petal Length'

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
Cell In[16], line 1
----> 1 iris["Petal Length"].mean()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\frame.py:4378, in DataFrame.__getitem__(self, key)
   4376 if self.columns.nlevels > 1:
   4377     return self._getitem_multilevel(key)
-> 4378 indexer = self.columns.get_loc(key)
   4379 if is_integer(indexer):
   4380     indexer = [indexer]

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\indexes\base.py:3648, in Index.get_loc(self, key)
   3643     if isinstance(casted_key, slice) or (
   3644         isinstance(casted_key, abc.Iterable)
   3645         and any(isinstance(x, slice) for x in casted_key)
   3646     ):
   3647         raise InvalidIndexError(key) from err
-> 3648     raise KeyError(key) from err
   3649 except TypeError:
   3650     # If we have a listlike key, _check_indexing_error will raise
   3651     #  InvalidIndexError. Otherwise we fall through and re-raise
   3652     #  the TypeError.
   3653     self._check_indexing_error(key)

KeyError: 'Petal Length'
print(iris.columns.to_list())
print(iris["petal_length"].mean().round(2))
['sepal_length', 'sepal_width', 'petal_length', 'petal_width', 'species', 'petal_area']
3.76

Check for gaps and duplicates before mining#

Problem: models fail or mislead on missing and repeated rows.

titanic = pd.read_csv("titanic.csv")
print(titanic.shape)
titanic.isna().sum()
(891, 12)
PassengerId      0
Survived         0
Pclass           0
Name             0
Sex              0
Age            177
SibSp            0
Parch            0
Ticket           0
Fare             0
Cabin          687
Embarked         2
dtype: int64

Quick check: if we drop every row with a gap, how many of the 891 passengers survive the cut?

print(len(titanic.dropna()))
183
titanic = titanic.drop(columns="Cabin")
titanic = titanic.dropna(subset=["Embarked"])
print(titanic.shape)
print(titanic["Age"].isna().sum())
(889, 11)
177
titanic["Age"].describe().round(1)
count    712.0
mean      29.6
std       14.5
min        0.4
25%       20.0
50%       28.0
75%       38.0
max       80.0
Name: Age, dtype: float64
titanic["Embarked"].value_counts()
Embarked
S    644
C    168
Q     77
Name: count, dtype: int64
print(iris.duplicated().sum())
iris[iris.duplicated(keep=False)]
1
sepal_length sepal_width petal_length petal_width species petal_area
101 5.8 2.7 5.1 1.9 virginica 9.69
142 5.8 2.7 5.1 1.9 virginica 9.69

Your turn: count the duplicated rows in the Titanic table.

print(titanic.duplicated().sum())
print(titanic["PassengerId"].is_unique)
0
True

Train and test a classifier with scikit-learn#

Problem: name an iris species from its four measurements.

  • X holds the features, y holds the answers
  • fit learns from training rows, predict labels new rows
  • score checks accuracy on rows the model never saw
from sklearn.model_selection import train_test_split

X = iris[features]
y = iris["species"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)
print(X_train.shape, X_test.shape)
(120, 4) (30, 4)

Common mistake: predicting before the model has learned

from sklearn.tree import DecisionTreeClassifier
tree = DecisionTreeClassifier(max_depth=3, random_state=42)
tree.predict(X_test)
---------------------------------------------------------------------------
NotFittedError                            Traceback (most recent call last)
Cell In[26], line 3
      1 from sklearn.tree import DecisionTreeClassifier
      2 tree = DecisionTreeClassifier(max_depth=3, random_state=42)
----> 3 tree.predict(X_test)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\tree\_classes.py:524, in BaseDecisionTree.predict(self, X, check_input)
    501 def predict(self, X, check_input=True):
    502     """Predict class or regression value for X.
    503 
    504     For a classification model, the predicted class for each sample in X is
   (...)
    522         The predicted classes, or the predict values.
    523     """
--> 524     check_is_fitted(self)
    525     X = self._validate_X_predict(X, check_input)
    526     proba = self.tree_.predict(X)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:1721, in check_is_fitted(estimator, attributes, msg, all_or_any)
   1718     return
   1720 if not _is_fitted(estimator, attributes, all_or_any):
-> 1721     raise NotFittedError(msg % {"name": type(estimator).__name__})

NotFittedError: This DecisionTreeClassifier instance is not fitted yet. Call 'fit' with appropriate arguments before using this estimator.
tree.fit(X_train, y_train)
print(tree.score(X_test, y_test))
0.9666666666666667
new_rows = X_test.head(3)
print(tree.predict(new_rows))
print(y_test.head(3).to_list())
['setosa' 'virginica' 'versicolor']
['setosa', 'virginica', 'versicolor']
pred = tree.predict(X_test)
pd.crosstab(y_test, pred, rownames=["true"], colnames=["predicted"])
predicted setosa versicolor virginica
true
setosa 10 0 0
versicolor 0 9 1
virginica 0 0 10
imp = pd.Series(tree.feature_importances_, index=features)
print(imp.sort_values(ascending=False).round(3))
petal_length    0.579
petal_width     0.421
sepal_width     0.000
sepal_length    0.000
dtype: float64
from sklearn.tree import export_text
print(export_text(tree, feature_names=features))
|--- petal_length <= 2.45
|   |--- class: setosa
|--- petal_length >  2.45
|   |--- petal_width <= 1.65
|   |   |--- petal_length <= 4.95
|   |   |   |--- class: versicolor
|   |   |--- petal_length >  4.95
|   |   |   |--- class: virginica
|   |--- petal_width >  1.65
|   |   |--- petal_length <= 4.85
|   |   |   |--- class: virginica
|   |   |--- petal_length >  4.85
|   |   |   |--- class: virginica

Your turn: swap the tree for a k-nearest neighbours classifier and score it.

from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train, y_train)
print(knn.score(X_test, y_test))
1.0

Find groups without labels using k-means#

Problem: discover natural groups when nobody tells us the answer.

from sklearn.cluster import KMeans

petal = iris[["petal_length", "petal_width"]]
km = KMeans(n_clusters=3, n_init=10, random_state=42)
raw = km.fit_predict(petal)
print(sorted(pd.Series(raw).value_counts().to_list()))
[48, 50, 52]
order = petal.groupby(raw)["petal_length"].mean().rank()
iris["cluster"] = pd.Series(raw).map(order.astype(int) - 1)
pd.crosstab(iris["cluster"], iris["species"])
species setosa versicolor virginica
cluster
0 50 0 0
1 0 48 4
2 0 2 46
iris.groupby("cluster")[["petal_length", "petal_width"]].mean().round(2)
petal_length petal_width
cluster
0 1.46 0.25
1 4.27 1.34
2 5.60 2.04
import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=(7, 4))
ax.scatter(iris["petal_length"], iris["petal_width"],
           c=iris["cluster"], cmap="viridis", alpha=0.7)
ax.set_title("k-means finds three petal groups in the iris data")
ax.set_xlabel("Petal length (cm)")
ax.set_ylabel("Petal width (cm)")
plt.show()
No description has been provided for this image

Quick check: we chose 3 clusters because we know there are 3 species. What if we didn't?

for k in range(1, 7):
    km_k = KMeans(n_clusters=k, n_init=10, random_state=42)
    km_k.fit(petal)
    print(k, round(km_k.inertia_, 1))
1 550.9
2 86.4
3 31.4
4 19.5
5 13.9
6 11.0

Mini-project: who survived the Titanic?#

Use class, sex and age to predict survival, and beat a simple baseline.

rates = titanic.groupby(["Sex", "Pclass"])["Survived"].mean()
rates.unstack().round(2)
Pclass 1 2 3
Sex
female 0.97 0.92 0.50
male 0.37 0.16 0.14
titanic["is_female"] = (titanic["Sex"] == "female").astype(int)
cols = ["Pclass", "is_female", "Age"]
Xt_train, Xt_test, yt_train, yt_test = train_test_split(
    titanic[cols], titanic["Survived"], test_size=0.2,
    random_state=42, stratify=titanic["Survived"])
print(Xt_train.shape, Xt_test.shape)
(711, 3) (178, 3)

Common mistake: fitting a model on data with gaps

from sklearn.linear_model import LogisticRegression
logit = LogisticRegression(max_iter=1000)
logit.fit(Xt_train, yt_train)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[40], line 3
      1 from sklearn.linear_model import LogisticRegression
      2 logit = LogisticRegression(max_iter=1000)
----> 3 logit.fit(Xt_train, yt_train)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py:1403, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1393     estimator._validate_params()
   1395 with (
   1396     config_context(
   1397         skip_parameter_validation=(
   (...)
   1401     callback_management_context(estimator),
   1402 ):
-> 1403     return fit_method(estimator, *args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\linear_model\_logistic.py:1437, in LogisticRegression.fit(self, X, y, sample_weight)
   1434 if self.n_jobs is not None:
   1435     warnings.warn(msg, category=FutureWarning)
-> 1437 X, y = validate_data(
   1438     self,
   1439     X,
   1440     y,
   1441     accept_sparse="csr",
   1442     dtype=[xp.float64, xp.float32],
   1443     order="C",
   1444     accept_large_sparse=solver not in ["liblinear", "sag", "saga"],
   1445 )
   1446 n_features = X.shape[1]
   1447 check_classification_targets(y)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:3058, in validate_data(_estimator, X, y, reset, validate_separately, skip_check_array, **check_params)
   3056         y = check_array(y, input_name="y", **check_y_params)
   3057     else:
-> 3058         X, y = check_X_y(X, y, **check_params)
   3059     out = X, y
   3061 if not no_val_X and check_params.get("ensure_2d", True):

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:1330, in check_X_y(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, ensure_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, estimator)
   1325         estimator_name = _check_estimator_name(estimator)
   1326     raise ValueError(
   1327         f"{estimator_name} requires y to be passed, but the target y is None"
   1328     )
-> 1330 X = check_array(
   1331     X,
   1332     accept_sparse=accept_sparse,
   1333     accept_large_sparse=accept_large_sparse,
   1334     dtype=dtype,
   1335     order=order,
   1336     copy=copy,
   1337     force_writeable=force_writeable,
   1338     ensure_all_finite=ensure_all_finite,
   1339     ensure_2d=ensure_2d,
   1340     allow_nd=allow_nd,
   1341     ensure_min_samples=ensure_min_samples,
   1342     ensure_min_features=ensure_min_features,
   1343     estimator=estimator,
   1344     input_name="X",
   1345 )
   1347 y = _check_y(y, multi_output=multi_output, y_numeric=y_numeric, estimator=estimator)
   1349 check_consistent_length(X, y)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:1090, in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, ensure_all_finite, ensure_non_negative, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)
   1084     raise ValueError(
   1085         f"Found array with dim {array.ndim},"
   1086         f" while dim <= 2 is required{context}."
   1087     )
   1089 if ensure_all_finite:
-> 1090     _assert_all_finite(
   1091         array,
   1092         input_name=input_name,
   1093         estimator_name=estimator_name,
   1094         allow_nan=ensure_all_finite == "allow-nan",
   1095     )
   1097 if copy:
   1098     if _is_numpy_namespace(xp):
   1099         # only make a copy if `array` and `array_orig` may share memory`

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:137, in _assert_all_finite(X, allow_nan, msg_dtype, estimator_name, input_name)
    134 if first_pass_isfinite:
    135     return
--> 137 _assert_all_finite_element_wise(
    138     X,
    139     xp=xp,
    140     allow_nan=allow_nan,
    141     msg_dtype=msg_dtype,
    142     estimator_name=estimator_name,
    143     input_name=input_name,
    144 )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:186, in _assert_all_finite_element_wise(X, xp, allow_nan, msg_dtype, estimator_name, input_name)
    169 if estimator_name and input_name == "X" and has_nan_error:
    170     # Improve the error message on how to handle missing values in
    171     # scikit-learn.
    172     msg_err += (
    173         f"\n{estimator_name} does not accept missing values"
    174         " encoded as NaN natively. For supervised learning, you might want"
   (...)
    184         "#estimators-that-handle-nan-values"
    185     )
--> 186 raise ValueError(msg_err)

ValueError: Input X contains NaN.
LogisticRegression does not accept missing values encoded as NaN natively. For supervised learning, you might want to consider sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept missing values encoded as NaNs natively. Alternatively, it is possible to preprocess the data, for instance by using an imputer transformer in a pipeline or drop samples with missing values. See https://scikit-learn.org/stable/modules/impute.html You can find a list of all estimators that handle NaN values at the following page: https://scikit-learn.org/stable/modules/impute.html#estimators-that-handle-nan-values
age_median = Xt_train["Age"].median()
Xt_train = Xt_train.fillna({"Age": age_median})
Xt_test = Xt_test.fillna({"Age": age_median})
logit.fit(Xt_train, yt_train)
print(age_median, round(logit.score(Xt_test, yt_test), 3))
28.0 0.792
baseline = (yt_test == 0).mean()
print(round(baseline, 3))
0.618
coefs = pd.Series(logit.coef_[0], index=cols)
print(coefs.round(2))
Pclass      -1.08
is_female    2.51
Age         -0.03
dtype: float64
probs = logit.predict_proba(Xt_test.head(8))[:, 1]
show = Xt_test.head(8).assign(p_survive=probs.round(2))
show.assign(survived=yt_test.head(8))
Pclass is_female Age p_survive survived
160 3 0 44.0 0.06 0
126 3 0 28.0 0.10 0
428 3 0 28.0 0.10 0
422 3 0 29.0 0.09 0
565 3 0 24.0 0.11 0
297 1 1 2.0 0.96 0
75 3 0 25.0 0.11 0
879 1 1 56.0 0.83 1

Recap#

  • Lists, dicts and functions hold and transform values; wrap them in pandas for maths
  • Check dtypes, gaps and duplicates before any model
  • Every scikit-learn model: fit, then predict or score
  • Hold out a test set, and fill gaps using training data only
  • Clustering finds groups with no labels; relabel them for stable names

Challenges#

  1. Add Fare to the Titanic features. Does test accuracy change?
  2. Run k-means on all four iris measurements. Do the clusters match species better?
  3. Try max_depth=1 and max_depth=5 for the iris tree. What happens to the score?

What's next#

Lesson 05: Introduction to Exploratory Data Mining Explore 9,835 real grocery baskets and find which items go together.

Data Mining 05: Introduction to Exploratory Data Mining#

In this lesson you'll be able to:

  • Load ragged transaction data into a tidy pandas table
  • Profile baskets and items with counts, shares and support
  • Measure which items go together with confidence and lift
  • Compare patterns across segments of the data
  • Let the Apriori algorithm search for rules automatically Dataset: 9,835 real grocery baskets (data/mining/groceries_baskets.csv)

Warm-up: Python lists meet real baskets#

Lesson 04 used lists, dictionaries and pandas. Same tools, new data.

import pandas as pd

with open("groceries_baskets.csv") as f:
    lines = f.read().splitlines()
print(len(lines))
for line in lines[:3]:
    print(line)
9835
citrus fruit,semi-finished bread,margarine,ready soups
tropical fruit,yogurt,coffee
whole milk

Load ragged transactions into a tidy table#

Problem: each basket has a different number of items.

Common mistake: reading ragged lines with a plain read_csv

pd.read_csv("groceries_baskets.csv", header=None)
---------------------------------------------------------------------------
ParserError                               Traceback (most recent call last)
Cell In[3], line 1
----> 1 pd.read_csv("data/mining/groceries_baskets.csv", header=None)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\io\parsers\readers.py:872, in read_csv(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, skip_blank_lines, parse_dates, date_format, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, encoding_errors, dialect, on_bad_lines, low_memory, memory_map, float_precision, storage_options, dtype_backend)
    860 kwds_defaults = _refine_defaults_read(
    861     dialect,
    862     delimiter,
   (...)
    868     dtype_backend=dtype_backend,
    869 )
    870 kwds.update(kwds_defaults)
--> 872 return _read(filepath_or_buffer, kwds)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\io\parsers\readers.py:306, in _read(filepath_or_buffer, kwds)
    303     return parser
    305 with parser:
--> 306     return parser.read(nrows)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\io\parsers\readers.py:1950, in TextFileReader.read(self, nrows)
   1943 nrows = validate_integer("nrows", nrows)
   1944 try:
   1945     # error: "ParserBase" has no attribute "read"
   1946     (
   1947         index,
   1948         columns,
   1949         col_dict,
-> 1950     ) = self._engine.read(  # type: ignore[attr-defined]
   1951         nrows
   1952     )
   1953 except Exception:
   1954     self.close()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\io\parsers\c_parser_wrapper.py:238, in CParserWrapper.read(self, nrows)
    236 try:
    237     if self.low_memory:
--> 238         chunks = self._reader.read_low_memory(nrows)
    239         # destructive to chunks
    240         data = _concatenate_chunks(
    241             chunks, self._low_memory_column_labels(chunks)
    242         )

File pandas/_libs/parsers.pyx:832, in pandas._libs.parsers.TextReader.read_low_memory()

File pandas/_libs/parsers.pyx:897, in pandas._libs.parsers.TextReader._read_rows()

File pandas/_libs/parsers.pyx:868, in pandas._libs.parsers.TextReader._tokenize_rows()

File pandas/_libs/parsers.pyx:885, in pandas._libs.parsers.TextReader._check_tokenize_status()

File pandas/_libs/parsers.pyx:2090, in pandas._libs.parsers.raise_parser_error()

ParserError: Error tokenizing data. C error: Expected 4 fields in line 6, saw 5
baskets = [line.split(",") for line in lines]
print(baskets[0])
print(max(len(b) for b in baskets))
['citrus fruit', 'semi-finished bread', 'margarine', 'ready soups']
32
repeats = sum(len(b) != len(set(b)) for b in baskets)
print(repeats)
0
rows = [(i, item) for i, basket in enumerate(baskets)
        for item in basket]
long = pd.DataFrame(rows, columns=["basket", "item"])
print(long.shape)
long.head()
(43367, 2)
basket item
0 0 citrus fruit
1 0 semi-finished bread
2 0 margarine
3 0 ready soups
4 1 tropical fruit
import os

long.to_csv("m05_baskets_long.csv", index=False)
check = pd.read_csv("m05_baskets_long.csv")
print(check.shape, check.columns.to_list())
(43367, 2) ['basket', 'item']

Profile the baskets: how much do people buy?#

Problem: know what a typical shopping trip looks like.

sizes = long.groupby("basket").size()
sizes.describe().round(2)
count    9835.00
mean        4.41
std         3.59
min         1.00
25%         2.00
50%         3.00
75%         6.00
max        32.00
dtype: float64

Quick check: what share of baskets hold just one item?

print(round((sizes == 1).mean(), 3))
0.22
sizes.value_counts().sort_index().head(8)
1    2159
2    1643
3    1299
4    1005
5     855
6     645
7     545
8     438
Name: count, dtype: int64
biggest = sizes.idxmax()
print(biggest, sizes[biggest])
print(baskets[biggest][:6])
1216 32
['frankfurter', 'sausage', 'liver loaf', 'ham', 'chicken', 'beef']
import matplotlib.pyplot as plt

ax = sizes.plot(kind="hist", bins=range(1, 34), figsize=(7, 4),
                edgecolor="white")
ax.set_title("Most grocery baskets are small")
ax.set_xlabel("Items in basket")
ax.set_ylabel("Number of baskets")
plt.show()
No description has been provided for this image

Your turn: what share of baskets have 10 or more items?

big = sizes >= 10
print(big.sum(), round(big.mean(), 3))
896 0.091

Profile the items: popularity and support#

Problem: find the best sellers and the long tail of rare items.

print(long["item"].nunique())
long["item"].value_counts().head(5)
169
item
whole milk          2513
other vegetables    1903
rolls/buns          1809
soda                1715
yogurt              1372
Name: count, dtype: int64
  • Support = share of baskets that contain an item
  • Support of 0.25 means one basket in four
n_baskets = len(baskets)
per_basket = long.drop_duplicates()["item"].value_counts()
support = per_basket / n_baskets
support.head(8).round(3)
item
whole milk          0.256
other vegetables    0.193
rolls/buns          0.184
soda                0.174
yogurt              0.140
bottled water       0.111
root vegetables     0.109
tropical fruit      0.105
Name: count, dtype: float64

Common mistake: trusting that text values are clean

support["cream cheese"]
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\indexes\base.py:3641, in Index.get_loc(self, key)
   3640 try:
-> 3641     return self._engine.get_loc(casted_key)
   3642 except KeyError as err:

File pandas/_libs/index.pyx:168, in pandas._libs.index.IndexEngine.get_loc()

File pandas/_libs/index.pyx:197, in pandas._libs.index.IndexEngine.get_loc()

File pandas/_libs/hashtable_class_helper.pxi:7668, in pandas._libs.hashtable.PyObjectHashTable.get_item()

File pandas/_libs/hashtable_class_helper.pxi:7676, in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'cream cheese'

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
Cell In[16], line 1
----> 1 support["cream cheese"]

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\series.py:959, in Series.__getitem__(self, key)
    954     key = unpack_1tuple(key)
    956 elif key_is_scalar:
    957     # Note: GH#50617 in 3.0 we changed int key to always be treated as
    958     #  a label, matching DataFrame behavior.
--> 959     return self._get_value(key)
    961 # Convert generator to list before going through hashable part
    962 # (We will iterate through the generator there to check for slices)
    963 if is_iterator(key):

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\series.py:1046, in Series._get_value(self, label, takeable)
   1043     return self._values[label]
   1045 # Similar to Index.get_value, but we do not fall back to positional
-> 1046 loc = self.index.get_loc(label)
   1048 if is_integer(loc):
   1049     return self._values[loc]

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\indexes\base.py:3648, in Index.get_loc(self, key)
   3643     if isinstance(casted_key, slice) or (
   3644         isinstance(casted_key, abc.Iterable)
   3645         and any(isinstance(x, slice) for x in casted_key)
   3646     ):
   3647         raise InvalidIndexError(key) from err
-> 3648     raise KeyError(key) from err
   3649 except TypeError:
   3650     # If we have a listlike key, _check_indexing_error will raise
   3651     #  InvalidIndexError. Otherwise we fall through and re-raise
   3652     #  the TypeError.
   3653     self._check_indexing_error(key)

KeyError: 'cream cheese'
print([s for s in support.index if s != s.strip()])
long["item"] = long["item"].str.strip()
per_basket = long.drop_duplicates()["item"].value_counts()
support = per_basket / n_baskets
print(round(support["cream cheese"], 3))
['cream cheese ', 'roll products ']
0.04
support.describe().round(3)
count    169.000
mean       0.026
std        0.039
min        0.000
25%        0.004
50%        0.010
75%        0.031
max        0.256
Name: count, dtype: float64
rare = support < 0.01
print(rare.sum(), len(support))
print(support.tail(3).round(5))
81 169
item
preservation products    0.0002
baby food                0.0001
sound storage medium     0.0001
Name: count, dtype: float64
shares = long["item"].value_counts(normalize=True)
print(round(shares.head(20).sum(), 3))
0.504
ax = support.head(10).sort_values().plot(kind="barh",
                                         figsize=(7, 4))
ax.set_title("Top 10 grocery items by support")
ax.set_xlabel("Share of baskets containing the item")
ax.set_ylabel("Item")
plt.show()
No description has been provided for this image

Your turn: find the support of beef and its rank among all items.

rank = support.rank(ascending=False, method="min")
print(round(support["beef"], 3), int(rank["beef"]))
0.052 27

Find items that go together: confidence and lift#

Problem: tell a real link from two items that are simply popular.

onehot = pd.crosstab(long["basket"], long["item"]) > 0
print(onehot.shape)
onehot[["whole milk", "yogurt", "soda"]].head(3)
(9835, 169)
item whole milk yogurt soda
basket
0 False False False
1 False True False
2 True False False
both = (onehot["yogurt"] & onehot["whole milk"]).mean()
conf = both / support["yogurt"]
lift = conf / support["whole milk"]
print(round(both, 3), round(conf, 3), round(lift, 2))
0.056 0.402 1.57
def pair_stats(a, b):
    """Support, confidence and lift for the rule a -> b."""
    both = (onehot[a] & onehot[b]).mean()
    conf = both / support[a]
    lift = conf / support[b]
    return round(float(both), 3), round(float(conf), 3), round(lift, 2)

print(*pair_stats("root vegetables", "other vegetables"))
0.047 0.435 2.25
print(*pair_stats("whole milk", "yogurt"))
print(*pair_stats("yogurt", "whole milk"))
0.056 0.219 1.57
0.056 0.402 1.57

Quick check: soda is the 4th most popular item. Does soda make whole milk more likely?

print(*pair_stats("soda", "whole milk"))
0.04 0.23 0.9

Your turn: test the rule bottled beer -> whole milk.

print(*pair_stats("bottled beer", "whole milk"))
0.02 0.254 0.99
top = support.head(8).index
m = onehot[top].astype(int)
pair_support = (m.T @ m) / n_baskets
expected = support[top].to_frame() @ support[top].to_frame().T
lift_table = (pair_support / expected).round(2)
lift_table.iloc[:4, :4]
item whole milk other vegetables rolls/buns soda
item
whole milk 3.91 1.51 1.21 0.90
other vegetables 1.51 5.17 1.20 0.97
rolls/buns 1.21 1.20 5.44 1.20
soda 0.90 0.97 1.20 5.73
import numpy as np
import seaborn as sns

lift_table = lift_table.mask(np.eye(len(top), dtype=bool))
fig, ax = plt.subplots(figsize=(8, 6))
sns.heatmap(lift_table, annot=True, cmap="RdBu_r", center=1, ax=ax)
ax.set_title("Lift between the 8 best-selling items")
ax.set_xlabel("Then item")
ax.set_ylabel("If item")
plt.show()
No description has been provided for this image

Check a pattern across segments#

Problem: an overall rate can hide very different groups.

Common mistake: a label list that doesn't match the bin edges

pd.cut(sizes, bins=[0, 1, 3, 6, 32], labels=["1", "2-3", "4+"])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[31], line 1
----> 1 pd.cut(sizes, bins=[0, 1, 3, 6, 32], labels=["1", "2-3", "4+"])

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\reshape\tile.py:281, in cut(x, bins, right, labels, retbins, precision, include_lowest, duplicates, ordered)
    278     if not bins.is_monotonic_increasing:
    279         raise ValueError("bins must increase monotonically.")
--> 281 fac, bins = _bins_to_cuts(
    282     x_idx,
    283     bins,
    284     right=right,
    285     labels=labels,
    286     precision=precision,
    287     include_lowest=include_lowest,
    288     duplicates=duplicates,
    289     ordered=ordered,
    290 )
    292 return _postprocess_for_cut(fac, bins, retbins, original)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\reshape\tile.py:536, in _bins_to_cuts(x_idx, bins, right, labels, precision, include_lowest, duplicates, ordered)
    531     raise ValueError(
    532         "labels must be unique if ordered=True; pass ordered=False "
    533         "for duplicate labels"
    534     )
    535 elif len(labels) != len(bins) - 1:
--> 536     raise ValueError(
    537         "Bin labels must be one fewer than the number of bin edges"
    538     )
    540 if not isinstance(getattr(labels, "dtype", None), CategoricalDtype):
    541     labels = Categorical(
    542         labels,
    543         categories=labels if len(set(labels)) == len(labels) else None,
    544         ordered=ordered,
    545     )

ValueError: Bin labels must be one fewer than the number of bin edges
bands = pd.cut(sizes, bins=[0, 1, 3, 6, 32],
               labels=["1", "2-3", "4-6", "7+"])
milk = onehot["whole milk"].groupby(bands, observed=True).mean()
print(bands.value_counts().sort_index())
print(milk.round(3))
1      2159
2-3    2942
4-6    2505
7+     2229
Name: count, dtype: int64
1      0.056
2-3    0.155
4-6    0.323
7+     0.505
Name: whole milk, dtype: float64
ax = milk.plot(kind="bar", rot=0, figsize=(7, 4))
ax.set_title("Bigger baskets are far more likely to hold milk")
ax.set_xlabel("Items in basket")
ax.set_ylabel("Share of baskets with whole milk")
plt.show()
No description has been provided for this image
for band in ["2-3", "4-6", "7+"]:
    seg = onehot[bands == band]
    both = (seg["yogurt"] & seg["whole milk"]).mean()
    exp = seg["yogurt"].mean() * seg["whole milk"].mean()
    print(band, round(both / exp, 2))
2-3 0.8
4-6 0.94
7+ 1.11

Your turn: repeat the band comparison for bottled water.

water = onehot["bottled water"].groupby(bands, observed=True)
print(water.mean().round(3))
1      0.031
2-3    0.077
4-6    0.132
7+     0.208
Name: bottled water, dtype: float64

Mini-project: what should sit next to the vegetables?#

Let Apriori search every combination and rank the rules by lift.

from mlxtend.frequent_patterns import apriori, association_rules

freq = apriori(onehot, min_support=0.02, use_colnames=True)
print(len(freq))
freq.sort_values("support", ascending=False).head(3).round(3)
122
support itemsets
57 0.256 frozenset({whole milk})
39 0.193 frozenset({other vegetables})
43 0.184 frozenset({rolls/buns})
rules = association_rules(freq, metric="lift", min_threshold=1.5)
rules["if"] = rules["antecedents"].apply(sorted).str.join(", ")
rules["then"] = rules["consequents"].apply(sorted).str.join(", ")
rules = rules.round(3)
print(len(rules))
76
sure = rules[rules["confidence"] >= 0.4]
print(len(sure))
sure["then"].value_counts()
15
then
whole milk          12
other vegetables     3
Name: count, dtype: int64
cols = ["if", "then", "support", "confidence", "lift"]
veg = rules[rules["then"] == "root vegetables"]
veg.sort_values(["lift", "if"], ascending=[False, True])[cols]
if then support confidence lift
65 other vegetables, whole milk root vegetables 0.023 0.310 2.842
30 other vegetables root vegetables 0.047 0.245 2.247
48 tropical fruit root vegetables 0.021 0.201 1.840
51 whole milk root vegetables 0.049 0.191 1.756
52 yogurt root vegetables 0.026 0.185 1.698
best = rules.sort_values(["lift", "if"], ascending=[False, True])
top_rule = best.iloc[0]
print(f"If {top_rule['if']} then {top_rule['then']}:")
print(f"lift {top_rule['lift']}, confidence "
      f"{top_rule['confidence']:.0%}")
If other vegetables, whole milk then root vegetables:
lift 2.842, confidence 31%
fig, ax = plt.subplots(figsize=(7, 4))
pts = ax.scatter(rules["support"], rules["confidence"],
                 c=rules["lift"], cmap="viridis")
fig.colorbar(pts, ax=ax, label="Lift")
ax.set_title("76 grocery rules: support vs confidence")
ax.set_xlabel("Support (share of baskets)")
ax.set_ylabel("Confidence")
plt.show()
No description has been provided for this image

Recap#

  • Ragged transactions: split lines, then build a long table
  • Profile first: basket sizes, item support, the long tail
  • Clean text labels before looking them up
  • Confidence alone misleads; lift above 1 means a real link
  • Check patterns per segment, then let Apriori search at scale

Challenges#

  1. Find the three items with the highest lift alongside yogurt.
  2. Rerun Apriori with min_support=0.01. How many rules now?
  3. Compare the shopping bags rate across basket size bands.

What's next#

Lesson 06: Summarization and Descriptive Statistics Mean, median, spread and shape, on real customer data.

Data Mining 06: Summarization and Descriptive Statistics#

In this lesson you'll be able to:

  • Choose between mean, median and mode for the centre
  • Measure spread with range, standard deviation and quartiles
  • Describe shape with skew and flag outliers with the IQR rule
  • Summarise categories with counts, shares and crosstabs
  • Build group summaries that profile customers Datasets: IBM Telco customer churn (data/mining/telco_churn.csv) and grocery baskets (data/mining/groceries_baskets.csv)

Warm-up: basket sizes from lesson 05#

Lesson 05 explored grocery baskets. One number there deserves a second look.

import pandas as pd

with open("groceries_baskets.csv") as f:
    lines = f.read().splitlines()
sizes = pd.Series([len(line.split(",")) for line in lines])
print(round(sizes.mean(), 2), sizes.median())
4.41 3.0

Load the data and check types before summarising#

Problem: statistics only work on columns that are really numbers.

df = pd.read_csv("telco_churn.csv")
print(df.shape)
cols = ["tenure", "MonthlyCharges", "TotalCharges", "Churn"]
df[cols].head()
(7043, 21)
tenure MonthlyCharges TotalCharges Churn
0 1 29.85 29.85 No
1 34 56.95 1889.5 No
2 2 53.85 108.15 Yes
3 45 42.30 1840.75 No
4 2 70.70 151.65 Yes
df[cols].dtypes
tenure              int64
MonthlyCharges    float64
TotalCharges          str
Churn                 str
dtype: object

Common mistake: averaging a column that only looks numeric

df["TotalCharges"].mean()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[5], line 1
----> 1 df["TotalCharges"].mean()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\util\_decorators.py:336, in deprecate_nonkeyword_arguments.<locals>.decorate.<locals>.wrapper(*args, **kwargs)
    330 if len(args) > num_allow_args:
    331     warnings.warn(
    332         msg.format(arguments=_format_argument_list(allow_args)),
    333         klass,
    334         stacklevel=find_stack_level(),
    335     )
--> 336 return func(*args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\series.py:8113, in Series.mean(self, axis, skipna, numeric_only, **kwargs)
   8063 @deprecate_nonkeyword_arguments(Pandas4Warning, allowed_args=["self"], name="mean")
   8064 def mean(
   8065     self,
   (...)
   8069     **kwargs,
   8070 ) -> Any:
   8071     """
   8072     Return the mean of the values over the requested axis.
   8073 
   (...)
   8111     2.0
   8112     """
-> 8113     return NDFrame.mean(
   8114         self, axis=axis, skipna=skipna, numeric_only=numeric_only, **kwargs
   8115     )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\generic.py:11819, in NDFrame.mean(self, axis, skipna, numeric_only, **kwargs)
  11811 def mean(
  11812     self,
  11813     *,
   (...)
  11817     **kwargs,
  11818 ) -> Series | float:
> 11819     return self._stat_function(
  11820         "mean", nanops.nanmean, axis, skipna, numeric_only, **kwargs
  11821     )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\generic.py:11773, in NDFrame._stat_function(self, name, func, axis, skipna, numeric_only, **kwargs)
  11769 nv.validate_func(name, (), kwargs)
  11771 validate_bool_kwarg(skipna, "skipna", none_allowed=False)
> 11773 return self._reduce(
  11774     func, name=name, axis=axis, skipna=skipna, numeric_only=numeric_only
  11775 )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\series.py:7480, in Series._reduce(self, op, name, axis, skipna, numeric_only, filter_type, **kwds)
   7476     self._get_axis_number(axis)
   7478 if isinstance(delegate, ExtensionArray):
   7479     # dispatch to ExtensionArray interface
-> 7480     result = delegate._reduce(name, skipna=skipna, **kwds)
   7482 else:
   7483     # dispatch to numpy arrays
   7484     if numeric_only and self.dtype.kind not in "iufcb":
   7485         # i.e. not is_numeric_dtype(self.dtype)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\string_arrow.py:564, in ArrowStringArray._reduce(self, name, skipna, keepdims, **kwargs)
    562     result = self._reduce_calc(name, skipna=skipna, keepdims=keepdims, **kwargs)
    563 else:
--> 564     raise TypeError(f"Cannot perform reduction '{name}' with string dtype")
    566 if name in ("argmin", "argmax") and isinstance(result, pa.Array):
    567     return self._convert_int_result(result)

TypeError: Cannot perform reduction 'mean' with string dtype
df["TotalCharges"] = pd.to_numeric(df["TotalCharges"],
                                   errors="coerce")
gaps = df[df["TotalCharges"].isna()]
print(len(gaps))
print(gaps["tenure"].unique())
11
[0]
print(round(df["TotalCharges"].mean(), 2))
print(df["TotalCharges"].count(), len(df))
2283.3
7032 7043
print(df.select_dtypes("number").columns.to_list())
['SeniorCitizen', 'tenure', 'MonthlyCharges', 'TotalCharges']

Measure the centre: mean, median and mode#

Problem: one number for a typical value, without being fooled.

  • Mean: the total shared out equally; pulled by extremes
  • Median: the middle value once sorted; resists extremes
  • Mode: the most common value; works for categories too
bill = df["MonthlyCharges"]
print(round(bill.mean(), 2), bill.median())
64.76 70.35
print(df["tenure"].mode().to_list())
df["tenure"].value_counts().head(3)
[1]
tenure
1     613
72    362
2     238
Name: count, dtype: int64

Quick check: for TotalCharges, will the mean be above or below the median?

total = df["TotalCharges"]
print(round(total.mean(), 2), round(total.median(), 2))
2283.3 1397.48
from scipy import stats

trimmed = stats.trim_mean(total.dropna(), 0.1)
print(round(trimmed, 2))
1970.14
cut = sizes[sizes <= sizes.quantile(0.99)]
print(len(sizes) - len(cut))
print(round(cut.mean(), 2), cut.median())
95
4.26 3.0
print(df["PaymentMethod"].mode()[0])
Electronic check

Measure the spread: range, standard deviation, quartiles#

Problem: two groups can share an average yet behave very differently.

print(bill.min(), bill.max())
print(round(bill.max() - bill.min(), 2))
18.25 118.75
100.5
print(round(bill.std(), 2), round(bill.var(), 2))
30.09 905.41
import numpy as np

print(round(bill.std(), 4), round(np.std(bill), 4))
30.09 30.0879
q = bill.quantile([0.1, 0.25, 0.5, 0.75, 0.9])
print(q.round(2))
print(round(q[0.75] - q[0.25], 2))
0.10     20.05
0.25     35.50
0.50     70.35
0.75     89.85
0.90    102.60
Name: MonthlyCharges, dtype: float64
54.35
num = ["tenure", "MonthlyCharges", "TotalCharges"]
cv = df[num].std() / df[num].mean()
cv.round(2)
tenure            0.76
MonthlyCharges    0.46
TotalCharges      0.99
dtype: float64

Your turn: find the interquartile range of tenure.

q1, q3 = df["tenure"].quantile([0.25, 0.75])
print(q1, q3, q3 - q1)
9.0 55.0 46.0
df.groupby("Churn")["MonthlyCharges"].agg(["mean", "std"]).round(2)
mean std
Churn
No 61.27 31.09
Yes 74.44 24.67
df[num].describe().round(2)
tenure MonthlyCharges TotalCharges
count 7043.00 7043.00 7032.00
mean 32.37 64.76 2283.30
std 24.56 30.09 2266.77
min 0.00 18.25 18.80
25% 9.00 35.50 401.45
50% 29.00 70.35 1397.48
75% 55.00 89.85 3794.74
max 72.00 118.75 8684.80

Describe shape and flag outliers#

Problem: know whether data is lopsided, and which values are extreme.

print(df[num].skew().round(2))
print(round(sizes.skew(), 2))
tenure            0.24
MonthlyCharges   -0.22
TotalCharges      0.96
dtype: float64
1.64
import matplotlib.pyplot as plt

ax = df["tenure"].plot(kind="hist", bins=36, figsize=(7, 4),
                       edgecolor="white")
ax.set_title("Customer tenure piles up at both ends")
ax.set_xlabel("Tenure (months)")
ax.set_ylabel("Number of customers")
plt.show()
No description has been provided for this image
ax = bill.plot(kind="hist", bins=30, figsize=(7, 4),
               edgecolor="white")
ax.set_title("Monthly bills: a cheap cluster and a broad hump")
ax.set_xlabel("Monthly charge")
ax.set_ylabel("Number of customers")
plt.show()
No description has been provided for this image
  • IQR rule: outliers lie below Q1 - 1.5 x IQR or above Q3 + 1.5 x IQR
def iqr_outliers(s):
    """Return the values outside the 1.5 x IQR fences."""
    q1, q3 = s.quantile([0.25, 0.75])
    low, high = q1 - 1.5 * (q3 - q1), q3 + 1.5 * (q3 - q1)
    return s[(s < low) | (s > high)]

out = iqr_outliers(sizes)
print(len(out), out.min())
351 13

Quick check: how many telco customers will the IQR rule flag in each numeric column?

for col in num:
    print(col, len(iqr_outliers(df[col])))
tenure 0
MonthlyCharges 0
TotalCharges 0
z = (bill - bill.mean()) / bill.std()
print(round(z.min(), 2), round(z.max(), 2))
print((z.abs() > 2).sum())
-1.55 1.79
0

Summarise categories with counts, shares and crosstabs#

Problem: most columns here are text, and they need summaries too.

df["Churn"].value_counts()
Churn
No     5174
Yes    1869
Name: count, dtype: int64

Common mistake: taking the mean of a Yes/No column

df["Churn"].mean()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[30], line 1
----> 1 df["Churn"].mean()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\util\_decorators.py:336, in deprecate_nonkeyword_arguments.<locals>.decorate.<locals>.wrapper(*args, **kwargs)
    330 if len(args) > num_allow_args:
    331     warnings.warn(
    332         msg.format(arguments=_format_argument_list(allow_args)),
    333         klass,
    334         stacklevel=find_stack_level(),
    335     )
--> 336 return func(*args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\series.py:8113, in Series.mean(self, axis, skipna, numeric_only, **kwargs)
   8063 @deprecate_nonkeyword_arguments(Pandas4Warning, allowed_args=["self"], name="mean")
   8064 def mean(
   8065     self,
   (...)
   8069     **kwargs,
   8070 ) -> Any:
   8071     """
   8072     Return the mean of the values over the requested axis.
   8073 
   (...)
   8111     2.0
   8112     """
-> 8113     return NDFrame.mean(
   8114         self, axis=axis, skipna=skipna, numeric_only=numeric_only, **kwargs
   8115     )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\generic.py:11819, in NDFrame.mean(self, axis, skipna, numeric_only, **kwargs)
  11811 def mean(
  11812     self,
  11813     *,
   (...)
  11817     **kwargs,
  11818 ) -> Series | float:
> 11819     return self._stat_function(
  11820         "mean", nanops.nanmean, axis, skipna, numeric_only, **kwargs
  11821     )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\generic.py:11773, in NDFrame._stat_function(self, name, func, axis, skipna, numeric_only, **kwargs)
  11769 nv.validate_func(name, (), kwargs)
  11771 validate_bool_kwarg(skipna, "skipna", none_allowed=False)
> 11773 return self._reduce(
  11774     func, name=name, axis=axis, skipna=skipna, numeric_only=numeric_only
  11775 )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\series.py:7480, in Series._reduce(self, op, name, axis, skipna, numeric_only, filter_type, **kwds)
   7476     self._get_axis_number(axis)
   7478 if isinstance(delegate, ExtensionArray):
   7479     # dispatch to ExtensionArray interface
-> 7480     result = delegate._reduce(name, skipna=skipna, **kwds)
   7482 else:
   7483     # dispatch to numpy arrays
   7484     if numeric_only and self.dtype.kind not in "iufcb":
   7485         # i.e. not is_numeric_dtype(self.dtype)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\string_arrow.py:564, in ArrowStringArray._reduce(self, name, skipna, keepdims, **kwargs)
    562     result = self._reduce_calc(name, skipna=skipna, keepdims=keepdims, **kwargs)
    563 else:
--> 564     raise TypeError(f"Cannot perform reduction '{name}' with string dtype")
    566 if name in ("argmin", "argmax") and isinstance(result, pa.Array):
    567     return self._convert_int_result(result)

TypeError: Cannot perform reduction 'mean' with string dtype
df["churned"] = (df["Churn"] == "Yes").astype(int)
print(round(df["churned"].mean(), 3))
print(df["Churn"].value_counts(normalize=True).round(3))
0.265
Churn
No     0.735
Yes    0.265
Name: proportion, dtype: float64
print(df["SeniorCitizen"].value_counts())
print(round(df["SeniorCitizen"].mean(), 3))
SeniorCitizen
0    5901
1    1142
Name: count, dtype: int64
0.162
df.groupby("gender")["churned"].agg(["size", "mean"]).round(3)
size mean
gender
Female 3488 0.269
Male 3555 0.262
df["Contract"].value_counts()
Contract
Month-to-month    3875
Two year          1695
One year          1473
Name: count, dtype: int64
pd.crosstab(df["Contract"], df["Churn"], normalize="index").round(3)
Churn No Yes
Contract
Month-to-month 0.573 0.427
One year 0.887 0.113
Two year 0.972 0.028
pd.crosstab(df["Contract"], df["Churn"], margins=True)
Churn No Yes All
Contract
Month-to-month 2220 1655 3875
One year 1307 166 1473
Two year 1647 48 1695
All 5174 1869 7043

Your turn: find the churn rate for each payment method.

pay = df.groupby("PaymentMethod")["churned"].mean()
pay.sort_values(ascending=False).round(3)
PaymentMethod
Electronic check             0.453
Mailed check                 0.191
Bank transfer (automatic)    0.167
Credit card (automatic)      0.152
Name: churned, dtype: float64

Summarise by group with groupby and agg#

Problem: compare whole profiles across segments in one table.

internet = df.groupby("InternetService")["MonthlyCharges"]
internet.agg(["count", "mean", "median", "std"]).round(2)
count mean median std
InternetService
DSL 2421 58.10 56.15 16.26
Fiber optic 3096 91.50 91.68 12.66
No 1526 21.08 20.15 2.16
df.groupby("Churn")["tenure"].describe().round(1)
count mean std min 25% 50% 75% max
Churn
No 5174.0 37.6 24.1 0.0 15.0 38.0 61.0 72.0
Yes 1869.0 18.0 19.5 1.0 2.0 10.0 29.0 72.0

Quick check: churners pay more each month. Do they also have bigger total charges?

df.groupby("Churn")[num].median().round(1)
tenure MonthlyCharges TotalCharges
Churn
No 38.0 64.4 1683.6
Yes 10.0 79.6 703.6
rates = df.pivot_table(index="Contract", columns="InternetService",
                       values="churned", aggfunc="mean")
rates.round(3)
InternetService DSL Fiber optic No
Contract
Month-to-month 0.322 0.546 0.189
One year 0.093 0.193 0.025
Two year 0.019 0.072 0.008

Mini-project: profile the customers who leave#

Build one summary table a retention team could act on.

df["monthly"] = df["Contract"] == "Month-to-month"
df["fibre"] = df["InternetService"] == "Fiber optic"
df["e_check"] = df["PaymentMethod"] == "Electronic check"
print(df[["monthly", "fibre", "e_check"]].mean().round(3))
monthly    0.550
fibre      0.440
e_check    0.336
dtype: float64
profile = df.groupby("Churn").agg(
    customers=("churned", "size"),
    median_tenure=("tenure", "median"),
    median_bill=("MonthlyCharges", "median"),
    month_to_month=("monthly", "mean"),
    fibre=("fibre", "mean"),
    e_check=("e_check", "mean"),
)
profile.round(2)
customers median_tenure median_bill month_to_month fibre e_check
Churn
No 5174 38.0 64.43 0.43 0.35 0.25
Yes 1869 10.0 79.65 0.89 0.69 0.57
import os

profile.round(3).to_csv("m06_churn_profile.csv")
saved = pd.read_csv("m06_churn_profile.csv")
print(saved.shape)
(2, 7)
fig, ax = plt.subplots(figsize=(7, 4))
df.boxplot(column="tenure", by="Churn", ax=ax)
ax.set_title("Leavers are mostly new customers")
fig.suptitle("")
ax.set_xlabel("Churned")
ax.set_ylabel("Tenure (months)")
plt.show()
No description has been provided for this image
new = df[df["tenure"] <= 12]
print(len(new), round(new["churned"].mean(), 3))
print(round(df[df["tenure"] > 12]["churned"].mean(), 3))
2186 0.474
0.171

Recap#

  • Check dtypes first; convert text numbers with to_numeric
  • Mean vs median: a big gap signals skew and extremes
  • Spread: std, quartiles and IQR; CV compares across units
  • Categories: value_counts, normalize, and crosstab
  • groupby with agg turns statistics into customer profiles

Challenges#

  1. Compare mean and median MonthlyCharges for senior citizens vs others.
  2. Build a crosstab of InternetService and Churn with row shares.
  3. Use the IQR rule on TotalCharges for churners only.

What's next#

Lesson 07: Visualization - Histograms, Scatterplots, Heatmaps Turn these summaries into charts that reveal patterns at a glance.

Data Mining 07: Visualization- Histograms, Scatterplots, Heatmaps#

In this lesson you'll be able to:

  • Read the shape of a column with a histogram, and pick good bins
  • Tame skewed columns with a log scale and compare groups fairly
  • Show how two columns move together with a scatterplot
  • Turn a scatterplot into a map, coloured by a third column
  • Summarise many numbers at once with a heatmap Dataset: California housing, 1990 census (data/pandas/housing.csv)

Warm-up: numbers before pictures#

Lesson 06 summarised columns with statistics. Same idea, new data.

import pandas as pd
df = pd.read_csv("housing.csv")
print(df.shape)
df[["median_income", "median_house_value", "ocean_proximity"]].head()
(20640, 10)
median_income median_house_value ocean_proximity
0 8.3252 452600.0 NEAR BAY
1 8.3014 358500.0 NEAR BAY
2 7.2574 352100.0 NEAR BAY
3 5.6431 341300.0 NEAR BAY
4 3.8462 342200.0 NEAR BAY
cols = ["median_income", "median_house_value"]
df[cols].describe().round(2)
median_income median_house_value
count 20640.00 20640.00
mean 3.87 206855.82
std 1.90 115395.62
min 0.50 14999.00
25% 2.56 119600.00
50% 3.53 179700.00
75% 4.74 264725.00
max 15.00 500001.00

Histograms: see the shape of one column#

Problem: a mean and a median hide the shape. Where do values pile up?

  • Cut the range into equal-width bins
  • Draw one bar per bin; its height is how many rows fall inside
import matplotlib.pyplot as plt
ax = df["median_income"].plot(kind="hist", bins=30,
                               edgecolor="white", figsize=(7, 4))
ax.set_title("Income across California districts")
ax.set_xlabel("Median income (tens of thousands of dollars)")
ax.set_ylabel("Number of districts")
plt.show()
No description has been provided for this image
fig, axes = plt.subplots(1, 2, figsize=(10, 4))
for ax, n in zip(axes, [5, 80]):
    df["median_income"].plot(kind="hist", bins=n, ax=ax)
    ax.set_title(f"{n} bins")
    ax.set_xlabel("Median income")
    ax.set_ylabel("Districts")
plt.tight_layout()
No description has been provided for this image
import numpy as np
counts, edges = np.histogram(df["median_income"], bins=5)
print(counts)
print(edges.round(2))
[9683 9088 1488  271  110]
[ 0.5  3.4  6.3  9.2 12.1 15. ]
ax = df["median_house_value"].plot(kind="hist", bins=50,
                                    edgecolor="white", figsize=(7, 4))
ax.set_title("Median home value per district")
ax.set_xlabel("Median home value (dollars)")
ax.set_ylabel("Number of districts")
No description has been provided for this image

Quick check: what is the tall bar at the far right of that chart?

top = df["median_house_value"].max()
print(top)
print((df["median_house_value"] == top).sum())
500001.0
965

Your turn#

Draw a histogram of housing_median_age with 52 bins. Is it capped too?

ax = df["housing_median_age"].plot(kind="hist", bins=52,
                                    figsize=(7, 4))
ax.set_title("Median age of homes per district")
ax.set_xlabel("Median age of homes (years)")
ax.set_ylabel("Number of districts")
print(df["housing_median_age"].value_counts().head(3))
housing_median_age
52.0    1273
36.0     862
35.0     824
Name: count, dtype: int64
No description has been provided for this image

Skewed columns and the log scale#

Problem: a few huge districts squash every other bar into one corner.

ax = df["population"].plot(kind="hist", bins=50, figsize=(7, 4))
ax.set_title("Population per district")
ax.set_xlabel("People living in the district")
ax.set_ylabel("Number of districts")
No description has been provided for this image
skew_cols = ["population", "total_rooms", "median_income"]
df[skew_cols].skew().round(2)
population       4.94
total_rooms      4.15
median_income    1.65
dtype: float64
import seaborn as sns
ax = sns.histplot(data=df, x="population", bins=50, log_scale=True)
ax.set_title("Population per district (log scale)")
ax.set_xlabel("People living in the district")
ax.set_ylabel("Number of districts")
No description has been provided for this image

Your turn#

Draw total_rooms on a log scale. Where is the typical district?

ax = sns.histplot(data=df, x="total_rooms", bins=50, log_scale=True)
ax.set_title("Rooms per district (log scale)")
ax.set_xlabel("Total rooms in the district")
ax.set_ylabel("Number of districts")
print(df["total_rooms"].median())
2127.0
No description has been provided for this image

Compare groups with overlapping histograms#

Problem: do inland and coastal districts have different home values?

df["ocean_proximity"].value_counts()
ocean_proximity
<1H OCEAN     9136
INLAND        6551
NEAR OCEAN    2658
NEAR BAY      2290
ISLAND           5
Name: count, dtype: int64

Common mistake: a column name with the wrong capital letter

sns.histplot(data=df, x="median_house_value", hue="Ocean_proximity")
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[15], line 1
----> 1 sns.histplot(data=df, x="median_house_value", hue="Ocean_proximity")
      2 plt.show()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\seaborn\distributions.py:1379, in histplot(data, x, y, hue, weights, stat, bins, binwidth, binrange, discrete, cumulative, common_bins, common_norm, multiple, element, fill, shrink, kde, kde_kws, line_kws, thresh, pthresh, pmax, cbar, cbar_ax, cbar_kws, palette, hue_order, hue_norm, color, log_scale, legend, ax, **kwargs)
   1358 def histplot(
   1359     data=None, *,
   1360     # Vector variables
   (...)
   1376     **kwargs,
   1377 ):
-> 1379     p = _DistributionPlotter(
   1380         data=data,
   1381         variables=dict(x=x, y=y, hue=hue, weights=weights),
   1382     )
   1384     p.map_hue(palette=palette, order=hue_order, norm=hue_norm)
   1386     if ax is None:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\seaborn\distributions.py:110, in _DistributionPlotter.__init__(self, data, variables)
    104 def __init__(
    105     self,
    106     data=None,
    107     variables={},
    108 ):
--> 110     super().__init__(data=data, variables=variables)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\seaborn\_base.py:634, in VectorPlotter.__init__(self, data, variables)
    629 # var_ordered is relevant only for categorical axis variables, and may
    630 # be better handled by an internal axis information object that tracks
    631 # such information and is set up by the scale_* methods. The analogous
    632 # information for numeric axes would be information about log scales.
    633 self._var_ordered = {"x": False, "y": False}  # alt., used DefaultDict
--> 634 self.assign_variables(data, variables)
    636 # TODO Lots of tests assume that these are called to initialize the
    637 # mappings to default values on class initialization. I'd prefer to
    638 # move away from that and only have a mapping when explicitly called.
    639 for var in ["hue", "size", "style"]:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\seaborn\_base.py:679, in VectorPlotter.assign_variables(self, data, variables)
    674 else:
    675     # When dealing with long-form input, use the newer PlotData
    676     # object (internal but introduced for the objects interface)
    677     # to centralize / standardize data consumption logic.
    678     self.input_format = "long"
--> 679     plot_data = PlotData(data, variables)
    680     frame = plot_data.frame
    681     names = plot_data.names

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\seaborn\_core\data.py:58, in PlotData.__init__(self, data, variables)
     51 def __init__(
     52     self,
     53     data: DataSource,
     54     variables: dict[str, VariableSpec],
     55 ):
     57     data = handle_data_source(data)
---> 58     frame, names, ids = self._assign_variables(data, variables)
     60     self.frame = frame
     61     self.names = names

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\seaborn\_core\data.py:232, in PlotData._assign_variables(self, data, variables)
    230     else:
    231         err += "An entry with this name does not appear in `data`."
--> 232     raise ValueError(err)
    234 else:
    235 
    236     # Otherwise, assume the value somehow represents data
    237 
    238     # Ignore empty data structures
    239     if isinstance(val, Sized) and len(val) == 0:

ValueError: Could not interpret value `Ocean_proximity` for `hue`. An entry with this name does not appear in `data`.
two = df[df["ocean_proximity"].isin(["INLAND", "<1H OCEAN"])]
ax = sns.histplot(data=two, x="median_house_value",
                  hue="ocean_proximity", stat="density",
                  common_norm=False, element="step")
ax.set_title("Home values: inland vs within an hour of the ocean")
ax.set_xlabel("Median home value (dollars)")
ax.set_ylabel("Density")
No description has been provided for this image
by_loc = df.groupby("ocean_proximity")["median_house_value"]
by_loc.median().sort_values()
ocean_proximity
INLAND        108500.0
<1H OCEAN     214850.0
NEAR OCEAN    229450.0
NEAR BAY      233800.0
ISLAND        414700.0
Name: median_house_value, dtype: float64

Scatterplots: how two columns move together#

Problem: do richer districts have pricier homes, and how tightly?

ax = df.plot(kind="scatter", x="median_income",
             y="median_house_value", figsize=(7, 4))
ax.set_title("Income vs home value, every district")
ax.set_xlabel("Median income (tens of thousands of dollars)")
ax.set_ylabel("Median home value (dollars)")
No description has been provided for this image
ax = df.plot(kind="scatter", x="median_income",
             y="median_house_value", alpha=0.1, s=5,
             figsize=(7, 4))
ax.set_title("Income vs home value, with transparency")
ax.set_xlabel("Median income (tens of thousands of dollars)")
ax.set_ylabel("Median home value (dollars)")
No description has been provided for this image
ax = df.plot(kind="hexbin", x="median_income",
             y="median_house_value", gridsize=30,
             cmap="Blues", figsize=(7, 4))
ax.set_title("Income vs home value, counted in hexagons")
ax.set_xlabel("Median income (tens of thousands of dollars)")
ax.set_ylabel("Median home value (dollars)")
No description has been provided for this image
r = df["median_income"].corr(df["median_house_value"])
print(round(r, 2))
0.69

Quick check: what is the flat row of dots along the top of the chart?

rich = df[df["median_income"] >= 8]
capped = (rich["median_house_value"] == top).mean()
print(len(rich), round(capped, 2))
691 0.64

Common mistake: x and y with different lengths

plt.scatter(df["median_income"], df["median_house_value"].head(500))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[23], line 1
----> 1 plt.scatter(df["median_income"], df["median_house_value"].head(500))
      2 plt.show()

    [... skipping hidden 1 frame]

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\matplotlib\pyplot.py:4128, in scatter(x, y, s, c, marker, cmap, norm, vmin, vmax, alpha, linewidths, edgecolors, colorizer, plotnonfinite, data, **kwargs)
   4108 @_copy_docstring_and_deprecators(Axes.scatter)
   4109 def scatter(
   4110     x: float | ArrayLike,
   (...)
   4126     **kwargs,
   4127 ) -> PathCollection:
-> 4128     __ret = gca().scatter(
   4129         x,
   4130         y,
   4131         s=s,
   4132         c=c,
   4133         marker=marker,
   4134         cmap=cmap,
   4135         norm=norm,
   4136         vmin=vmin,
   4137         vmax=vmax,
   4138         alpha=alpha,
   4139         linewidths=linewidths,
   4140         edgecolors=edgecolors,
   4141         colorizer=colorizer,
   4142         plotnonfinite=plotnonfinite,
   4143         **({"data": data} if data is not None else {}),
   4144         **kwargs,
   4145     )
   4146     sci(__ret)
   4147     return __ret

    [... skipping hidden 2 frame]

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\matplotlib\axes\_axes.py:5329, in Axes.scatter(self, x, y, s, c, marker, cmap, norm, vmin, vmax, alpha, linewidths, edgecolors, colorizer, plotnonfinite, **kwargs)
   5327 y = np.ma.ravel(y)
   5328 if x.size != y.size:
-> 5329     raise ValueError("x and y must be the same size")
   5331 if s is None:
   5332     s = (20 if mpl.rcParams['_internal.classic_mode'] else
   5333          mpl.rcParams['lines.markersize'] ** 2.0)

ValueError: x and y must be the same size
No description has been provided for this image
sample = df.sample(500, random_state=42)
plt.scatter(sample["median_income"], sample["median_house_value"],
            alpha=0.5)
plt.title("Income vs home value, 500 sampled districts")
plt.xlabel("Median income (tens of thousands of dollars)")
plt.ylabel("Median home value (dollars)")
plt.show()
No description has been provided for this image

A scatterplot that is a map#

Problem: where in the state are the expensive districts?

ax = df.plot(kind="scatter", x="longitude", y="latitude",
             c="median_house_value", cmap="viridis",
             s=df["population"] / 200, alpha=0.4,
             colorbar=True, figsize=(8, 6))
ax.set_title("California districts coloured by home value")
ax.set_xlabel("Longitude")
ax.set_ylabel("Latitude")
No description has been provided for this image
dearest = df.nlargest(200, "median_house_value")
dearest["ocean_proximity"].value_counts()
ocean_proximity
<1H OCEAN     175
NEAR BAY       18
INLAND          6
NEAR OCEAN      1
Name: count, dtype: int64

Your turn#

Redraw the map with seaborn, coloured by ocean_proximity instead.

ax = sns.scatterplot(data=df, x="longitude", y="latitude",
                     hue="ocean_proximity", s=5, alpha=0.5)
ax.set_title("California districts by distance to the ocean")
ax.set_xlabel("Longitude")
ax.set_ylabel("Latitude")
No description has been provided for this image

Heatmaps: a whole table at a glance#

Problem: compare every pair of columns at once, not one scatter at a time.

Common mistake: asking for correlations with a text column inside

corr = df.corr()
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[28], line 1
----> 1 corr = df.corr()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\frame.py:13135, in DataFrame.corr(self, method, min_periods, numeric_only)
  13133 cols = data.columns
  13134 idx = cols.copy()
> 13135 mat = data.to_numpy(dtype=float, na_value=np.nan, copy=False)
  13137 if method == "pearson":
  13138     correl = libalgos.nancorr(mat, minp=min_periods)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\frame.py:2081, in DataFrame.to_numpy(self, dtype, copy, na_value)
   2079 if dtype is not None:
   2080     dtype = np.dtype(dtype)
-> 2081 result = self._mgr.as_array(dtype=dtype, copy=copy, na_value=na_value)
   2082 if result.dtype is not dtype:
   2083     result = np.asarray(result, dtype=dtype)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\managers.py:1885, in BlockManager.as_array(self, dtype, copy, na_value)
   1883         arr.flags.writeable = False
   1884 else:
-> 1885     arr = self._interleave(dtype=dtype, na_value=na_value)
   1886     # The underlying data was copied within _interleave, so no need
   1887     # to further copy if copy=True or setting na_value
   1889 if na_value is lib.no_default:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\managers.py:1938, in BlockManager._interleave(self, dtype, na_value)
   1932 rl = blk.mgr_locs
   1933 if blk.is_extension:
   1934     # Avoid implicit conversion of extension blocks to object
   1935 
   1936     # error: Item "ndarray" of "Union[ndarray, ExtensionArray]" has no
   1937     # attribute "to_numpy"
-> 1938     arr = blk.values.to_numpy(  # type: ignore[union-attr]
   1939         dtype=dtype,
   1940         na_value=na_value,
   1941     )
   1942 else:
   1943     arr = blk.get_values(dtype)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\arrow\array.py:1803, in ArrowExtensionArray.to_numpy(self, dtype, copy, na_value)
   1801 result = data._pa_array.to_numpy()
   1802 if dtype is not None:
-> 1803     result = result.astype(dtype, copy=False)
   1804 if copy:
   1805     result = result.copy()

ValueError: could not convert string to float: 'NEAR BAY'
corr = df.corr(numeric_only=True)
corr["median_house_value"].sort_values().round(2)
latitude             -0.14
longitude            -0.05
population           -0.02
total_bedrooms        0.05
households            0.07
housing_median_age    0.11
total_rooms           0.13
median_income         0.69
median_house_value    1.00
Name: median_house_value, dtype: float64
plt.figure(figsize=(9, 7))
sns.heatmap(corr, annot=True, fmt=".2f", cmap="coolwarm",
            vmin=-1, vmax=1)
plt.title("Correlation between housing columns")
plt.show()
No description has been provided for this image

Quick check: which pair of different columns is most strongly linked?

upper = np.triu(np.ones(corr.shape, dtype=bool), k=1)
pairs = corr.where(upper).stack()
pairs.sort_values(ascending=False).head(3).round(2)
total_bedrooms  households        0.98
total_rooms     total_bedrooms    0.93
                households        0.92
dtype: float64

Mini-project: does the coast add value beyond income?#

Plan: income bands, a pivot table of median values, a heatmap, a finding.

edges = [0, 2, 3, 4, 5, 6, 16]
names = ["<20k", "20-30k", "30-40k", "40-50k", "50-60k", "60k+"]
df["income_band"] = pd.cut(df["median_income"], bins=edges,
                           labels=names)
df["income_band"].value_counts(sort=False)
income_band
<20k      2457
20-30k    4946
30-40k    5150
40-50k    3598
50-60k    2127
60k+      2362
Name: count, dtype: int64
table = df.pivot_table(index="ocean_proximity", columns="income_band",
                       values="median_house_value",
                       aggfunc="median", observed=True)
table = table.drop(index="ISLAND")
(table / 1000).round(0)
income_band <20k 20-30k 30-40k 40-50k 50-60k 60k+
ocean_proximity
<1H OCEAN 128.0 163.0 192.0 218.0 257.0 366.0
INLAND 68.0 91.0 118.0 146.0 185.0 267.0
NEAR BAY 115.0 160.0 218.0 252.0 291.0 404.0
NEAR OCEAN 119.0 159.0 216.0 266.0 305.0 482.0
plt.figure(figsize=(9, 4))
sns.heatmap(table / 1000, annot=True, fmt=".0f", cmap="YlGnBu")
plt.title("Median home value ($1000s) by location and income")
plt.xlabel("Income band")
plt.ylabel("Location")
plt.show()
No description has been provided for this image
gap = table.loc["<1H OCEAN"] - table.loc["INLAND"]
(gap / 1000).round(0)
income_band
<20k      60.0
20-30k    72.0
30-40k    73.0
40-50k    72.0
50-60k    72.0
60k+      99.0
dtype: float64
ratio = table.loc["<1H OCEAN"] / table.loc["INLAND"]
print(f"Coastal homes cost {ratio.min():.1f} to "
      f"{ratio.max():.1f} times inland, at equal income")
Coastal homes cost 1.4 to 1.9 times inland, at equal income

Recap#

  • Histograms show the shape of one column; try a few bin counts
  • Spikes at the edge often mean capped values; check before modelling
  • Use a log scale for skewed columns and density to compare groups
  • Scatterplots need transparency or sampling when points pile up
  • Heatmaps summarise a correlation matrix or a pivot table at once

Challenges#

  1. Histogram of households on a log scale, coloured by ocean_proximity.
  2. Scatter housing_median_age against median_house_value. Any link?
  3. Pivot median income by location and age band, then draw a heatmap.

What's next#

Lesson 08: Exploratory Data Analysis and Visualization Basics Combine these charts into a full EDA workflow on a new dataset.

Data Mining 08: Exploratory Data Analysis and Visualization Basics#

In this lesson you'll be able to:

  • Follow a step-by-step EDA checklist on a brand-new dataset
  • Catch columns that only look numeric, before they bite
  • Profile numeric and text columns with statistics and charts
  • Compare every column against the target to find strong signals
  • Combine two features into a heatmap of risk segments Dataset: IBM Telco customer churn (data/mining/telco_churn.csv)

Warm-up: one histogram#

Lesson 07 drew histograms, scatterplots and heatmaps. One more, on new data.

import pandas as pd
import matplotlib.pyplot as plt
df = pd.read_csv("telco_churn.csv")
print(df.shape)
df[["customerID", "tenure", "Contract", "Churn"]].head()
(7043, 21)
customerID tenure Contract Churn
0 7590-VHVEG 1 Month-to-month No
1 5575-GNVDE 34 One year No
2 3668-QPYBK 2 Month-to-month Yes
3 7795-CFOCW 45 One year No
4 9237-HQITU 2 Month-to-month Yes
ax = df["tenure"].plot(kind="hist", bins=36, edgecolor="white",
                       figsize=(7, 4))
ax.set_title("How long customers have stayed")
ax.set_xlabel("Tenure (months)")
ax.set_ylabel("Number of customers")
No description has been provided for this image

Step 1: first look at size, types and the target#

Problem: know what you're holding before you trust a single number.

EDA checklist:

  1. First look: size, types, duplicates, target
  2. One column at a time: numbers, then categories
  3. Each column against the target
  4. Columns together: correlations and segments
df.dtypes.value_counts()
str        18
int64       2
float64     1
Name: count, dtype: int64
print(df["customerID"].duplicated().sum())
print(df.isna().sum().sum())
0
0

Common mistake: trusting a column that only looks numeric

df["TotalCharges"].astype(float)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[6], line 1
----> 1 df["TotalCharges"].astype(float)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\generic.py:6529, in NDFrame.astype(self, dtype, copy, errors)
   6525     results = [ser.astype(dtype, errors=errors) for _, ser in self.items()]
   6527 else:
   6528     # else, only a single dtype is given
-> 6529     new_data = self._mgr.astype(dtype=dtype, errors=errors)
   6530     res = self._constructor_from_mgr(new_data, axes=new_data.axes)
   6531     return res.__finalize__(self, method="astype")

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\managers.py:627, in BaseBlockManager.astype(self, dtype, errors)
    626 def astype(self, dtype, errors: str = "raise") -> Self:
--> 627     return self.apply("astype", dtype=dtype, errors=errors)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\managers.py:445, in BaseBlockManager.apply(self, f, align_keys, **kwargs)
    443         applied = b.apply(f, **kwargs)
    444     else:
--> 445         applied = getattr(b, f)(**kwargs)
    446     result_blocks = extend_blocks(applied, result_blocks)
    448 out = type(self).from_blocks(result_blocks, [ax.view() for ax in self.axes])

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\blocks.py:607, in Block.astype(self, dtype, errors, squeeze)
    604         raise ValueError("Can not squeeze with more than one column.")
    605     values = values[0, :]  # type: ignore[call-overload]
--> 607 new_values = astype_array_safe(values, dtype, errors=errors)
    609 new_values = maybe_coerce_values(new_values)
    611 refs = None

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\dtypes\astype.py:240, in astype_array_safe(values, dtype, copy, errors)
    237     dtype = dtype.numpy_dtype
    239 try:
--> 240     new_values = astype_array(values, dtype, copy=copy)
    241 except (ValueError, TypeError):
    242     # e.g. _astype_nansafe can fail on object-dtype of strings
    243     #  trying to convert to float
    244     if errors == "ignore":

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\dtypes\astype.py:182, in astype_array(values, dtype, copy)
    178     return values
    180 if not isinstance(values, np.ndarray):
    181     # i.e. ExtensionArray
--> 182     values = values.astype(dtype, copy=copy)
    184 else:
    185     values = _astype_nansafe(values, dtype, copy=copy)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\string_arrow.py:336, in ArrowStringArray.astype(self, dtype, copy)
    334     return dtype.__from_arrow__(data)
    335 elif isinstance(dtype, np.dtype) and np.issubdtype(dtype, np.floating):
--> 336     return self.to_numpy(dtype=dtype, na_value=np.nan)
    338 return super().astype(dtype, copy=copy)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\arrow\array.py:1803, in ArrowExtensionArray.to_numpy(self, dtype, copy, na_value)
   1801 result = data._pa_array.to_numpy()
   1802 if dtype is not None:
-> 1803     result = result.astype(dtype, copy=False)
   1804 if copy:
   1805     result = result.copy()

ValueError: could not convert string to float: ' '
tc = pd.to_numeric(df["TotalCharges"], errors="coerce")
print(tc.isna().sum())
df.loc[tc.isna(), ["customerID", "tenure", "TotalCharges"]].head()
11
customerID tenure TotalCharges
488 4472-LVYGI 0
753 3115-CZMZD 0
936 5709-LVOEQ 0
1082 4367-NUYAO 0
1340 1371-DWPAZ 0
df["TotalCharges"] = tc.fillna(0)
print(df["TotalCharges"].describe().round(2))
count    7043.00
mean     2279.73
std      2266.79
min         0.00
25%       398.55
50%      1394.55
75%      3786.60
max      8684.80
Name: TotalCharges, dtype: float64
print(df["Churn"].value_counts())
df["churned"] = (df["Churn"] == "Yes").astype(int)
print(round(df["churned"].mean(), 3))
Churn
No     5174
Yes    1869
Name: count, dtype: int64
0.265

Step 2: numeric columns, one at a time#

Problem: learn each number's range, centre and shape.

num = ["tenure", "MonthlyCharges", "TotalCharges"]
df[num].describe().round(2)
tenure MonthlyCharges TotalCharges
count 7043.00 7043.00 7043.00
mean 32.37 64.76 2279.73
std 24.56 30.09 2266.79
min 0.00 18.25 0.00
25% 9.00 35.50 398.55
50% 29.00 70.35 1394.55
75% 55.00 89.85 3786.60
max 72.00 118.75 8684.80
axes = df[num].hist(bins=30, figsize=(12, 3.5), layout=(1, 3))
for ax in axes.ravel():
    ax.set_xlabel(ax.get_title())
    ax.set_ylabel("Customers")
plt.suptitle("Numeric columns at a glance", y=1.05)
No description has been provided for this image
counts = df["tenure"].value_counts()
print(counts[1], counts[72])
print(df["tenure"].max())
613 362
72

Quick check: who makes up the tall spike near 20 dollars in MonthlyCharges?

df.groupby("InternetService")["MonthlyCharges"].median()
InternetService
DSL            56.150
Fiber optic    91.675
No             20.150
Name: MonthlyCharges, dtype: float64
df[num].skew().round(2)
tenure            0.24
MonthlyCharges   -0.22
TotalCharges      0.96
dtype: float64

Step 3: categorical columns#

Problem: see which categories exist, and which column is just an ID.

df.nunique().sort_values().tail(4)
tenure              73
MonthlyCharges    1585
TotalCharges      6531
customerID        7043
dtype: int64
df["Contract"].value_counts()
Contract
Month-to-month    3875
Two year          1695
One year          1473
Name: count, dtype: int64
ax = df["Contract"].value_counts().plot(kind="bar", rot=0,
                                        figsize=(7, 4))
ax.set_title("Customers by contract type")
ax.set_xlabel("Contract")
ax.set_ylabel("Number of customers")
No description has been provided for this image
three = ["InternetService", "TechSupport", "PaperlessBilling"]
fig, axes = plt.subplots(1, 3, figsize=(12, 3.5))
for ax, col in zip(axes, three):
    df[col].value_counts().plot(kind="bar", ax=ax, rot=0)
    ax.set_title(col)
    ax.set_ylabel("Customers")
plt.tight_layout()
No description has been provided for this image

Quick check: TechSupport has a third category. Where does it come from?

pd.crosstab(df["InternetService"], df["TechSupport"])
TechSupport No No internet service Yes
InternetService
DSL 1243 0 1178
Fiber optic 2230 0 866
No 0 1526 0

Your turn#

What share of customers uses each PaymentMethod? Round to 3 places.

df["PaymentMethod"].value_counts(normalize=True).round(3)
PaymentMethod
Electronic check             0.336
Mailed check                 0.229
Bank transfer (automatic)    0.219
Credit card (automatic)      0.216
Name: proportion, dtype: float64

Step 4: numbers against the target#

Problem: do customers who leave look different from those who stay?

Common mistake: averaging a whole table that contains text

df.groupby("Churn").mean()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[21], line 1
----> 1 df.groupby("Churn").mean()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\groupby\groupby.py:2285, in GroupBy.mean(self, numeric_only, skipna, engine, engine_kwargs)
   2277     return self._numba_agg_general(
   2278         grouped_mean,
   2279         executor.float_dtype_mapping,
   (...)
   2282         skipna=skipna,
   2283     )
   2284 else:
-> 2285     result = self._cython_agg_general(
   2286         "mean",
   2287         alt=lambda x: Series(x, copy=False).mean(
   2288             numeric_only=numeric_only, skipna=skipna
   2289         ),
   2290         numeric_only=numeric_only,
   2291         skipna=skipna,
   2292     )
   2293     return result.__finalize__(self.obj, method="groupby")

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\groupby\groupby.py:1808, in GroupBy._cython_agg_general(self, how, alt, numeric_only, min_count, **kwargs)
   1805     result = self._agg_py_fallback(how, values, ndim=data.ndim, alt=alt)
   1806     return result
-> 1808 new_mgr = data.grouped_reduce(array_func)
   1809 res = self._wrap_agged_manager(new_mgr)
   1810 if how in ["idxmin", "idxmax"]:
   1811     # mypy expects how to be Literal["idxmin", "idxmax"].

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\managers.py:1667, in BlockManager.grouped_reduce(self, func)
   1665             result_blocks = extend_blocks(applied, result_blocks)
   1666     else:
-> 1667         applied = blk.apply(func)
   1668         result_blocks = extend_blocks(applied, result_blocks)
   1670 if len(result_blocks) == 0:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\blocks.py:347, in Block.apply(self, func, **kwargs)
    341 @final
    342 def apply(self, func, **kwargs) -> list[Block]:
    343     """
    344     apply the function to my values; return a block if we are not
    345     one
    346     """
--> 347     result = func(self.values, **kwargs)
    349     result = maybe_coerce_values(result)
    350     return self._split_op_result(result)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\groupby\groupby.py:1783, in GroupBy._cython_agg_general.<locals>.array_func(values)
   1781 def array_func(values: ArrayLike) -> ArrayLike:
   1782     try:
-> 1783         result = self._grouper._cython_operation(
   1784             "aggregate",
   1785             values,
   1786             how,
   1787             axis=data.ndim - 1,
   1788             min_count=min_count,
   1789             **kwargs,
   1790         )
   1791     except NotImplementedError:
   1792         # generally if we have numeric_only=False
   1793         # and non-applicable functions
   1794         # try to python agg
   1795         # TODO: shouldn't min_count matter?
   1796         # TODO: avoid special casing SparseArray here
   1797         if how in ["any", "all"] and isinstance(values, SparseArray):

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\groupby\ops.py:963, in BaseGrouper._cython_operation(self, kind, values, how, axis, min_count, **kwargs)
    959 assert kind in ["transform", "aggregate"]
    961 cy_op = WrappedCythonOp(kind=kind, how=how, has_dropped_na=self.has_dropped_na)
--> 963 return cy_op.cython_operation(
    964     values=values,
    965     axis=axis,
    966     min_count=min_count,
    967     comp_ids=self.ids,
    968     ngroups=self.ngroups,
    969     **kwargs,
    970 )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\groupby\ops.py:553, in WrappedCythonOp.cython_operation(self, values, axis, min_count, comp_ids, ngroups, **kwargs)
    549 self._validate_axis(axis, values)
    551 if not isinstance(values, np.ndarray):
    552     # i.e. ExtensionArray
--> 553     return values._groupby_op(
    554         how=self.how,
    555         has_dropped_na=self.has_dropped_na,
    556         min_count=min_count,
    557         ngroups=ngroups,
    558         ids=comp_ids,
    559         **kwargs,
    560     )
    562 return self._cython_op_ndim_compat(
    563     values,
    564     min_count=min_count,
   (...)
    568     **kwargs,
    569 )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\arrow\array.py:2748, in ArrowExtensionArray._groupby_op(self, how, has_dropped_na, min_count, ngroups, ids, **kwargs)
   2736 if isinstance(self.dtype, StringDtype):
   2737     if how in [
   2738         "prod",
   2739         "mean",
   (...)
   2746         "skew",
   2747     ]:
-> 2748         raise TypeError(
   2749             f"dtype '{self.dtype}' does not support operation '{how}'"
   2750         )
   2751     return super()._groupby_op(
   2752         how=how,
   2753         has_dropped_na=has_dropped_na,
   (...)
   2757         **kwargs,
   2758     )
   2760 # maybe convert to a compatible dtype optimized for groupby

TypeError: dtype 'str' does not support operation 'mean'
df.groupby("Churn")[num].median().round(2)
tenure MonthlyCharges TotalCharges
Churn
No 38.0 64.43 1679.52
Yes 10.0 79.65 703.55
import seaborn as sns
ax = sns.boxplot(data=df, x="Churn", y="tenure")
ax.set_title("Tenure of customers who stayed vs left")
ax.set_xlabel("Left the company?")
ax.set_ylabel("Tenure (months)")
No description has been provided for this image
ax = sns.histplot(data=df, x="MonthlyCharges", hue="Churn", bins=30,
                  stat="density", common_norm=False,
                  element="step")
ax.set_title("Monthly charges: stayed vs left")
ax.set_xlabel("Monthly charge (dollars)")
ax.set_ylabel("Density")
No description has been provided for this image
pts = df.sample(1000, random_state=42)
ax = sns.scatterplot(data=pts, x="tenure", y="MonthlyCharges",
                     hue="Churn", alpha=0.6)
ax.set_title("Tenure vs monthly charge, 1,000 sampled customers")
ax.set_xlabel("Tenure (months)")
ax.set_ylabel("Monthly charge (dollars)")
No description has been provided for this image

Your turn#

Draw a boxplot of TotalCharges for stayers and leavers.

ax = sns.boxplot(data=df, x="Churn", y="TotalCharges")
ax.set_title("Total charges of customers who stayed vs left")
ax.set_xlabel("Left the company?")
ax.set_ylabel("Total charges (dollars)")
print(df.groupby("Churn")["TotalCharges"].median())
Churn
No     1679.525
Yes     703.550
Name: TotalCharges, dtype: float64
No description has been provided for this image
left = df[df["churned"] == 1]
early = (left["tenure"] <= 12).mean()
print(len(left), round(early, 2))
1869 0.55

Step 5: categories against the target#

Problem: which categories have the highest churn rate?

rate = df.groupby("Contract")["churned"].mean()
rate.round(3)
Contract
Month-to-month    0.427
One year          0.113
Two year          0.028
Name: churned, dtype: float64
ax = rate.plot(kind="bar", rot=0, figsize=(7, 4))
ax.axhline(df["churned"].mean(), color="red", linestyle="--",
           label="All customers")
ax.set_title("Churn rate by contract type")
ax.set_xlabel("Contract")
ax.set_ylabel("Share who left")
ax.legend()
No description has been provided for this image
mix = pd.crosstab(df["InternetService"], df["Churn"],
                  normalize="index")
mix.round(3)
Churn No Yes
InternetService
DSL 0.810 0.190
Fiber optic 0.581 0.419
No 0.926 0.074
ax = mix.plot(kind="bar", stacked=True, rot=0, figsize=(7, 4))
ax.set_title("Stayed vs left by internet service")
ax.set_xlabel("Internet service")
ax.set_ylabel("Share of customers")
No description has been provided for this image

Quick check: which payment method has the highest churn rate?

pay = df.groupby("PaymentMethod")["churned"].agg(["size", "mean"])
pay.sort_values("mean").round(3)
size mean
PaymentMethod
Credit card (automatic) 1522 0.152
Bank transfer (automatic) 1544 0.167
Mailed check 1612 0.191
Electronic check 2365 0.453
cats = ["Contract", "InternetService", "PaymentMethod",
        "TechSupport", "PaperlessBilling", "gender"]
for col in cats:
    r = df.groupby(col)["churned"].mean()
    print(f"{col:16} spread {r.max() - r.min():.2f}")
Contract         spread 0.40
InternetService  spread 0.34
PaymentMethod    spread 0.30
TechSupport      spread 0.34
PaperlessBilling spread 0.17
gender           spread 0.01

Your turn#

Find the churn rate and head count for SeniorCitizen (0 or 1).

df.groupby("SeniorCitizen")["churned"].agg(["size", "mean"]).round(3)
size mean
SeniorCitizen
0 5901 0.236
1 1142 0.417

Step 6: numbers together with correlations#

Problem: spot columns that repeat each other, and their link to churn.

corr = df[num + ["churned"]].corr().round(2)
corr
tenure MonthlyCharges TotalCharges churned
tenure 1.00 0.25 0.83 -0.35
MonthlyCharges 0.25 1.00 0.65 0.19
TotalCharges 0.83 0.65 1.00 -0.20
churned -0.35 0.19 -0.20 1.00
plt.figure(figsize=(6, 5))
sns.heatmap(corr, annot=True, cmap="coolwarm", vmin=-1, vmax=1)
plt.title("Correlation of numeric columns with churn")
plt.show()
No description has been provided for this image

Quick check: why are TotalCharges and tenure so strongly linked?

paying = df[df["tenure"] > 0]
approx = paying["tenure"] * paying["MonthlyCharges"]
print((paying["TotalCharges"] / approx).median().round(3))
1.0

Mini-project: who is most at risk of leaving?#

Combine the two strongest signals, contract and tenure, into segments.

bands = ["0-12", "13-24", "25-48", "49-72"]
df["tenure_band"] = pd.cut(df["tenure"], bins=[0, 12, 24, 48, 72],
                           labels=bands, include_lowest=True)
df["tenure_band"].value_counts(sort=False)
tenure_band
0-12     2186
13-24    1024
25-48    1594
49-72    2239
Name: count, dtype: int64
seg = df.pivot_table(index="Contract", columns="tenure_band",
                     values="churned", aggfunc="mean",
                     observed=True)
seg.round(2)
tenure_band 0-12 13-24 25-48 49-72
Contract
Month-to-month 0.51 0.38 0.33 0.26
One year 0.10 0.08 0.11 0.13
Two year 0.00 0.00 0.02 0.03
pd.crosstab(df["Contract"], df["tenure_band"])
tenure_band 0-12 13-24 25-48 49-72
Contract
Month-to-month 1994 737 802 342
One year 124 197 518 634
Two year 68 90 274 1263
plt.figure(figsize=(8, 4))
sns.heatmap(seg, annot=True, fmt=".2f", cmap="Reds")
plt.title("Churn rate by contract and tenure")
plt.xlabel("Tenure band (months)")
plt.ylabel("Contract")
plt.show()
No description has been provided for this image
risky = (df["Contract"] == "Month-to-month") & (df["tenure"] <= 12)
share = df.loc[risky, "churned"].sum() / df["churned"].sum()
print(f"{risky.sum()} customers, {risky.mean():.0%} of the base")
print(f"churn {df.loc[risky, 'churned'].mean():.0%}, "
      f"{share:.0%} of all leavers")
1994 customers, 28% of the base
churn 51%, 55% of all leavers

Recap#

  • Follow a checklist: first look, single columns, vs target, together
  • Check types early: to_numeric with errors='coerce' exposes hidden blanks
  • A 0/1 target makes groupby means into churn rates
  • Compare groups with boxplots and density histograms
  • Pivot two strong signals into a heatmap to find risk segments

Challenges#

  1. Draw a boxplot of MonthlyCharges by Contract.
  2. Rank OnlineSecurity, StreamingTV and Partner by churn-rate spread.
  3. Build a heatmap of churn by InternetService and tenure band.

What's next#

Lesson 09: Data Cleaning- Handling Missing Values, Noise, and Outliers Find hidden missing values, fill them sensibly, and tame outliers.

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.