Lesson 1 · Data Mining Full Course
Data Mining Full Course Part 1 (3 Hours): Foundations & Exploratory Data Analysis in Python
All 8 lesson notebooks from Part 1 of the Data Mining Full Course in Python, in teaching order. Each lesson starts with a heading; the timestamps below jump…
- CourseData Mining Full Course
- Lesson1 of 6
- Video3 h 9 min
- FormatJupyter notebook · 303 code cells
- Data8 datasets
What you'll learn
- Data Mining 01: Introduction to Data Mining and KDD Process
- Warm-up: a table you already know
- From data to knowledge
- The KDD process: five steps from file to finding
- Step 1, Selection: choose the target and the evidence
- Step 2, Preprocessing: find and handle the gaps
- Step 3, Transformation: numbers a model can learn from
- Step 4, Data mining: let a model find the pattern
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
- titanic.csv preview58.9 KB
- groceries_baskets.csv preview489.1 KB
- penguins.csv preview13.2 KB
- telco_churn.csv preview947.7 KB
- breast_cancer.csv preview121.9 KB
- sms_spam.csv preview469.5 KB
- iris.csv preview3.8 KB
- housing.csv preview1.4 MB
📓 Full notebook
Download .ipynbData Mining Full Course, Part 1: Foundations and Exploratory Data Analysis#
All 8 lesson notebooks from Part 1 of the Data Mining Full Course in Python, in teaching order. Each lesson starts with a heading; the timestamps below jump to it in the video.
- 0:00 Lesson 1: Introduction to Data Mining and KDD Process
- 0:22:55 Lesson 2: Data Mining vs Data Science vs Machine Learning
- 0:45:41 Lesson 3: Applications of Data Mining in Business, Healthcare, and Security
- 1:08:37 Lesson 4: Getting Started with Python, Pandas, and scikit-learn
- 1:33:16 Lesson 5: Introduction to Exploratory Data Mining
- 1:57:22 Lesson 6: Summarization and Descriptive Statistics
- 2:21:41 Lesson 7: Visualization- Histograms, Scatterplots, Heatmaps
- 2:45:25 Lesson 8: Exploratory Data Analysis and Visualization Basics
Data files: save the datasets from the lesson page in the same folder as this notebook. Files the lessons create are written to that folder too.
Data Mining 01: Introduction to Data Mining and KDD Process#
In this lesson you'll be able to:
- Explain what data mining is, and how data differs from knowledge
- Name the five steps of the KDD process
- Select, clean and transform a real table ready for mining
- Train a first model and judge it on passengers it has never seen
- Reuse the same five steps to mine shopping baskets Datasets: Titanic passengers (data/pandas/titanic.csv) and real grocery baskets (data/mining/groceries_baskets.csv)
Warm-up: a table you already know#
If you followed the Pandas series, loading and peeking will feel familiar.
import pandas as pd
df = pd.read_csv("titanic.csv")
print(df.shape)
df.head(3)
print(df["Survived"].mean().round(3))
df.groupby("Sex")["Survived"].mean().round(3)
From data to knowledge#
Problem: a file of facts is not yet an answer. What turns it into one?
- Data: recorded facts, one row at a time
- Knowledge: a pattern that holds across many rows and helps a decision
- Data mining: finding those patterns automatically, in large data
df.loc[0, ["Name", "Pclass", "Fare", "Survived"]]
Quick check: is "third class passengers paid less" data or knowledge?
fares = df.groupby("Pclass")["Fare"].median()
fares.round(2)
The KDD process: five steps from file to finding#
Problem: mining goes wrong when steps are skipped. KDD gives the order.
KDD = Knowledge Discovery in Databases
- Selection: pick the question, the rows and the columns
- Preprocessing: find and fix gaps and errors
- Transformation: reshape columns into a form a method can use
- Data mining: run a method that finds the pattern
- Interpretation / evaluation: is the pattern true and useful?
Our question: which passengers were most likely to survive, and why?
Step 1, Selection: choose the target and the evidence#
Problem: which column is the answer, and which columns could explain it?
print(list(df.columns))
id_cols = ["PassengerId", "Name", "Ticket"]
df[id_cols].nunique()
target = "Survived"
features = ["Pclass", "Sex", "Age", "SibSp", "Parch", "Fare",
"Embarked"]
data = df[features + [target]].copy()
print(data.shape)
Common mistake: selecting several columns with single brackets.
df["Pclass", "Sex"]
df[["Pclass", "Sex"]].head(3)
Step 2, Preprocessing: find and handle the gaps#
Problem: real data has holes, and most methods refuse to work with them.
data.isna().sum()
Quick check: if we drop every row of the FULL table that has any gap, how many passengers are left?
print(len(df), len(df.dropna()))
print(len(data.dropna()))
top_port = data["Embarked"].mode()[0]
data["Embarked"] = data["Embarked"].fillna(top_port)
print(top_port, data["Embarked"].isna().sum())
Age has too many gaps to drop. We'll fill them with a typical age, the median. But which passengers should that median come from? Step 3 answers that.
Step 3, Transformation: numbers a model can learn from#
Problem: models need numbers, and they must be tested on data they never saw.
Common mistake: handing text columns straight to a model.
from sklearn.tree import DecisionTreeClassifier
model = DecisionTreeClassifier(random_state=42)
model.fit(data[features], data[target])
X = pd.get_dummies(data[features], columns=["Sex", "Embarked"],
dtype=int)
y = data[target]
print(list(X.columns))
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y)
print(len(X_train), len(X_test))
age_median = X_train["Age"].median()
X_train["Age"] = X_train["Age"].fillna(age_median)
X_test["Age"] = X_test["Age"].fillna(age_median)
print(age_median, X_train["Age"].isna().sum(), X_test["Age"].isna().sum())
Your turn#
Add a column family to X_train and X_test: SibSp + Parch + 1 (the passenger plus relatives aboard).
for part in (X_train, X_test):
part["family"] = part["SibSp"] + part["Parch"] + 1
X_train["family"].value_counts().sort_index().head(6)
Step 4, Data mining: let a model find the pattern#
Problem: hundreds of rows, many columns. Which combination predicts survival?
A decision tree learns a flowchart of yes/no questions, like "Is the passenger female?" Each question splits the passengers into purer groups of survivors and non-survivors.
baseline = (y_test == 0).mean()
print(f"Guess everyone died: {baseline:.1%}")
tree = DecisionTreeClassifier(max_depth=3, random_state=42)
tree.fit(X_train, y_train)
test_acc = tree.score(X_test, y_test)
print(f"Tree accuracy on unseen passengers: {test_acc:.1%}")
from sklearn.tree import export_text
rules = export_text(tree, feature_names=list(X_train.columns),
decimals=1)
print(rules)
sample = X_test.head(4)
check = sample[["Pclass", "Sex_female", "Age"]].copy()
check["predicted"] = tree.predict(sample)
check["actual"] = y_test.head(4)
check
Quick check: which single column does the tree rely on most?
importance = pd.Series(tree.feature_importances_,
index=X_train.columns)
importance.sort_values(ascending=False).round(3).head(5)
Step 5, Interpretation and evaluation: is the pattern real?#
Problem: a pattern can fit the past perfectly and still fail on new cases.
Quick check: a tree with no depth limit will fit the training passengers better. Will it also win on the test passengers?
deep = DecisionTreeClassifier(random_state=42)
deep.fit(X_train, y_train)
for name, m in [("depth 3", tree), ("no limit", deep)]:
train_acc = m.score(X_train, y_train)
print(name, round(train_acc, 3), round(m.score(X_test, y_test), 3))
Rule: never judge a model on the rows it learned from. Always use held-out data.
pred = tree.predict(X_test)
pd.crosstab(y_test, pred, rownames=["actual"], colnames=["predicted"])
rates = df.pivot_table(index="Sex", columns="Pclass",
values="Survived", aggfunc="mean")
rates.round(2)
import matplotlib.pyplot as plt
ax = rates.T.plot(kind="bar", figsize=(7, 4))
ax.set_title("Titanic survival rate by class and sex")
ax.set_xlabel("Passenger class")
ax.set_ylabel("Share who survived")
plt.show()
Finding: sex and class drive survival. Women in 1st and 2nd class mostly lived; men in 2nd and 3rd mostly did not. This matches the "women and children first" policy of 1912: the pattern makes sense.
Mini-project: the same five steps on shopping baskets#
Question: when a basket holds whole milk, which other items are unusually likely?
path = "groceries_baskets.csv"
lines = pd.read_csv(path, header=None, sep="|")[0]
print(len(lines))
lines.head(3)
onehot = lines.str.get_dummies(sep=",")
print(onehot.shape)
onehot.sum().sort_values(ascending=False).head(5)
milk = onehot["whole milk"] == 1
with_milk = onehot[milk].mean()
overall = onehot.mean()
lift = (with_milk / overall).drop("whole milk")
common = overall[overall >= 0.03].index
lift[lift.index.isin(common)].sort_values(ascending=False).round(2).head(5)
top = lift[lift.index.isin(common)].nlargest(8)
ax = top.sort_values().plot(kind="barh", figsize=(7, 4))
ax.axvline(1, color="grey", linestyle="--")
ax.set_title("Items bought with whole milk: lift")
ax.set_xlabel("Lift (1 = no link)")
ax.set_ylabel("Item")
best = lift[lift.index.isin(common)].idxmax()
print(f"Milk shoppers buy {best} {lift[best]:.2f} times as often")
Recap#
- Data is facts; knowledge is a pattern across many rows that helps a decision
- KDD: selection, preprocessing, transformation, mining, interpretation
- Fill gaps with values learned from the training rows only
- Beat a baseline, and always score on held-out data
- The same five steps work for prediction and for pattern discovery
Challenges#
- Refit the depth-3 tree without Fare and Embarked. Does test accuracy change?
- Try max_depth from 1 to 8 and print train and test accuracy for each.
- Repeat the basket mini-project for "yogurt" instead of whole milk.
What's next#
Lesson 02: Data Mining vs Data Science vs Machine Learning Three words people mix up, shown side by side on one real dataset.
Data Mining 02: Data Mining vs Data Science vs Machine Learning#
In this lesson you'll be able to:
- Tell the three fields apart by the question each one asks
- Answer a data science question and communicate it with a chart
- Mine hidden groups from data that has no labels
- Train a machine learning model and test it on unseen cases Dataset: Palmer penguins, 344 real birds from Antarctica (data/pandas/penguins.csv)
Warm-up: the first KDD steps on new data#
Lesson 01 walked the KDD process. Selection and preprocessing come first, whatever the field.
import pandas as pd
df = pd.read_csv("penguins.csv")
print(df.shape)
df.head(3)
df.isna().sum()
Three fields, three kinds of question#
Problem: the words overlap, so define each one by what it is for.
- Data science: the whole job, from a question to a communicated answer
- Data mining: discovering patterns nobody told us to look for, in large data
- Machine learning: algorithms that learn from examples to predict new cases Data science contains the other two; mining borrows many ML algorithms.
Quick check: which field fits each question? A. How much heavier are Gentoo penguins, and how do we show it? B. Are there natural groups of penguins, if nobody tells us the species? C. Which species is a newly measured penguin?
Answers: A is data science, B is data mining, C is machine learning. We'll now answer all three, one section each.
Data science: answer a question, then show it#
Question A: how much heavier are Gentoo penguins, and does sex matter?
Common mistake: averaging a whole table that still holds text columns.
df.groupby("species").mean()
cols = ["bill_length_mm", "flipper_length_mm", "body_mass_g"]
df.groupby("species")[cols].mean().round(1)
mass = df.pivot_table(index="species", columns="sex",
values="body_mass_g", aggfunc="mean")
mass.round(0)
import matplotlib.pyplot as plt
ax = mass.plot(kind="bar", figsize=(7, 4), rot=0)
ax.set_title("Average body mass of Palmer penguins")
ax.set_xlabel("Species")
ax.set_ylabel("Body mass (g)")
plt.show()
gap = mass.loc["Gentoo"].mean() - mass.loc["Adelie"].mean()
print(f"Gentoo penguins are about {gap:.0f} g heavier than Adelie")
Your turn#
Which island has the heaviest penguins on average?
df.groupby("island")["body_mass_g"].mean().round(0)
pd.crosstab(df["island"], df["species"])
That's data science: a clear question, a summary, a chart and a plain sentence.
Data mining: discover groups nobody labelled#
Question B: hide the species. Do the measurements alone fall into natural groups?
k-means places k centre points and gives each bird to its nearest centre. It then moves each centre to the middle of its birds, and repeats until nothing changes.
num = ["bill_length_mm", "bill_depth_mm", "flipper_length_mm",
"body_mass_g"]
from sklearn.cluster import KMeans
km = KMeans(n_clusters=3, n_init=10, random_state=42)
Common mistake: clustering data that still has gaps.
km.fit(df[num])
birds = df.dropna(subset=num).copy()
print(len(birds))
birds[num].std().round(1)
Quick check: body mass spreads hundreds of times more than bill depth. What will k-means care about most?
km.fit(birds[num])
pd.crosstab(km.labels_, birds["species"], rownames=["group"])
from sklearn.preprocessing import StandardScaler
scaled = StandardScaler().fit_transform(birds[num])
km.fit(scaled)
birds["raw_group"] = km.labels_
order = birds.groupby("raw_group")["flipper_length_mm"].mean()
names = {g: i for i, g in enumerate(order.sort_values().index)}
birds["group"] = birds["raw_group"].map(names)
pd.crosstab(birds["group"], birds["species"])
birds.groupby("group")[num].mean().round(1)
ax = birds.plot.scatter(x="flipper_length_mm", y="bill_length_mm",
c="group", cmap="viridis", figsize=(7, 4))
ax.set_title("Penguin groups found by k-means (no labels used)")
ax.set_xlabel("Flipper length (mm)")
ax.set_ylabel("Bill length (mm)")
from sklearn.metrics import silhouette_score
for k in range(2, 6):
test_km = KMeans(n_clusters=k, n_init=10, random_state=42)
labels = test_km.fit_predict(scaled)
print(k, round(silhouette_score(scaled, labels), 3))
Mining finds structure; a person decides whether it is useful.
Machine learning: predict penguins it has never seen#
Question C: given a new bird's measurements, which species is it?
k-nearest neighbours finds the 5 most similar known birds and takes a vote. Learning from labelled examples like this is called supervised learning.
from sklearn.model_selection import train_test_split
X, y = birds[num], birds["species"]
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=42, stratify=y)
print(len(X_train), len(X_test))
from sklearn.pipeline import make_pipeline
from sklearn.neighbors import KNeighborsClassifier
knn = make_pipeline(StandardScaler(), KNeighborsClassifier(5))
knn.fit(X_train, y_train)
print(f"Test accuracy: {knn.score(X_test, y_test):.1%}")
Quick check: what happens to kNN accuracy if we skip the scaler?
raw_knn = KNeighborsClassifier(5).fit(X_train, y_train)
print(f"Without scaling: {raw_knn.score(X_test, y_test):.1%}")
new_birds = X_test.head(4)
result = new_birds.copy()
result["predicted"] = knn.predict(new_birds)
result["actual"] = y_test.head(4)
result[["flipper_length_mm", "body_mass_g", "predicted", "actual"]]
Common mistake: predicting with different columns from training.
knn.predict(result)
print(list(knn.feature_names_in_))
knn.predict(result[knn.feature_names_in_])
Your turn#
Train the same pipeline using only the two bill columns. How much accuracy is lost?
bill = ["bill_length_mm", "bill_depth_mm"]
knn_bill = make_pipeline(StandardScaler(), KNeighborsClassifier(5))
knn_bill.fit(X_train[bill], y_train)
print(f"{knn_bill.score(X_test[bill], y_test):.1%}")
Side by side: how the three fields differ#
Problem: knowing which field you're in tells you how to judge success.
| Data science | Data mining | Machine learning | |
|---|---|---|---|
| Question | What happened, and why? | What patterns are hidden? | What will this new case be? |
| Labels needed? | Sometimes | Often not | Yes, for supervised |
| Judged by | Clear, correct communication | Useful, new patterns | Accuracy on unseen data |
Quick check: our k-means groups matched species well. Is that a machine learning accuracy score?
match = pd.crosstab(birds["group"], birds["species"]).max(axis=1)
print(f"{match.sum() / len(birds):.1%} of birds sit with their kind")
Mini-project: can measurements tell male from female?#
A machine learning question, judged against a baseline, with a data science answer.
sexed = birds.dropna(subset=["sex"])
Xs = pd.get_dummies(sexed[num + ["species"]], columns=["species"],
dtype=int)
ys = sexed["sex"]
print(len(sexed), ys.value_counts().to_dict())
from sklearn.linear_model import LogisticRegression
Xs_tr, Xs_te, ys_tr, ys_te = train_test_split(
Xs, ys, test_size=0.25, random_state=42, stratify=ys)
sex_model = make_pipeline(StandardScaler(), LogisticRegression())
sex_model.fit(Xs_tr, ys_tr)
print("Trained on", len(Xs_tr), "birds")
baseline = (ys_te == ys_tr.mode()[0]).mean()
acc = sex_model.score(Xs_te, ys_te)
print(f"Baseline {baseline:.1%}, model {acc:.1%}")
no_species = make_pipeline(StandardScaler(), LogisticRegression())
no_species.fit(Xs_tr[num], ys_tr)
print(f"{no_species.score(Xs_te[num], ys_te):.1%}")
coefs = sex_model[-1].coef_[0]
weights = pd.Series(coefs, index=Xs.columns).round(2)
weights.sort_values(ascending=False)
probs = sex_model.predict_proba(Xs_te.head(3)).round(2)
print(list(sex_model.classes_))
print(probs, list(ys_te.head(3)))
right = pd.Series(sex_model.predict(Xs_te) == ys_te, index=ys_te.index)
kind = sexed.loc[ys_te.index, "species"]
right.groupby(kind).agg(["mean", "size"]).round(3)
Finding: measurements identify sex far better than guessing, and knowing the species helps.
Recap#
- Data science runs from a question to a communicated answer
- Data mining discovers patterns, often without labels
- Machine learning learns from examples to predict new cases
- Scale features before distance-based methods like k-means and kNN
- Judge each by its own yardstick: clarity, usefulness, or test accuracy
Challenges#
- Average flipper length by island and species. Which island holds which species?
- Run k-means with 2 groups. Which species end up together?
- Try KNeighborsClassifier with 1, 3, 7 and 15 neighbours. Does accuracy change?
What's next#
Lesson 03: Applications of Data Mining in Business, Healthcare, and Security Real shopping baskets, tumour scans and spam messages.
Data Mining 03: Applications of Data Mining in Business, Healthcare, and Security#
In this lesson you'll be able to:
- Describe any application as question, data, method, decision and cost of error
- Mine customer records for the segment most likely to leave
- Build a tumour classifier and tune how cautious it is
- Build a spam filter and check what it wrongly blocks Datasets: telco_churn.csv, breast_cancer.csv and sms_spam.csv (all in data/mining)
Warm-up: which field is this?#
Lesson 02 split work into data science, data mining and machine learning.
import pandas as pd
telco = pd.read_csv("telco_churn.csv")
print(telco.shape)
telco["Churn"].value_counts()
telco["left"] = (telco["Churn"] == "Yes").astype(int)
print(round(telco["left"].mean(), 3))
Every application has the same shape#
Problem: fields look different, but the same five questions decide every project.
| Field | Question | Decision | Worst mistake |
|---|---|---|---|
| Business | Who will leave? | Offer a deal | Missing a leaver |
| Healthcare | Is this tumour malignant? | Order a biopsy | Missing a cancer |
| Security | Is this message spam? | Block it | Blocking a real message |
Quick check: in which field is a false alarm cheapest?
Healthcare: an extra check on a benign tumour costs far less than a missed cancer. In security it's the opposite: a blocked real message is the costly error.
Business: find the customers about to leave#
Problem: keeping a customer is cheaper than winning a new one. Who is at risk?
Common mistake: trusting a column that only looks numeric.
telco["TotalCharges"].astype(float)
total = pd.to_numeric(telco["TotalCharges"], errors="coerce")
telco["TotalCharges"] = total
telco.loc[total.isna(), ["tenure", "MonthlyCharges"]].head()
by_contract = telco.groupby("Contract")["left"].agg(["mean", "size"])
by_contract.round(3)
import matplotlib.pyplot as plt
ax = by_contract["mean"].plot(kind="bar", rot=0, figsize=(7, 4))
ax.set_title("Share of customers who left, by contract")
ax.set_xlabel("Contract type")
ax.set_ylabel("Churn rate")
edges = [0, 12, 24, 48, 72]
bands = ["0-12", "13-24", "25-48", "49-72"]
telco["tenure_band"] = pd.cut(telco["tenure"], bins=edges,
labels=bands, include_lowest=True)
telco.groupby("tenure_band", observed=True)["left"].mean().round(3)
Quick check: which contract and tenure combination do you expect to be riskiest?
combo = telco.pivot_table(index="Contract", columns="tenure_band",
values="left", aggfunc="mean",
observed=True)
combo.round(2)
Your turn#
Find the churn rate for each PaymentMethod. Which one stands out?
pay = telco.groupby("PaymentMethod")["left"].mean()
pay.sort_values(ascending=False).round(3)
new_monthly = (telco["Contract"] == "Month-to-month") & (
telco["tenure"] <= 12)
seg = telco[new_monthly]
share = seg["left"].sum() / telco["left"].sum()
print(len(seg), round(seg["left"].mean(), 3), round(share, 3))
Business finding: new month-to-month customers are the place to spend the retention budget.
Healthcare: flag tumours that need a closer look#
Problem: doctors need help deciding which tumours are likely malignant.
cancer = pd.read_csv("breast_cancer.csv")
print(cancer.shape)
cancer["diagnosis"].value_counts()
cols = ["mean radius", "worst concave points"]
cancer.groupby("diagnosis")[cols].mean().round(3)
fig, ax = plt.subplots(figsize=(7, 4))
for label, part in cancer.groupby("diagnosis"):
part["mean radius"].plot(kind="hist", bins=30, alpha=0.6,
label=label, ax=ax)
ax.set_title("Nucleus radius by diagnosis")
ax.set_xlabel("Mean radius")
ax.set_ylabel("Number of tumours")
ax.legend()
Logistic regression weighs every measurement and turns the total into a probability.
from sklearn.model_selection import train_test_split
X = cancer.drop(columns="diagnosis")
y = (cancer["diagnosis"] == "malignant").astype(int)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=42, stratify=y)
print(len(X_train), len(X_test))
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
clf = make_pipeline(StandardScaler(), LogisticRegression())
clf.fit(X_train, y_train)
print(f"Accuracy: {clf.score(X_test, y_test):.1%}")
Quick check: accuracy is high. Is that enough for a hospital?
pred = clf.predict(X_test)
pd.crosstab(y_test, pred, rownames=["actual"], colnames=["predicted"])
from sklearn.metrics import recall_score
print(f"Recall: {recall_score(y_test, pred):.1%}")
Screening policy, set in advance: refer any tumour with at least a 30% chance.
prob = clf.predict_proba(X_test)[:, 1]
cautious = (prob >= 0.3).astype(int)
print(pd.crosstab(y_test, cautious, rownames=["actual"],
colnames=["flagged"]))
print(f"Recall: {recall_score(y_test, cautious):.1%}")
view = y_test.head(5).to_frame("actual")
view["chance"] = prob[:5].round(3)
view["flagged"] = cautious[:5]
view
Your turn#
How many test patients does the 30% policy send for a biopsy, and what share is that?
referred = cautious.sum()
print(referred, f"{referred / len(cautious):.1%}")
Security: filter spam before it reaches people#
Problem: spam texts carry scams and phishing links. Block them, but never real messages.
sms = pd.read_csv("sms_spam.csv")
print(sms["label"].value_counts())
print(sms["message"].iloc[2][:70])
sms["length"] = sms["message"].str.len()
sms.groupby("label")["length"].median()
fig, ax = plt.subplots(figsize=(7, 4))
for label, part in sms.groupby("label"):
part["length"].clip(upper=300).plot(kind="hist", bins=30,
alpha=0.6, label=label, ax=ax)
ax.set_title("Text message length: ham vs spam")
ax.set_xlabel("Characters (capped at 300)")
ax.set_ylabel("Number of messages")
ax.legend()
Naive Bayes counts how often each word appears in spam and in normal messages. A new message is scored by how spammy its words are, taken together.
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
spam_filter = make_pipeline(CountVectorizer(), MultinomialNB())
Common mistake: giving a text model a table instead of a column of text.
spam_filter.fit(sms[["message"]], sms["label"])
m_train, m_test, l_train, l_test = train_test_split(
sms["message"], sms["label"], test_size=0.25, random_state=42,
stratify=sms["label"])
spam_filter.fit(m_train, l_train)
print(f"Accuracy: {spam_filter.score(m_test, l_test):.1%}")
guess = spam_filter.predict(m_test)
pd.crosstab(l_test, guess, rownames=["actual"], colnames=["predicted"])
Your turn#
Run the filter on the first 4 held-back messages and compare with their labels.
sample = m_test.head(4)
verdicts = spam_filter.predict(sample)
for text, v, truth in zip(sample, verdicts, l_test.head(4)):
print(v, truth, "|", text[:50])
nb = spam_filter[-1]
words = spam_filter[0].get_feature_names_out()
spamminess = nb.feature_log_prob_[1] - nb.feature_log_prob_[0]
pd.Series(spamminess, index=words).nlargest(8).round(2)
Same tools, different costs#
Problem: one score can't serve every field. Choose the metric that matches the worst mistake.
- Recall: of the real positives, how many did we catch?
- Precision: of everything we flagged, how much was really positive?
Quick check: which model lets more of its real positives slip through?
from sklearn.metrics import precision_score
for name, truth, flags, pos in [
("tumours", y_test, cautious, 1),
("spam", l_test, guess, "spam")]:
p = precision_score(truth, flags, pos_label=pos)
r = recall_score(truth, flags, pos_label=pos)
print(f"{name}: precision {p:.3f}, recall {r:.3f}")
The spam filter is built to be sure before it blocks; the tumour model to catch nearly every cancer.
Mini-project: would you switch this spam filter on?#
Check exactly what it gets wrong, then decide.
blocked_real = m_test[(guess == "spam") & (l_test == "ham")]
print(len(blocked_real), "of", (l_test == "ham").sum())
for text in blocked_real:
print("-", text[:75])
missed = (guess == "ham") & (l_test == "spam")
print(missed.sum(), "of", (l_test == "spam").sum())
per_1000 = 1000 / len(m_test)
wrong_block = len(blocked_real) * per_1000
let_through = missed.sum() * per_1000
print(f"Per 1000 texts: {wrong_block:.1f} real blocked, "
f"{let_through:.1f} spam let through")
Decision: switch it on, but let users check a spam folder for the rare wrongly blocked message.
Recap#
- Every application: question, data, method, decision, cost of error
- Business: simple group rates can reveal the riskiest customers
- Healthcare: favour recall, and set the threshold as a policy
- Security: favour precision, and read the actual mistakes
- Clean real messy values first, such as blank text in a numeric column
Challenges#
- Find the churn rate by InternetService. Which service loses the most customers?
- Refit the tumour model on only the 10 'mean' columns. How does recall change?
- Add CountVectorizer(lowercase=False) to the spam filter. Better or worse?
What's next#
Lesson 04: Getting Started with Python, Pandas, and scikit-learn A tour of the toolkit behind every lesson in this series.
Data Mining 04: Getting Started with Python, Pandas, and scikit-learn#
In this lesson you'll be able to:
- Use lists, dictionaries and functions on real data values
- Load, inspect, select and summarise a table with pandas
- Check data for gaps and duplicates before mining it
- Train and test a classifier with scikit-learn's fit, predict, score
- Find groups without labels using k-means clustering Datasets: Iris flowers (data/pandas/iris.csv) and the Titanic passenger list (data/pandas/titanic.csv)
Warm-up: every application starts with a table#
Lesson 03 toured data mining in business, healthcare and security. Each one began with rows (records) and columns (features).
import pandas as pd
iris = pd.read_csv("iris.csv")
print(iris.shape)
iris.head()
Python building blocks on real values#
Problem: hold, count and transform values before pandas takes over.
lengths = iris["petal_length"].head(5).to_list()
print(lengths)
print(type(lengths), len(lengths))
total = sum(lengths)
average = total / len(lengths)
print(round(total, 2), round(average, 2))
counts = iris["species"].value_counts().to_dict()
for name, n in counts.items():
print(f"{name}: {n} flowers")
def cm_to_mm(values):
"""Convert a list of centimetre values to millimetres."""
return [round(v * 10, 1) for v in values]
print(cm_to_mm(lengths))
all_petals = iris["petal_length"].to_list()
big = [v for v in all_petals if v > 6]
print(len(big), max(big))
if len(big) > 5:
print("Several giant petals")
else:
print("Only a handful of giant petals")
Common mistake: expecting a plain list to behave like a pandas column
lengths.mean()
petals = pd.Series(lengths)
print(petals.mean(), petals.max())
Inspect, select and summarise with pandas#
Problem: understand a table's columns before asking it questions.
iris.dtypes
features = ["sepal_length", "sepal_width",
"petal_length", "petal_width"]
iris[features].describe().round(2)
long_petals = iris[iris["petal_length"] > 5]
long_petals["species"].value_counts()
iris.loc[[0, 50, 100], features]
iris["petal_area"] = iris["petal_length"] * iris["petal_width"]
top = iris.sort_values("petal_area", ascending=False)
top[["petal_area", "species"]].head(3).round(2)
Quick check: which species has the longest petals on average?
by_species = iris.groupby("species")[features].mean()
by_species.round(2)
Common mistake: guessing a column name
iris["Petal Length"].mean()
print(iris.columns.to_list())
print(iris["petal_length"].mean().round(2))
Check for gaps and duplicates before mining#
Problem: models fail or mislead on missing and repeated rows.
titanic = pd.read_csv("titanic.csv")
print(titanic.shape)
titanic.isna().sum()
Quick check: if we drop every row with a gap, how many of the 891 passengers survive the cut?
print(len(titanic.dropna()))
titanic = titanic.drop(columns="Cabin")
titanic = titanic.dropna(subset=["Embarked"])
print(titanic.shape)
print(titanic["Age"].isna().sum())
titanic["Age"].describe().round(1)
titanic["Embarked"].value_counts()
print(iris.duplicated().sum())
iris[iris.duplicated(keep=False)]
Your turn: count the duplicated rows in the Titanic table.
print(titanic.duplicated().sum())
print(titanic["PassengerId"].is_unique)
Train and test a classifier with scikit-learn#
Problem: name an iris species from its four measurements.
- X holds the features, y holds the answers
- fit learns from training rows, predict labels new rows
- score checks accuracy on rows the model never saw
from sklearn.model_selection import train_test_split
X = iris[features]
y = iris["species"]
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y)
print(X_train.shape, X_test.shape)
Common mistake: predicting before the model has learned
from sklearn.tree import DecisionTreeClassifier
tree = DecisionTreeClassifier(max_depth=3, random_state=42)
tree.predict(X_test)
tree.fit(X_train, y_train)
print(tree.score(X_test, y_test))
new_rows = X_test.head(3)
print(tree.predict(new_rows))
print(y_test.head(3).to_list())
pred = tree.predict(X_test)
pd.crosstab(y_test, pred, rownames=["true"], colnames=["predicted"])
imp = pd.Series(tree.feature_importances_, index=features)
print(imp.sort_values(ascending=False).round(3))
from sklearn.tree import export_text
print(export_text(tree, feature_names=features))
Your turn: swap the tree for a k-nearest neighbours classifier and score it.
from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train, y_train)
print(knn.score(X_test, y_test))
Find groups without labels using k-means#
Problem: discover natural groups when nobody tells us the answer.
from sklearn.cluster import KMeans
petal = iris[["petal_length", "petal_width"]]
km = KMeans(n_clusters=3, n_init=10, random_state=42)
raw = km.fit_predict(petal)
print(sorted(pd.Series(raw).value_counts().to_list()))
order = petal.groupby(raw)["petal_length"].mean().rank()
iris["cluster"] = pd.Series(raw).map(order.astype(int) - 1)
pd.crosstab(iris["cluster"], iris["species"])
iris.groupby("cluster")[["petal_length", "petal_width"]].mean().round(2)
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(7, 4))
ax.scatter(iris["petal_length"], iris["petal_width"],
c=iris["cluster"], cmap="viridis", alpha=0.7)
ax.set_title("k-means finds three petal groups in the iris data")
ax.set_xlabel("Petal length (cm)")
ax.set_ylabel("Petal width (cm)")
plt.show()
Quick check: we chose 3 clusters because we know there are 3 species. What if we didn't?
for k in range(1, 7):
km_k = KMeans(n_clusters=k, n_init=10, random_state=42)
km_k.fit(petal)
print(k, round(km_k.inertia_, 1))
Mini-project: who survived the Titanic?#
Use class, sex and age to predict survival, and beat a simple baseline.
rates = titanic.groupby(["Sex", "Pclass"])["Survived"].mean()
rates.unstack().round(2)
titanic["is_female"] = (titanic["Sex"] == "female").astype(int)
cols = ["Pclass", "is_female", "Age"]
Xt_train, Xt_test, yt_train, yt_test = train_test_split(
titanic[cols], titanic["Survived"], test_size=0.2,
random_state=42, stratify=titanic["Survived"])
print(Xt_train.shape, Xt_test.shape)
Common mistake: fitting a model on data with gaps
from sklearn.linear_model import LogisticRegression
logit = LogisticRegression(max_iter=1000)
logit.fit(Xt_train, yt_train)
age_median = Xt_train["Age"].median()
Xt_train = Xt_train.fillna({"Age": age_median})
Xt_test = Xt_test.fillna({"Age": age_median})
logit.fit(Xt_train, yt_train)
print(age_median, round(logit.score(Xt_test, yt_test), 3))
baseline = (yt_test == 0).mean()
print(round(baseline, 3))
coefs = pd.Series(logit.coef_[0], index=cols)
print(coefs.round(2))
probs = logit.predict_proba(Xt_test.head(8))[:, 1]
show = Xt_test.head(8).assign(p_survive=probs.round(2))
show.assign(survived=yt_test.head(8))
Recap#
- Lists, dicts and functions hold and transform values; wrap them in pandas for maths
- Check dtypes, gaps and duplicates before any model
- Every scikit-learn model: fit, then predict or score
- Hold out a test set, and fill gaps using training data only
- Clustering finds groups with no labels; relabel them for stable names
Challenges#
- Add Fare to the Titanic features. Does test accuracy change?
- Run k-means on all four iris measurements. Do the clusters match species better?
- Try max_depth=1 and max_depth=5 for the iris tree. What happens to the score?
What's next#
Lesson 05: Introduction to Exploratory Data Mining Explore 9,835 real grocery baskets and find which items go together.
Data Mining 05: Introduction to Exploratory Data Mining#
In this lesson you'll be able to:
- Load ragged transaction data into a tidy pandas table
- Profile baskets and items with counts, shares and support
- Measure which items go together with confidence and lift
- Compare patterns across segments of the data
- Let the Apriori algorithm search for rules automatically Dataset: 9,835 real grocery baskets (data/mining/groceries_baskets.csv)
Warm-up: Python lists meet real baskets#
Lesson 04 used lists, dictionaries and pandas. Same tools, new data.
import pandas as pd
with open("groceries_baskets.csv") as f:
lines = f.read().splitlines()
print(len(lines))
for line in lines[:3]:
print(line)
Load ragged transactions into a tidy table#
Problem: each basket has a different number of items.
Common mistake: reading ragged lines with a plain read_csv
pd.read_csv("groceries_baskets.csv", header=None)
baskets = [line.split(",") for line in lines]
print(baskets[0])
print(max(len(b) for b in baskets))
repeats = sum(len(b) != len(set(b)) for b in baskets)
print(repeats)
rows = [(i, item) for i, basket in enumerate(baskets)
for item in basket]
long = pd.DataFrame(rows, columns=["basket", "item"])
print(long.shape)
long.head()
import os
long.to_csv("m05_baskets_long.csv", index=False)
check = pd.read_csv("m05_baskets_long.csv")
print(check.shape, check.columns.to_list())
Profile the baskets: how much do people buy?#
Problem: know what a typical shopping trip looks like.
sizes = long.groupby("basket").size()
sizes.describe().round(2)
Quick check: what share of baskets hold just one item?
print(round((sizes == 1).mean(), 3))
sizes.value_counts().sort_index().head(8)
biggest = sizes.idxmax()
print(biggest, sizes[biggest])
print(baskets[biggest][:6])
import matplotlib.pyplot as plt
ax = sizes.plot(kind="hist", bins=range(1, 34), figsize=(7, 4),
edgecolor="white")
ax.set_title("Most grocery baskets are small")
ax.set_xlabel("Items in basket")
ax.set_ylabel("Number of baskets")
plt.show()
Your turn: what share of baskets have 10 or more items?
big = sizes >= 10
print(big.sum(), round(big.mean(), 3))
Profile the items: popularity and support#
Problem: find the best sellers and the long tail of rare items.
print(long["item"].nunique())
long["item"].value_counts().head(5)
- Support = share of baskets that contain an item
- Support of 0.25 means one basket in four
n_baskets = len(baskets)
per_basket = long.drop_duplicates()["item"].value_counts()
support = per_basket / n_baskets
support.head(8).round(3)
Common mistake: trusting that text values are clean
support["cream cheese"]
print([s for s in support.index if s != s.strip()])
long["item"] = long["item"].str.strip()
per_basket = long.drop_duplicates()["item"].value_counts()
support = per_basket / n_baskets
print(round(support["cream cheese"], 3))
support.describe().round(3)
rare = support < 0.01
print(rare.sum(), len(support))
print(support.tail(3).round(5))
shares = long["item"].value_counts(normalize=True)
print(round(shares.head(20).sum(), 3))
ax = support.head(10).sort_values().plot(kind="barh",
figsize=(7, 4))
ax.set_title("Top 10 grocery items by support")
ax.set_xlabel("Share of baskets containing the item")
ax.set_ylabel("Item")
plt.show()
Your turn: find the support of beef and its rank among all items.
rank = support.rank(ascending=False, method="min")
print(round(support["beef"], 3), int(rank["beef"]))
Find items that go together: confidence and lift#
Problem: tell a real link from two items that are simply popular.
onehot = pd.crosstab(long["basket"], long["item"]) > 0
print(onehot.shape)
onehot[["whole milk", "yogurt", "soda"]].head(3)
both = (onehot["yogurt"] & onehot["whole milk"]).mean()
conf = both / support["yogurt"]
lift = conf / support["whole milk"]
print(round(both, 3), round(conf, 3), round(lift, 2))
def pair_stats(a, b):
"""Support, confidence and lift for the rule a -> b."""
both = (onehot[a] & onehot[b]).mean()
conf = both / support[a]
lift = conf / support[b]
return round(float(both), 3), round(float(conf), 3), round(lift, 2)
print(*pair_stats("root vegetables", "other vegetables"))
print(*pair_stats("whole milk", "yogurt"))
print(*pair_stats("yogurt", "whole milk"))
Quick check: soda is the 4th most popular item. Does soda make whole milk more likely?
print(*pair_stats("soda", "whole milk"))
Your turn: test the rule bottled beer -> whole milk.
print(*pair_stats("bottled beer", "whole milk"))
top = support.head(8).index
m = onehot[top].astype(int)
pair_support = (m.T @ m) / n_baskets
expected = support[top].to_frame() @ support[top].to_frame().T
lift_table = (pair_support / expected).round(2)
lift_table.iloc[:4, :4]
import numpy as np
import seaborn as sns
lift_table = lift_table.mask(np.eye(len(top), dtype=bool))
fig, ax = plt.subplots(figsize=(8, 6))
sns.heatmap(lift_table, annot=True, cmap="RdBu_r", center=1, ax=ax)
ax.set_title("Lift between the 8 best-selling items")
ax.set_xlabel("Then item")
ax.set_ylabel("If item")
plt.show()
Check a pattern across segments#
Problem: an overall rate can hide very different groups.
Common mistake: a label list that doesn't match the bin edges
pd.cut(sizes, bins=[0, 1, 3, 6, 32], labels=["1", "2-3", "4+"])
bands = pd.cut(sizes, bins=[0, 1, 3, 6, 32],
labels=["1", "2-3", "4-6", "7+"])
milk = onehot["whole milk"].groupby(bands, observed=True).mean()
print(bands.value_counts().sort_index())
print(milk.round(3))
ax = milk.plot(kind="bar", rot=0, figsize=(7, 4))
ax.set_title("Bigger baskets are far more likely to hold milk")
ax.set_xlabel("Items in basket")
ax.set_ylabel("Share of baskets with whole milk")
plt.show()
for band in ["2-3", "4-6", "7+"]:
seg = onehot[bands == band]
both = (seg["yogurt"] & seg["whole milk"]).mean()
exp = seg["yogurt"].mean() * seg["whole milk"].mean()
print(band, round(both / exp, 2))
Your turn: repeat the band comparison for bottled water.
water = onehot["bottled water"].groupby(bands, observed=True)
print(water.mean().round(3))
Mini-project: what should sit next to the vegetables?#
Let Apriori search every combination and rank the rules by lift.
from mlxtend.frequent_patterns import apriori, association_rules
freq = apriori(onehot, min_support=0.02, use_colnames=True)
print(len(freq))
freq.sort_values("support", ascending=False).head(3).round(3)
rules = association_rules(freq, metric="lift", min_threshold=1.5)
rules["if"] = rules["antecedents"].apply(sorted).str.join(", ")
rules["then"] = rules["consequents"].apply(sorted).str.join(", ")
rules = rules.round(3)
print(len(rules))
sure = rules[rules["confidence"] >= 0.4]
print(len(sure))
sure["then"].value_counts()
cols = ["if", "then", "support", "confidence", "lift"]
veg = rules[rules["then"] == "root vegetables"]
veg.sort_values(["lift", "if"], ascending=[False, True])[cols]
best = rules.sort_values(["lift", "if"], ascending=[False, True])
top_rule = best.iloc[0]
print(f"If {top_rule['if']} then {top_rule['then']}:")
print(f"lift {top_rule['lift']}, confidence "
f"{top_rule['confidence']:.0%}")
fig, ax = plt.subplots(figsize=(7, 4))
pts = ax.scatter(rules["support"], rules["confidence"],
c=rules["lift"], cmap="viridis")
fig.colorbar(pts, ax=ax, label="Lift")
ax.set_title("76 grocery rules: support vs confidence")
ax.set_xlabel("Support (share of baskets)")
ax.set_ylabel("Confidence")
plt.show()
Recap#
- Ragged transactions: split lines, then build a long table
- Profile first: basket sizes, item support, the long tail
- Clean text labels before looking them up
- Confidence alone misleads; lift above 1 means a real link
- Check patterns per segment, then let Apriori search at scale
Challenges#
- Find the three items with the highest lift alongside yogurt.
- Rerun Apriori with min_support=0.01. How many rules now?
- Compare the shopping bags rate across basket size bands.
What's next#
Lesson 06: Summarization and Descriptive Statistics Mean, median, spread and shape, on real customer data.
Data Mining 06: Summarization and Descriptive Statistics#
In this lesson you'll be able to:
- Choose between mean, median and mode for the centre
- Measure spread with range, standard deviation and quartiles
- Describe shape with skew and flag outliers with the IQR rule
- Summarise categories with counts, shares and crosstabs
- Build group summaries that profile customers Datasets: IBM Telco customer churn (data/mining/telco_churn.csv) and grocery baskets (data/mining/groceries_baskets.csv)
Warm-up: basket sizes from lesson 05#
Lesson 05 explored grocery baskets. One number there deserves a second look.
import pandas as pd
with open("groceries_baskets.csv") as f:
lines = f.read().splitlines()
sizes = pd.Series([len(line.split(",")) for line in lines])
print(round(sizes.mean(), 2), sizes.median())
Load the data and check types before summarising#
Problem: statistics only work on columns that are really numbers.
df = pd.read_csv("telco_churn.csv")
print(df.shape)
cols = ["tenure", "MonthlyCharges", "TotalCharges", "Churn"]
df[cols].head()
df[cols].dtypes
Common mistake: averaging a column that only looks numeric
df["TotalCharges"].mean()
df["TotalCharges"] = pd.to_numeric(df["TotalCharges"],
errors="coerce")
gaps = df[df["TotalCharges"].isna()]
print(len(gaps))
print(gaps["tenure"].unique())
print(round(df["TotalCharges"].mean(), 2))
print(df["TotalCharges"].count(), len(df))
print(df.select_dtypes("number").columns.to_list())
Measure the centre: mean, median and mode#
Problem: one number for a typical value, without being fooled.
- Mean: the total shared out equally; pulled by extremes
- Median: the middle value once sorted; resists extremes
- Mode: the most common value; works for categories too
bill = df["MonthlyCharges"]
print(round(bill.mean(), 2), bill.median())
print(df["tenure"].mode().to_list())
df["tenure"].value_counts().head(3)
Quick check: for TotalCharges, will the mean be above or below the median?
total = df["TotalCharges"]
print(round(total.mean(), 2), round(total.median(), 2))
from scipy import stats
trimmed = stats.trim_mean(total.dropna(), 0.1)
print(round(trimmed, 2))
cut = sizes[sizes <= sizes.quantile(0.99)]
print(len(sizes) - len(cut))
print(round(cut.mean(), 2), cut.median())
print(df["PaymentMethod"].mode()[0])
Measure the spread: range, standard deviation, quartiles#
Problem: two groups can share an average yet behave very differently.
print(bill.min(), bill.max())
print(round(bill.max() - bill.min(), 2))
print(round(bill.std(), 2), round(bill.var(), 2))
import numpy as np
print(round(bill.std(), 4), round(np.std(bill), 4))
q = bill.quantile([0.1, 0.25, 0.5, 0.75, 0.9])
print(q.round(2))
print(round(q[0.75] - q[0.25], 2))
num = ["tenure", "MonthlyCharges", "TotalCharges"]
cv = df[num].std() / df[num].mean()
cv.round(2)
Your turn: find the interquartile range of tenure.
q1, q3 = df["tenure"].quantile([0.25, 0.75])
print(q1, q3, q3 - q1)
df.groupby("Churn")["MonthlyCharges"].agg(["mean", "std"]).round(2)
df[num].describe().round(2)
Describe shape and flag outliers#
Problem: know whether data is lopsided, and which values are extreme.
print(df[num].skew().round(2))
print(round(sizes.skew(), 2))
import matplotlib.pyplot as plt
ax = df["tenure"].plot(kind="hist", bins=36, figsize=(7, 4),
edgecolor="white")
ax.set_title("Customer tenure piles up at both ends")
ax.set_xlabel("Tenure (months)")
ax.set_ylabel("Number of customers")
plt.show()
ax = bill.plot(kind="hist", bins=30, figsize=(7, 4),
edgecolor="white")
ax.set_title("Monthly bills: a cheap cluster and a broad hump")
ax.set_xlabel("Monthly charge")
ax.set_ylabel("Number of customers")
plt.show()
- IQR rule: outliers lie below Q1 - 1.5 x IQR or above Q3 + 1.5 x IQR
def iqr_outliers(s):
"""Return the values outside the 1.5 x IQR fences."""
q1, q3 = s.quantile([0.25, 0.75])
low, high = q1 - 1.5 * (q3 - q1), q3 + 1.5 * (q3 - q1)
return s[(s < low) | (s > high)]
out = iqr_outliers(sizes)
print(len(out), out.min())
Quick check: how many telco customers will the IQR rule flag in each numeric column?
for col in num:
print(col, len(iqr_outliers(df[col])))
z = (bill - bill.mean()) / bill.std()
print(round(z.min(), 2), round(z.max(), 2))
print((z.abs() > 2).sum())
Summarise categories with counts, shares and crosstabs#
Problem: most columns here are text, and they need summaries too.
df["Churn"].value_counts()
Common mistake: taking the mean of a Yes/No column
df["Churn"].mean()
df["churned"] = (df["Churn"] == "Yes").astype(int)
print(round(df["churned"].mean(), 3))
print(df["Churn"].value_counts(normalize=True).round(3))
print(df["SeniorCitizen"].value_counts())
print(round(df["SeniorCitizen"].mean(), 3))
df.groupby("gender")["churned"].agg(["size", "mean"]).round(3)
df["Contract"].value_counts()
pd.crosstab(df["Contract"], df["Churn"], normalize="index").round(3)
pd.crosstab(df["Contract"], df["Churn"], margins=True)
Your turn: find the churn rate for each payment method.
pay = df.groupby("PaymentMethod")["churned"].mean()
pay.sort_values(ascending=False).round(3)
Summarise by group with groupby and agg#
Problem: compare whole profiles across segments in one table.
internet = df.groupby("InternetService")["MonthlyCharges"]
internet.agg(["count", "mean", "median", "std"]).round(2)
df.groupby("Churn")["tenure"].describe().round(1)
Quick check: churners pay more each month. Do they also have bigger total charges?
df.groupby("Churn")[num].median().round(1)
rates = df.pivot_table(index="Contract", columns="InternetService",
values="churned", aggfunc="mean")
rates.round(3)
Mini-project: profile the customers who leave#
Build one summary table a retention team could act on.
df["monthly"] = df["Contract"] == "Month-to-month"
df["fibre"] = df["InternetService"] == "Fiber optic"
df["e_check"] = df["PaymentMethod"] == "Electronic check"
print(df[["monthly", "fibre", "e_check"]].mean().round(3))
profile = df.groupby("Churn").agg(
customers=("churned", "size"),
median_tenure=("tenure", "median"),
median_bill=("MonthlyCharges", "median"),
month_to_month=("monthly", "mean"),
fibre=("fibre", "mean"),
e_check=("e_check", "mean"),
)
profile.round(2)
import os
profile.round(3).to_csv("m06_churn_profile.csv")
saved = pd.read_csv("m06_churn_profile.csv")
print(saved.shape)
fig, ax = plt.subplots(figsize=(7, 4))
df.boxplot(column="tenure", by="Churn", ax=ax)
ax.set_title("Leavers are mostly new customers")
fig.suptitle("")
ax.set_xlabel("Churned")
ax.set_ylabel("Tenure (months)")
plt.show()
new = df[df["tenure"] <= 12]
print(len(new), round(new["churned"].mean(), 3))
print(round(df[df["tenure"] > 12]["churned"].mean(), 3))
Recap#
- Check dtypes first; convert text numbers with to_numeric
- Mean vs median: a big gap signals skew and extremes
- Spread: std, quartiles and IQR; CV compares across units
- Categories: value_counts, normalize, and crosstab
- groupby with agg turns statistics into customer profiles
Challenges#
- Compare mean and median MonthlyCharges for senior citizens vs others.
- Build a crosstab of InternetService and Churn with row shares.
- Use the IQR rule on TotalCharges for churners only.
What's next#
Lesson 07: Visualization - Histograms, Scatterplots, Heatmaps Turn these summaries into charts that reveal patterns at a glance.
Data Mining 07: Visualization- Histograms, Scatterplots, Heatmaps#
In this lesson you'll be able to:
- Read the shape of a column with a histogram, and pick good bins
- Tame skewed columns with a log scale and compare groups fairly
- Show how two columns move together with a scatterplot
- Turn a scatterplot into a map, coloured by a third column
- Summarise many numbers at once with a heatmap Dataset: California housing, 1990 census (data/pandas/housing.csv)
Warm-up: numbers before pictures#
Lesson 06 summarised columns with statistics. Same idea, new data.
import pandas as pd
df = pd.read_csv("housing.csv")
print(df.shape)
df[["median_income", "median_house_value", "ocean_proximity"]].head()
cols = ["median_income", "median_house_value"]
df[cols].describe().round(2)
Histograms: see the shape of one column#
Problem: a mean and a median hide the shape. Where do values pile up?
- Cut the range into equal-width bins
- Draw one bar per bin; its height is how many rows fall inside
import matplotlib.pyplot as plt
ax = df["median_income"].plot(kind="hist", bins=30,
edgecolor="white", figsize=(7, 4))
ax.set_title("Income across California districts")
ax.set_xlabel("Median income (tens of thousands of dollars)")
ax.set_ylabel("Number of districts")
plt.show()
fig, axes = plt.subplots(1, 2, figsize=(10, 4))
for ax, n in zip(axes, [5, 80]):
df["median_income"].plot(kind="hist", bins=n, ax=ax)
ax.set_title(f"{n} bins")
ax.set_xlabel("Median income")
ax.set_ylabel("Districts")
plt.tight_layout()
import numpy as np
counts, edges = np.histogram(df["median_income"], bins=5)
print(counts)
print(edges.round(2))
ax = df["median_house_value"].plot(kind="hist", bins=50,
edgecolor="white", figsize=(7, 4))
ax.set_title("Median home value per district")
ax.set_xlabel("Median home value (dollars)")
ax.set_ylabel("Number of districts")
Quick check: what is the tall bar at the far right of that chart?
top = df["median_house_value"].max()
print(top)
print((df["median_house_value"] == top).sum())
Your turn#
Draw a histogram of housing_median_age with 52 bins. Is it capped too?
ax = df["housing_median_age"].plot(kind="hist", bins=52,
figsize=(7, 4))
ax.set_title("Median age of homes per district")
ax.set_xlabel("Median age of homes (years)")
ax.set_ylabel("Number of districts")
print(df["housing_median_age"].value_counts().head(3))
Skewed columns and the log scale#
Problem: a few huge districts squash every other bar into one corner.
ax = df["population"].plot(kind="hist", bins=50, figsize=(7, 4))
ax.set_title("Population per district")
ax.set_xlabel("People living in the district")
ax.set_ylabel("Number of districts")
skew_cols = ["population", "total_rooms", "median_income"]
df[skew_cols].skew().round(2)
import seaborn as sns
ax = sns.histplot(data=df, x="population", bins=50, log_scale=True)
ax.set_title("Population per district (log scale)")
ax.set_xlabel("People living in the district")
ax.set_ylabel("Number of districts")
Your turn#
Draw total_rooms on a log scale. Where is the typical district?
ax = sns.histplot(data=df, x="total_rooms", bins=50, log_scale=True)
ax.set_title("Rooms per district (log scale)")
ax.set_xlabel("Total rooms in the district")
ax.set_ylabel("Number of districts")
print(df["total_rooms"].median())
Compare groups with overlapping histograms#
Problem: do inland and coastal districts have different home values?
df["ocean_proximity"].value_counts()
Common mistake: a column name with the wrong capital letter
sns.histplot(data=df, x="median_house_value", hue="Ocean_proximity")
two = df[df["ocean_proximity"].isin(["INLAND", "<1H OCEAN"])]
ax = sns.histplot(data=two, x="median_house_value",
hue="ocean_proximity", stat="density",
common_norm=False, element="step")
ax.set_title("Home values: inland vs within an hour of the ocean")
ax.set_xlabel("Median home value (dollars)")
ax.set_ylabel("Density")
by_loc = df.groupby("ocean_proximity")["median_house_value"]
by_loc.median().sort_values()
Scatterplots: how two columns move together#
Problem: do richer districts have pricier homes, and how tightly?
ax = df.plot(kind="scatter", x="median_income",
y="median_house_value", figsize=(7, 4))
ax.set_title("Income vs home value, every district")
ax.set_xlabel("Median income (tens of thousands of dollars)")
ax.set_ylabel("Median home value (dollars)")
ax = df.plot(kind="scatter", x="median_income",
y="median_house_value", alpha=0.1, s=5,
figsize=(7, 4))
ax.set_title("Income vs home value, with transparency")
ax.set_xlabel("Median income (tens of thousands of dollars)")
ax.set_ylabel("Median home value (dollars)")
ax = df.plot(kind="hexbin", x="median_income",
y="median_house_value", gridsize=30,
cmap="Blues", figsize=(7, 4))
ax.set_title("Income vs home value, counted in hexagons")
ax.set_xlabel("Median income (tens of thousands of dollars)")
ax.set_ylabel("Median home value (dollars)")
r = df["median_income"].corr(df["median_house_value"])
print(round(r, 2))
Quick check: what is the flat row of dots along the top of the chart?
rich = df[df["median_income"] >= 8]
capped = (rich["median_house_value"] == top).mean()
print(len(rich), round(capped, 2))
Common mistake: x and y with different lengths
plt.scatter(df["median_income"], df["median_house_value"].head(500))
sample = df.sample(500, random_state=42)
plt.scatter(sample["median_income"], sample["median_house_value"],
alpha=0.5)
plt.title("Income vs home value, 500 sampled districts")
plt.xlabel("Median income (tens of thousands of dollars)")
plt.ylabel("Median home value (dollars)")
plt.show()
A scatterplot that is a map#
Problem: where in the state are the expensive districts?
ax = df.plot(kind="scatter", x="longitude", y="latitude",
c="median_house_value", cmap="viridis",
s=df["population"] / 200, alpha=0.4,
colorbar=True, figsize=(8, 6))
ax.set_title("California districts coloured by home value")
ax.set_xlabel("Longitude")
ax.set_ylabel("Latitude")
dearest = df.nlargest(200, "median_house_value")
dearest["ocean_proximity"].value_counts()
Your turn#
Redraw the map with seaborn, coloured by ocean_proximity instead.
ax = sns.scatterplot(data=df, x="longitude", y="latitude",
hue="ocean_proximity", s=5, alpha=0.5)
ax.set_title("California districts by distance to the ocean")
ax.set_xlabel("Longitude")
ax.set_ylabel("Latitude")
Heatmaps: a whole table at a glance#
Problem: compare every pair of columns at once, not one scatter at a time.
Common mistake: asking for correlations with a text column inside
corr = df.corr()
corr = df.corr(numeric_only=True)
corr["median_house_value"].sort_values().round(2)
plt.figure(figsize=(9, 7))
sns.heatmap(corr, annot=True, fmt=".2f", cmap="coolwarm",
vmin=-1, vmax=1)
plt.title("Correlation between housing columns")
plt.show()
Quick check: which pair of different columns is most strongly linked?
upper = np.triu(np.ones(corr.shape, dtype=bool), k=1)
pairs = corr.where(upper).stack()
pairs.sort_values(ascending=False).head(3).round(2)
Mini-project: does the coast add value beyond income?#
Plan: income bands, a pivot table of median values, a heatmap, a finding.
edges = [0, 2, 3, 4, 5, 6, 16]
names = ["<20k", "20-30k", "30-40k", "40-50k", "50-60k", "60k+"]
df["income_band"] = pd.cut(df["median_income"], bins=edges,
labels=names)
df["income_band"].value_counts(sort=False)
table = df.pivot_table(index="ocean_proximity", columns="income_band",
values="median_house_value",
aggfunc="median", observed=True)
table = table.drop(index="ISLAND")
(table / 1000).round(0)
plt.figure(figsize=(9, 4))
sns.heatmap(table / 1000, annot=True, fmt=".0f", cmap="YlGnBu")
plt.title("Median home value ($1000s) by location and income")
plt.xlabel("Income band")
plt.ylabel("Location")
plt.show()
gap = table.loc["<1H OCEAN"] - table.loc["INLAND"]
(gap / 1000).round(0)
ratio = table.loc["<1H OCEAN"] / table.loc["INLAND"]
print(f"Coastal homes cost {ratio.min():.1f} to "
f"{ratio.max():.1f} times inland, at equal income")
Recap#
- Histograms show the shape of one column; try a few bin counts
- Spikes at the edge often mean capped values; check before modelling
- Use a log scale for skewed columns and density to compare groups
- Scatterplots need transparency or sampling when points pile up
- Heatmaps summarise a correlation matrix or a pivot table at once
Challenges#
- Histogram of households on a log scale, coloured by ocean_proximity.
- Scatter housing_median_age against median_house_value. Any link?
- Pivot median income by location and age band, then draw a heatmap.
What's next#
Lesson 08: Exploratory Data Analysis and Visualization Basics Combine these charts into a full EDA workflow on a new dataset.
Data Mining 08: Exploratory Data Analysis and Visualization Basics#
In this lesson you'll be able to:
- Follow a step-by-step EDA checklist on a brand-new dataset
- Catch columns that only look numeric, before they bite
- Profile numeric and text columns with statistics and charts
- Compare every column against the target to find strong signals
- Combine two features into a heatmap of risk segments Dataset: IBM Telco customer churn (data/mining/telco_churn.csv)
Warm-up: one histogram#
Lesson 07 drew histograms, scatterplots and heatmaps. One more, on new data.
import pandas as pd
import matplotlib.pyplot as plt
df = pd.read_csv("telco_churn.csv")
print(df.shape)
df[["customerID", "tenure", "Contract", "Churn"]].head()
ax = df["tenure"].plot(kind="hist", bins=36, edgecolor="white",
figsize=(7, 4))
ax.set_title("How long customers have stayed")
ax.set_xlabel("Tenure (months)")
ax.set_ylabel("Number of customers")
Step 1: first look at size, types and the target#
Problem: know what you're holding before you trust a single number.
EDA checklist:
- First look: size, types, duplicates, target
- One column at a time: numbers, then categories
- Each column against the target
- Columns together: correlations and segments
df.dtypes.value_counts()
print(df["customerID"].duplicated().sum())
print(df.isna().sum().sum())
Common mistake: trusting a column that only looks numeric
df["TotalCharges"].astype(float)
tc = pd.to_numeric(df["TotalCharges"], errors="coerce")
print(tc.isna().sum())
df.loc[tc.isna(), ["customerID", "tenure", "TotalCharges"]].head()
df["TotalCharges"] = tc.fillna(0)
print(df["TotalCharges"].describe().round(2))
print(df["Churn"].value_counts())
df["churned"] = (df["Churn"] == "Yes").astype(int)
print(round(df["churned"].mean(), 3))
Step 2: numeric columns, one at a time#
Problem: learn each number's range, centre and shape.
num = ["tenure", "MonthlyCharges", "TotalCharges"]
df[num].describe().round(2)
axes = df[num].hist(bins=30, figsize=(12, 3.5), layout=(1, 3))
for ax in axes.ravel():
ax.set_xlabel(ax.get_title())
ax.set_ylabel("Customers")
plt.suptitle("Numeric columns at a glance", y=1.05)
counts = df["tenure"].value_counts()
print(counts[1], counts[72])
print(df["tenure"].max())
Quick check: who makes up the tall spike near 20 dollars in MonthlyCharges?
df.groupby("InternetService")["MonthlyCharges"].median()
df[num].skew().round(2)
Step 3: categorical columns#
Problem: see which categories exist, and which column is just an ID.
df.nunique().sort_values().tail(4)
df["Contract"].value_counts()
ax = df["Contract"].value_counts().plot(kind="bar", rot=0,
figsize=(7, 4))
ax.set_title("Customers by contract type")
ax.set_xlabel("Contract")
ax.set_ylabel("Number of customers")
three = ["InternetService", "TechSupport", "PaperlessBilling"]
fig, axes = plt.subplots(1, 3, figsize=(12, 3.5))
for ax, col in zip(axes, three):
df[col].value_counts().plot(kind="bar", ax=ax, rot=0)
ax.set_title(col)
ax.set_ylabel("Customers")
plt.tight_layout()
Quick check: TechSupport has a third category. Where does it come from?
pd.crosstab(df["InternetService"], df["TechSupport"])
Your turn#
What share of customers uses each PaymentMethod? Round to 3 places.
df["PaymentMethod"].value_counts(normalize=True).round(3)
Step 4: numbers against the target#
Problem: do customers who leave look different from those who stay?
Common mistake: averaging a whole table that contains text
df.groupby("Churn").mean()
df.groupby("Churn")[num].median().round(2)
import seaborn as sns
ax = sns.boxplot(data=df, x="Churn", y="tenure")
ax.set_title("Tenure of customers who stayed vs left")
ax.set_xlabel("Left the company?")
ax.set_ylabel("Tenure (months)")
ax = sns.histplot(data=df, x="MonthlyCharges", hue="Churn", bins=30,
stat="density", common_norm=False,
element="step")
ax.set_title("Monthly charges: stayed vs left")
ax.set_xlabel("Monthly charge (dollars)")
ax.set_ylabel("Density")
pts = df.sample(1000, random_state=42)
ax = sns.scatterplot(data=pts, x="tenure", y="MonthlyCharges",
hue="Churn", alpha=0.6)
ax.set_title("Tenure vs monthly charge, 1,000 sampled customers")
ax.set_xlabel("Tenure (months)")
ax.set_ylabel("Monthly charge (dollars)")
Your turn#
Draw a boxplot of TotalCharges for stayers and leavers.
ax = sns.boxplot(data=df, x="Churn", y="TotalCharges")
ax.set_title("Total charges of customers who stayed vs left")
ax.set_xlabel("Left the company?")
ax.set_ylabel("Total charges (dollars)")
print(df.groupby("Churn")["TotalCharges"].median())
left = df[df["churned"] == 1]
early = (left["tenure"] <= 12).mean()
print(len(left), round(early, 2))
Step 5: categories against the target#
Problem: which categories have the highest churn rate?
rate = df.groupby("Contract")["churned"].mean()
rate.round(3)
ax = rate.plot(kind="bar", rot=0, figsize=(7, 4))
ax.axhline(df["churned"].mean(), color="red", linestyle="--",
label="All customers")
ax.set_title("Churn rate by contract type")
ax.set_xlabel("Contract")
ax.set_ylabel("Share who left")
ax.legend()
mix = pd.crosstab(df["InternetService"], df["Churn"],
normalize="index")
mix.round(3)
ax = mix.plot(kind="bar", stacked=True, rot=0, figsize=(7, 4))
ax.set_title("Stayed vs left by internet service")
ax.set_xlabel("Internet service")
ax.set_ylabel("Share of customers")
Quick check: which payment method has the highest churn rate?
pay = df.groupby("PaymentMethod")["churned"].agg(["size", "mean"])
pay.sort_values("mean").round(3)
cats = ["Contract", "InternetService", "PaymentMethod",
"TechSupport", "PaperlessBilling", "gender"]
for col in cats:
r = df.groupby(col)["churned"].mean()
print(f"{col:16} spread {r.max() - r.min():.2f}")
Your turn#
Find the churn rate and head count for SeniorCitizen (0 or 1).
df.groupby("SeniorCitizen")["churned"].agg(["size", "mean"]).round(3)
Step 6: numbers together with correlations#
Problem: spot columns that repeat each other, and their link to churn.
corr = df[num + ["churned"]].corr().round(2)
corr
plt.figure(figsize=(6, 5))
sns.heatmap(corr, annot=True, cmap="coolwarm", vmin=-1, vmax=1)
plt.title("Correlation of numeric columns with churn")
plt.show()
Quick check: why are TotalCharges and tenure so strongly linked?
paying = df[df["tenure"] > 0]
approx = paying["tenure"] * paying["MonthlyCharges"]
print((paying["TotalCharges"] / approx).median().round(3))
Mini-project: who is most at risk of leaving?#
Combine the two strongest signals, contract and tenure, into segments.
bands = ["0-12", "13-24", "25-48", "49-72"]
df["tenure_band"] = pd.cut(df["tenure"], bins=[0, 12, 24, 48, 72],
labels=bands, include_lowest=True)
df["tenure_band"].value_counts(sort=False)
seg = df.pivot_table(index="Contract", columns="tenure_band",
values="churned", aggfunc="mean",
observed=True)
seg.round(2)
pd.crosstab(df["Contract"], df["tenure_band"])
plt.figure(figsize=(8, 4))
sns.heatmap(seg, annot=True, fmt=".2f", cmap="Reds")
plt.title("Churn rate by contract and tenure")
plt.xlabel("Tenure band (months)")
plt.ylabel("Contract")
plt.show()
risky = (df["Contract"] == "Month-to-month") & (df["tenure"] <= 12)
share = df.loc[risky, "churned"].sum() / df["churned"].sum()
print(f"{risky.sum()} customers, {risky.mean():.0%} of the base")
print(f"churn {df.loc[risky, 'churned'].mean():.0%}, "
f"{share:.0%} of all leavers")
Recap#
- Follow a checklist: first look, single columns, vs target, together
- Check types early: to_numeric with errors='coerce' exposes hidden blanks
- A 0/1 target makes groupby means into churn rates
- Compare groups with boxplots and density histograms
- Pivot two strong signals into a heatmap to find risk segments
Challenges#
- Draw a boxplot of MonthlyCharges by Contract.
- Rank OnlineSecurity, StreamingTV and Partner by churn-rate spread.
- Build a heatmap of churn by InternetService and tenure band.
What's next#
Lesson 09: Data Cleaning- Handling Missing Values, Noise, and Outliers Find hidden missing values, fill them sensibly, and tame outliers.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.



