Lesson 51 · Pandas Full Course
Scaling and Normalizing Data in Pandas and scikit-learn (Do It Right) | Pandas Course #51
In this lesson you'll be able to: Explain why features on different scales mislead a model Min-max scale and standardise columns with plain pandas Use…
- CoursePandas Full Course
- Lesson51 of 65
- FormatJupyter notebook · 40 code cells
- Data2 datasets
What you'll learn
- Warm-up: encoding, from lesson 50
- Why scale? Big numbers shout loudest
- Min-max scaling: squeeze every column into 0 to 1
- Standardisation: how many standard deviations from average?
- The scikit-learn scalers
- Outliers and skew: robust scaling and logs
- Fit on training data only
- Mini-project: does scaling help a model tell species apart?
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
- penguins.csv preview13.2 KB
- housing.csv preview1.4 MB
📓 Full notebook
Download .ipynbPandas 51: Scaling and Normalizing Data#
In this lesson you'll be able to:
- Explain why features on different scales mislead a model
- Min-max scale and standardise columns with plain pandas
- Use scikit-learn's StandardScaler, MinMaxScaler and RobustScaler
- Tame outliers and skew with robust scaling and a log transform
- Fit a scaler on training data only, and prove scaling helps Data: Palmer penguins, plus California housing for outliers
Warm-up: encoding, from lesson 50#
Lesson 50 turned text categories into numbers. Same idea, penguins.
import pandas as pd
pens = pd.read_csv("data/pandas/penguins.csv")
print(pens.shape)
pens.head()
dummies = pd.get_dummies(pens["island"], dtype=int)
dummies.sum()
Why scale? Big numbers shout loudest#
Problem: columns measured in different units can't be compared fairly.
num = ["bill_length_mm", "bill_depth_mm",
"flipper_length_mm", "body_mass_g"]
pens = pens.dropna(subset=num)
pens[num].agg(["min", "max", "mean"]).round(1)
a = pens.loc[0, num]
b = pens.loc[1, num]
gap = (a - b).abs()
gap
Quick check: which column decides the distance between these two birds?
sq = gap ** 2
print(round(sq.sum() ** 0.5, 1))
(sq / sq.sum()).round(3)
Scaling puts every column on a similar range, so each gets a fair vote.
Min-max scaling: squeeze every column into 0 to 1#
Formula: (x - min) / (max - min). Smallest becomes 0, largest becomes 1.
mass = pens["body_mass_g"]
mass_mm = (mass - mass.min()) / (mass.max() - mass.min())
print(mass.head(3).tolist())
print(mass_mm.head(3).round(3).tolist())
feats = pens[num]
mm = (feats - feats.min()) / (feats.max() - feats.min())
mm.agg(["min", "max"])
mm.head().round(3)
Common mistake: scaling the whole table, text columns included
(pens - pens.min()) / (pens.max() - pens.min())
nums = pens.select_dtypes("number")
print(nums.columns.tolist())
((nums - nums.min()) / (nums.max() - nums.min())).max()
Your turn#
Min-max scale flipper length. Which penguin scores exactly 1?
fl = pens["flipper_length_mm"]
fl_mm = (fl - fl.min()) / (fl.max() - fl.min())
pens.loc[fl_mm == 1, ["species", "island", "flipper_length_mm"]]
Standardisation: how many standard deviations from average?#
Formula: (x - mean) / std. The result is called a z-score.
z = (feats - feats.mean()) / feats.std()
z.head().round(2)
z.agg(["mean", "std"]).round(3)
Quick check: how far above average is the heaviest penguin?
heavy = z["body_mass_g"].idxmax()
print(pens.loc[heavy, "species"], pens.loc[heavy, "body_mass_g"])
print(round(z.loc[heavy, "body_mass_g"], 2))
unusual = (z.abs() > 2).sum()
unusual
zgap = (z.loc[0] - z.loc[1]) ** 2
(zgap / zgap.sum()).round(3)
ax = z.plot(kind="box", figsize=(8, 3.5), rot=0)
ax.set_title("Penguin measurements as z-scores")
ax.set_xlabel("Measurement")
ax.set_ylabel("Standard deviations from mean")
Min-max: a fixed 0 to 1 range. Z-scores: centred on 0, spread of 1.
The scikit-learn scalers#
Problem: remember the scaling rule and apply it to new data later.
from sklearn.preprocessing import StandardScaler
std = StandardScaler().set_output(transform="pandas")
z_sk = std.fit_transform(feats)
z_sk.head().round(2)
print(std.mean_.round(1))
print(std.scale_.round(1))
Quick check: the pandas and scikit-learn z-scores differ slightly. Why?
print(round(feats["body_mass_g"].std(), 2))
print(round(feats["body_mass_g"].std(ddof=0), 2))
Common mistake: passing a single column with single brackets
std.fit_transform(pens["body_mass_g"])
one = std.fit_transform(pens[["body_mass_g"]])
one.describe().round(2)
from sklearn.preprocessing import MinMaxScaler
mms = MinMaxScaler().set_output(transform="pandas")
mm_sk = mms.fit_transform(feats)
print(round((mm_sk - mm).abs().max().max(), 10))
wide = MinMaxScaler(feature_range=(-1, 1))
wide.set_output(transform="pandas")
wide.fit_transform(feats).agg(["min", "max"])
back = mms.inverse_transform(mm_sk.head(3))
back = pd.DataFrame(back, columns=num)
back.round(1)
Outliers and skew: robust scaling and logs#
Problem: a few huge values squash everyone else together.
house = pd.read_csv("data/pandas/housing.csv")
pop = house["population"]
pop.describe().round(0)
pop_mm = (pop - pop.min()) / (pop.max() - pop.min())
print(round(pop_mm.median(), 3))
print(round((pop_mm < 0.1).mean(), 3))
from sklearn.preprocessing import RobustScaler
rob = RobustScaler().set_output(transform="pandas")
pop_rob = rob.fit_transform(house[["population"]])
pop_rob.describe().round(2)
import numpy as np
cols = ["population", "total_rooms", "median_income"]
logged = np.log1p(house[cols])
pd.DataFrame({"before": house[cols].skew(),
"after": logged.skew()}).round(2)
import matplotlib.pyplot as plt
fig, axes = plt.subplots(1, 2, figsize=(9, 3.5))
pop.plot.hist(bins=50, ax=axes[0])
axes[0].set_title("Raw population")
logged["population"].plot.hist(bins=50, ax=axes[1])
axes[1].set_title("log1p(population)")
for ax in axes:
ax.set_xlabel("Value")
ax.set_ylabel("Number of blocks")
plt.show()
Your turn#
Log total_bedrooms, then standardise it. Report the new skew and mean.
beds = np.log1p(house["total_bedrooms"])
beds_z = (beds - beds.mean()) / beds.std()
print(round(beds_z.skew(), 2), round(beds_z.mean(), 2))
print(beds_z.isna().sum())
Fit on training data only#
Problem: the scaler must not peek at the data we test on.
from sklearn.model_selection import train_test_split
y = pens["species"]
X_tr, X_te, y_tr, y_te = train_test_split(
feats, y, test_size=0.25, random_state=42, stratify=y)
print(X_tr.shape, X_te.shape)
Common mistake: transforming before the scaler has been fitted
fresh = StandardScaler().set_output(transform="pandas")
fresh.transform(X_te)
fresh.fit(X_tr)
tr_z = fresh.transform(X_tr)
te_z = fresh.transform(X_te)
pd.DataFrame({"train": tr_z.mean(), "test": te_z.mean()}).round(2)
Quick check: the test means aren't 0. Is something wrong?
No: test rows reuse the training mean and spread, like real new data.
Your turn#
Fit a MinMaxScaler on X_tr, transform X_te. Do test values stay in 0 to 1?
mm_fit = MinMaxScaler().set_output(transform="pandas")
mm_fit.fit(X_tr)
mm_fit.transform(X_te).agg(["min", "max"]).round(3)
Mini-project: does scaling help a model tell species apart?#
Plan: k nearest neighbours on raw data, then on scaled data, then compare.
from sklearn.neighbors import KNeighborsClassifier
raw_knn = KNeighborsClassifier(n_neighbors=5)
raw_knn.fit(X_tr, y_tr)
raw_acc = raw_knn.score(X_te, y_te)
print(round(raw_acc, 3))
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(tr_z, y_tr)
sc_acc = knn.score(te_z, y_te)
print(round(sc_acc, 3))
wrong = X_te[raw_knn.predict(X_te) != y_te]
print(y_te[wrong.index].value_counts())
accs = pd.Series({"Raw": raw_acc, "Standardised": sc_acc})
ax = accs.plot(kind="bar", rot=0, figsize=(6, 3.5))
ax.set_title("KNN accuracy on penguin species")
ax.set_xlabel("Features")
ax.set_ylabel("Test accuracy")
plt.show()
gain = (sc_acc - raw_acc) * len(y_te)
print(f"Scaling fixed {gain:.0f} of {len(y_te)} test penguins")
Recap#
- Distance-based models let big-number columns dominate
- Min-max maps to 0 to 1; z-scores centre on 0 with spread 1
- Scale numeric columns only, and pass 2D data to scalers
- Robust scaling and log1p tame outliers and skew
- Fit on the training data, then transform both piles
Challenges#
- Repeat the mini-project with MinMaxScaler. Same result?
- Scale housing's median_income with Standard and Robust scalers; compare max.
- Try n_neighbors from 1 to 15 on raw data. Does any k match scaling?
What's next#
Lesson 52: Feature Engineering with Pandas Create new, more informative columns from the ones you already have.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.



