Lesson 50 · Pandas Full Course
Encode Categorical Variables in Pandas: One-Hot, Ordinal and Label | Pandas Course #50
In this lesson you'll be able to: Spot categorical columns and explain why models need numbers Encode yes or no columns and labels with map and cat.codes…
- CoursePandas Full Course
- Lesson50 of 65
- FormatJupyter notebook · 43 code cells
- Data2 datasets
What you'll learn
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
- titanic.csv preview58.9 KB
- diamonds.csv preview2.6 MB
📓 Full notebook
Download .ipynbPandas 50: Encoding Categorical Variables#
In this lesson you'll be able to:
- Spot categorical columns and explain why models need numbers
- Encode yes or no columns and labels with map and cat.codes
- Keep the real order of ranked categories with ordinal encoding
- One-hot encode with get_dummies and keep new data aligned
- Tame high-cardinality columns with frequency and rare grouping Data: the Titanic passenger list and 53,940 diamonds (data/pandas/titanic.csv, diamonds.csv)
Warm-up: tables full of labels#
Lesson 49 flattened JSON into tables. Those tables are full of text labels.
import pandas as pd
df = pd.read_csv("data/pandas/titanic.csv")
records = df.head(2).to_dict("records")
back = pd.json_normalize(records)
back[["Name", "Sex", "Pclass", "Embarked"]]
Spot the categorical columns#
Problem: find the columns that hold labels, and see why models reject them.
labels = df.select_dtypes(exclude="number")
labels.nunique()
Quick check: Pclass is stored as a number. Is it categorical?
df["Pclass"].value_counts().sort_index()
Your turn#
List every column, numeric or not, with fewer than 10 distinct values.
few = df.nunique()
few[few < 10].sort_values()
Common mistake: giving a model raw text
from sklearn.linear_model import LogisticRegression
model = LogisticRegression()
model.fit(df[["Sex", "Pclass"]], df["Survived"])
X = df[["Pclass"]].assign(is_male=(df["Sex"] == "male").astype(int))
model.fit(X, df["Survived"])
print(round(model.score(X, df["Survived"]), 3))
Binary and label encoding#
Problem: turn two-value and few-value columns into simple numbers.
df["is_male"] = (df["Sex"] == "male").astype(int)
df[["Sex", "is_male"]].value_counts()
ports = {"S": 0, "C": 1, "Q": 2}
df["port_code"] = df["Embarked"].map(ports)
df["port_code"].value_counts(dropna=False)
Quick check: what does map do with a label that isn't in the dictionary?
short = {"S": 0, "C": 1}
print(df["Embarked"].map(short).isna().sum())
port_cat = df["Embarked"].astype("category")
print(port_cat.cat.categories.tolist())
port_cat.cat.codes.value_counts().sort_index()
back_map = {num: port for port, num in ports.items()}
restored = df["port_code"].map(back_map)
print(restored.equals(df["Embarked"]))
print(back_map)
Your turn#
Encode Sex with cat.codes. Which number did male get, and why?
sex = df["Sex"].astype("category")
print(dict(enumerate(sex.cat.categories)))
Ordinal encoding: keep the real order#
Problem: some labels have a ranking, and the numbers must respect it.
gems = pd.read_csv("data/pandas/diamonds.csv")
print(gems.shape)
gems["cut"].value_counts()
alpha = gems["cut"].astype("category")
dict(enumerate(alpha.cat.categories))
Common mistake: ranking an unordered category
(alpha > "Good").sum()
order = ["Fair", "Good", "Very Good", "Premium", "Ideal"]
cut_type = pd.CategoricalDtype(order, ordered=True)
gems["cut"] = gems["cut"].astype(cut_type)
print((gems["cut"] > "Good").sum())
print(dict(enumerate(gems["cut"].cat.categories)))
gems["cut_code"] = gems["cut"].cat.codes
gems[["cut", "cut_code"]].drop_duplicates().sort_values("cut_code")
Quick check: better cuts should cost more. Do they?
by_cut = gems.groupby("cut", observed=True)
by_cut[["price", "carat"]].median()
one_carat = gems[gems["carat"].between(0.9, 1.1)]
fair_test = one_carat.groupby("cut_code")["price"].median()
fair_test
Your turn#
Ordinal-encode clarity, worst to best: I1, SI2, SI1, VS2, VS1, VVS2, VVS1, IF.
grades = ["I1", "SI2", "SI1", "VS2", "VS1", "VVS2", "VVS1", "IF"]
clar = pd.CategoricalDtype(grades, ordered=True)
gems["clarity_code"] = gems["clarity"].astype(clar).cat.codes
gems.groupby("clarity_code")["carat"].median()
One-hot encoding with get_dummies#
Problem: unordered labels need numbers that don't imply a ranking.
dummies = pd.get_dummies(df["Embarked"], prefix="port", dtype=int)
pd.concat([df["Embarked"], dummies], axis=1).head()
full = pd.get_dummies(df["Embarked"], prefix="port", dtype=int,
dummy_na=True)
full.sum()
slim = pd.get_dummies(df["Embarked"], prefix="port", dtype=int,
drop_first=True)
slim.head(3)
Common mistake: a column name with the wrong capital letter
pd.get_dummies(df, columns=["embarked"])
encoded = pd.get_dummies(df, columns=["Sex", "Embarked"], dtype=int)
print(encoded.shape)
print([c for c in encoded.columns if "_" in c][-5:])
everything = pd.get_dummies(df)
print(everything.shape)
Your turn#
One-hot encode Pclass. Careful: it's stored as a number.
cls = pd.get_dummies(df[["Pclass"]], columns=["Pclass"], dtype=int)
cls.sum()
Keep new data aligned#
Problem: new data may lack some labels, or bring ones we've never seen.
train = pd.get_dummies(df["Embarked"], prefix="port", dtype=int)
batch = df[df["Embarked"] == "Q"].head(3)
new = pd.get_dummies(batch["Embarked"], prefix="port", dtype=int)
print(train.columns.tolist())
print(new.columns.tolist())
aligned = new.reindex(columns=train.columns, fill_value=0)
aligned
from sklearn.preprocessing import OneHotEncoder
ports_seen = df[["Embarked"]].dropna()
enc = OneHotEncoder(sparse_output=False).fit(ports_seen)
print(enc.categories_)
Common mistake: a category the encoder has never seen
odd = pd.DataFrame({"Embarked": ["Unknown"]})
enc.transform(odd)
safe = OneHotEncoder(sparse_output=False, handle_unknown="ignore")
safe.fit(ports_seen)
print(safe.transform(odd))
names = safe.get_feature_names_out()
grid = safe.transform(df[["Embarked"]].fillna("Unknown").head(3))
pd.DataFrame(grid, columns=names)
Tame high-cardinality columns#
Problem: columns with hundreds of labels would explode into hundreds of columns.
counts = df["Ticket"].value_counts()
df["ticket_count"] = df["Ticket"].map(counts)
df["ticket_count"].value_counts().sort_index()
df["deck"] = df["Cabin"].str[0].fillna("Unknown")
df["deck"].value_counts()
deck_counts = df["deck"].value_counts()
rare = deck_counts[deck_counts < 20].index
df["deck"] = df["deck"].where(~df["deck"].isin(rare), "Other")
df["deck"].value_counts()
deck_rate = df.groupby("deck")["Survived"].mean()
deck_rate.sort_values(ascending=False).round(2)
Your turn#
Does ticket_count relate to survival? Show the survival rate for each count.
share = df.groupby("ticket_count")["Survived"].agg(["size", "mean"])
share.round(2)
Mini-project: which encoded feature tracks survival?#
Plan: build an all-numeric table, correlate with Survived, chart, test a model.
base = df[["Survived", "Pclass", "is_male", "ticket_count"]]
ports_1h = pd.get_dummies(df["Embarked"], prefix="port", dtype=int)
decks_1h = pd.get_dummies(df["deck"], prefix="deck", dtype=int)
features = pd.concat([base, ports_1h, decks_1h], axis=1)
print(features.shape)
print(features.dtypes.value_counts())
corr = features.corr()["Survived"].drop("Survived")
corr.sort_values().round(2)
import matplotlib.pyplot as plt
ax = corr.sort_values().plot(kind="barh", figsize=(7, 5))
ax.set_title("Correlation of encoded features with survival")
ax.set_xlabel("Correlation with Survived")
ax.set_ylabel("Feature")
plt.show()
from sklearn.model_selection import train_test_split
X = features.drop(columns="Survived")
y = features["Survived"]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
random_state=42)
model = LogisticRegression(max_iter=1000).fit(X_tr, y_tr)
print(round(model.score(X_te, y_te), 3))
top = corr.abs().idxmax()
print(f"Strongest feature: {top} ({corr[top]:.2f})")
Recap#
- Models need numbers: find label columns with select_dtypes
- Binary and label codes: comparisons, map and cat.codes
- Ranked labels: an ordered CategoricalDtype keeps the true order
- Unordered labels: get_dummies, then reindex new data to match
- Many labels: frequency encoding and pooling rare values
Challenges#
- Ordinal-encode diamond color (J worst to D best) and compare median prices.
- Frequency-encode Cabin and compare survival for shared cabins.
- Add one-hot Pclass to the features and see if the model improves.
What's next#
Lesson 51: Scaling and Normalizing Data Put numeric features on comparable scales before modelling.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.



