Mathew K AnalyticsLesson 19 · Python for Retail E-commerce Analytics
Encoding Categorical Retail Variables for Machine Learning in Python
In this lesson, we will learn to solve a common retail analytics problem: how to encode and work with categorical variables in retail and e-commerce…
- CoursePython for Retail E-commerce Analytics
- Lesson19 of 43
- Video26 min
- FormatJupyter notebook · 23 code cells
What you'll learn
Data
No separate download needed — the notebook creates or downloads everything it uses.
📓 Full notebook
Download .ipynbEncoding Categorical Retail Variables#
- In this lesson, we will learn to solve a common retail analytics problem: how to encode and work with categorical variables in retail and e-commerce datasets.
- Categorical variables like product category, country, or payment method are not directly usable in most analytics or machine learning models.
- Transforming these variables allows analysts and data scientists to extract valuable business insights and enable advanced algorithms.
- You will practice encoding categories in real retail data and interpreting the impact on key business metrics.
import pandas as pd
import numpy as np
import warnings
warnings.filterwarnings('ignore')
Key Concepts in Retail Categorical Variable Analytics#
- Retail datasets include information such as transactions, customer orders, product catalogs, and more.
- Typical business metrics include revenue, sales quantity, product prices, and category or country breakdowns.
- A common mistake is to treat categorical variables as numerical or forget to encode them for analysis.
- Correct handling of categorical variables leads to better grouping, segmentation, and actionable insights.
# Example 1: Load real retail transactions with categorical columns
url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/00502/online_retail_II.xlsx'
df = pd.read_excel(url, sheet_name='Year 2010-2011')
df['InvoiceDate'] = pd.to_datetime(df['InvoiceDate'])
print(df.shape)
print(df[['Invoice', 'Country', 'Description']].head(3))
# Example 2: Count unique categories in 'Country' and 'Description'
num_countries = df['Country'].nunique()
num_products = df['Description'].nunique()
print('Unique countries:', num_countries)
print('Unique product descriptions:', num_products)
# Example 3: Show most common countries in transactions
print(df['Country'].value_counts().head())
# Example 4: Simple label encoding of country column
from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
df['Country_code'] = le.fit_transform(df['Country'])
print(df[['Country', 'Country_code']].head(7))
# Example 5: One-hot encoding for countries
df_onehot = pd.get_dummies(df, columns=['Country'], prefix='cty')
print(df_onehot.filter(like='cty_').head(3))
# Example 6: Mean-encoding country by average unit price
country_mean_price = df.groupby('Country')['Price'].mean()
df['Country_Price_Mean'] = df['Country'].map(country_mean_price)
print(df[['Country', 'Country_Price_Mean']].head(5))
# Example 7: Frequency encoding for product Description
description_freq = df['Description'].value_counts(normalize=True)
df['Description_freq'] = df['Description'].map(description_freq)
print(df[['Description', 'Description_freq']].head(5))
# Example 8: Prepare a product catalog with categories
np.random.seed(42)
categories = ['Electronics','Clothing','Home','Sports','Beauty']
product_ids = list(range(1001,1101))
product_categories = np.random.choice(categories,100)
product_prices = np.round(np.random.uniform(5,500,100),2)
catalog = pd.DataFrame({'ProductID':product_ids,'Category':product_categories,'Price':product_prices})
print(catalog.head(3))
# Example 9: One-hot encode product category in the catalog
catalog_ohe = pd.get_dummies(catalog, columns=['Category'])
print(catalog_ohe.head())
# Example 10: Label encoding category for tree-based models
from sklearn.preprocessing import OrdinalEncoder
oe = OrdinalEncoder()
catalog['Category_Ordinal'] = oe.fit_transform(catalog[['Category']])
print(catalog[['Category', 'Category_Ordinal']].head(10))
# Example 11: Create a simulated retail order dataset
np.random.seed(42)
n_orders = 1000
order_ids = list(range(1, n_orders + 1))
customer_ids = np.random.randint(1000, 1500, n_orders)
product_ids = np.random.randint(1001, 1100, n_orders)
quantities = np.random.randint(1, 5, n_orders)
order_dates = pd.date_range('2023-01-01', periods=n_orders, freq='h')
orders = pd.DataFrame({'OrderID':order_ids, 'CustomerID':customer_ids, 'ProductID':product_ids, 'Quantity':quantities, 'OrderDate':order_dates})
print(orders.head(3))
# Example 12: Merge product categories into orders
orders = pd.merge(orders, catalog[['ProductID', 'Category']], on='ProductID', how='left')
print(orders[['OrderID', 'ProductID', 'Category']].head(5))
# Example 13: Label encode Category in merged orders table
orders['Category_code'] = le.fit_transform(orders['Category'])
print(orders[['Category', 'Category_code']].head(10))
# Example 14: Group orders by encoded category and count
cat_groups = orders.groupby('Category_code').size()
print(cat_groups)
# Example 15: Advanced - Target encode product categories with quantity mean
cat_quant_mean = orders.groupby('Category')['Quantity'].mean()
orders['Category_Quant_Mean'] = orders['Category'].map(cat_quant_mean)
print(orders[['Category', 'Category_Quant_Mean']].drop_duplicates().head())
# Example 16: Advanced - Combine multiple encodings for same feature
orders['Category_freq'] = orders['Category'].map(orders['Category'].value_counts(normalize=True))
orders = pd.concat([orders, pd.get_dummies(orders['Category'], prefix='cat')], axis=1)
print(orders.head(3))
# Example 17: Advanced - Encoding errors: missing values in category
orders_with_nan = orders.copy()
orders_with_nan.loc[::50, 'Category'] = np.nan
try:
orders_with_nan['Category_code'] = le.fit_transform(orders_with_nan['Category'].astype(str))
print('Category codes with NaN:', orders_with_nan['Category_code'].unique())
except Exception as e:
print('Encoding failed:', str(e))
# Example 18: Debugging - Ensure all categories encoded after missing fill
orders_with_nan['Category_fill'] = orders_with_nan['Category'].fillna('Unknown')
orders_with_nan['Category_code'] = le.fit_transform(orders_with_nan['Category_fill'])
print(orders_with_nan[['Category_fill', 'Category_code']].head(10))
# Example 19: Debugging - Catch grouping mistakes with encoded categories
# Incorrect: using codes instead of names for aggregation reporting
wrong_report = orders.groupby('Category_code').agg({'Quantity':'sum'})
print('WRONG: Aggregation by codes only')
print(wrong_report.head())
# Correct: map codes back to names for proper business reporting
labels = dict(zip(orders['Category_code'], orders['Category']))
wrong_report.index = wrong_report.index.to_series().map(labels)
print('CORRECTED: Aggregation with category names')
print(wrong_report.head())
Best Practices for Encoding Categorical Retail Variables#
- Use label or ordinal encoding for tree-based models (like decision trees or random forest).
- Use one-hot encoding for linear models or when there are few categories.
- Always handle missing category values before encoding.
- Avoid encoding category columns with thousands of unique values without dimensionality reduction.
- Use mean or frequency encoding to capture business-signal, but monitor for overfitting.
# Advanced Pattern: Customer segmentation using encoded categories
customer_category = orders.groupby(['CustomerID','Category'])['Quantity'].sum().unstack(fill_value=0)
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
cust_cat_scaled = scaler.fit_transform(customer_category)
print(pd.DataFrame(cust_cat_scaled, columns=customer_category.columns).head())
Tiny End-to-End Retail Analytics Task#
- Objective: Identify top-selling product categories for a key business region.
- Steps:
- Filter transactions for United Kingdom.
- Merge with product catalog to link to category.
- Aggregate sales by category and sort.
- Recommend which product category to prioritize.
# Filter for United Kingdom transactions and merge with product catalog
uk_df = df[df['Country'] == 'United Kingdom']
uk_df = pd.merge(uk_df, catalog[['ProductID', 'Category']], left_on='StockCode', right_on='ProductID', how='left')
print(uk_df[['Invoice', 'StockCode', 'Category']].head(7))
# Aggregate and rank top-selling categories in UK
uk_cat_sales = uk_df.groupby('Category')['Quantity'].sum().sort_values(ascending=False)
print('Top selling product categories in UK:')
print(uk_cat_sales)
Practice and Next Steps#
- Practice encoding other retail categorical variables, such as payment type, customer segment, or region.
- Try using encoded categories in a sales forecasting model or a customer cluster analysis.
- For visual demos: Search "retail analytics encoding" on YouTube for more step-by-step guides.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.
Related lessons
18 minLesson 19
Encoding Categorical and Survey Variables in Python | Market Research Analytics Tutorial
14 minLesson 1
1 Capstone ML Project in Python: Build and Evaluate a Machine Learning Model
10 minLesson 5
3 Cross-Validation in Python: Machine Learning Model Evaluation Techniques
15 minLesson 30