Mathew K AnalyticsLesson 49 · Market Research Analytics in Python
Retention and Churn Analysis Using Python for Market Research
Learn how to identify which customers will leave (churn) and what keeps them loyal. This analysis helps businesses improve customer loyalty, lower…
- CourseMarket Research Analytics in Python
- Lesson49 of 56
- Video24 min
- FormatJupyter notebook · 24 code cells
What you'll learn
Data
No separate download needed — the notebook creates or downloads everything it uses.
📓 Full notebook
Download .ipynbRetention and Churn Analysis in Market Research#
- Learn how to identify which customers will leave (churn) and what keeps them loyal.
- This analysis helps businesses improve customer loyalty, lower acquisition costs, and boost revenue.
- We will uncover how to measure, visualize, and predict customer retention and churn, creating actionable business insights.
- By the end, you will know how to find patterns, spot risks, and make recommendations to reduce churn.
import pandas as pd
import numpy as np
import openml
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
warnings.filterwarnings('ignore')
Core Market Research Concepts for Retention and Churn#
- In retention and churn studies, data often includes customer demographics, behaviors, purchase history, and survey responses.
- Each customer can have fields like gender, age, tenure, contract type, and whether they churned (left) or stayed.
- Retention means a customer keeps using a company's product; churn means a customer left.
- Beginners often forget to check for missing values, or mix up active customers with churned ones.
- Another common mistake is not understanding what time windows the retention or churn rates cover (monthly, yearly, etc).
dataset = openml.datasets.get_dataset(42178)
df, _, _, _ = dataset.get_data(dataset_format='dataframe')
print(df.shape)
print(df.head(3))
print(df.columns)
print(df.dtypes)
print(df['Churn'].value_counts())
churn_rate = df['Churn'].value_counts(normalize=True).get('Yes', 0) * 100
print(f'Overall churn rate: {churn_rate:.2f}%')
contract_churn = pd.crosstab(df['Contract'], df['Churn'], normalize='index')
print(contract_churn)
contract_churn['Yes'].plot(kind='bar', color='salmon')
plt.ylabel('Churn Rate')
plt.title('Churn Rate by Contract Type')
plt.show()
gender_senior_churn = df.groupby(['gender','SeniorCitizen'])['Churn'].value_counts(normalize=True).unstack().fillna(0)
print(gender_senior_churn)
df['tenure_month'] = df['tenure']
monthly_churn = df.groupby('tenure_month')['Churn'].value_counts(normalize=True).unstack().fillna(0)['Yes']
monthly_churn.plot()
plt.xlabel('Customer Tenure (Months)')
plt.ylabel('Monthly Churn Rate')
plt.title('Monthly Churn Trend')
plt.show()
url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/00502/online_retail_II.xlsx'
retail_df = pd.read_excel(url, sheet_name='Year 2010-2011')
retail_df['InvoiceDate'] = pd.to_datetime(retail_df['InvoiceDate'])
print(retail_df.shape)
print(retail_df.head(3))
retail_df = retail_df.dropna(subset=['Customer ID'])
retail_df['CohortMonth'] = retail_df.groupby('Customer ID')['InvoiceDate'].transform('min').dt.to_period('M')
retail_df['InvoiceMonth'] = retail_df['InvoiceDate'].dt.to_period('M')
cohort_data = retail_df.groupby(['CohortMonth', 'InvoiceMonth'])['Customer ID'].nunique().reset_index()
cohort_pivot = cohort_data.pivot(index='CohortMonth', columns='InvoiceMonth', values='Customer ID')
retention = cohort_pivot.divide(cohort_pivot.iloc[:,0], axis=0)
print(retention.round(3).head())
plt.figure(figsize=(12,6))
sns.heatmap(retention, annot=True, fmt='.0%', cmap='Blues')
plt.title('Customer Retention by Cohort Month')
plt.ylabel('Cohort Month')
plt.xlabel('Active Month')
plt.show()
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
churn_features = ['SeniorCitizen', 'tenure', 'MonthlyCharges']
X = df[churn_features]
y = df['Churn'].map({'No':0, 'Yes':1})
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
print(classification_report(y_test, y_pred))
X['ChurnProb'] = model.predict_proba(X)[:,1]
threshold = 0.7
high_risk = df.loc[X['ChurnProb'] > threshold]
print(high_risk[['gender', 'SeniorCitizen', 'tenure', 'MonthlyCharges', 'Churn']].head())
missing_counts = df.isnull().sum()
print(missing_counts[missing_counts > 0])
try:
print(df.groupby('Contract')['Churn'].value_counts())
except Exception as e:
print('Error:', e)
if 'NPS_Score' not in df.columns:
print('NPS_Score column not present in this dataset. Skipping.')
else:
nps_counts = df['NPS_Score'].value_counts().sort_index()
print(nps_counts)
segmentation = df.groupby(['gender','SeniorCitizen','Contract'])['Churn'].value_counts(normalize=True).unstack().fillna(0)['Yes']
print(segmentation.sort_values(ascending=False).head(10))
payment_crosstab = pd.crosstab(df['PaymentMethod'], df['Churn'], normalize='index')
print(payment_crosstab)
retention_index = (1 - churn_rate/100) * 100
print(f'Retention index (percentage who stay): {retention_index:.2f}%')
monthly_total = df.groupby('tenure_month').size()
monthly_retained = df[df['Churn'] == 'No'].groupby('tenure_month').size()
retention_trend = (monthly_retained / monthly_total * 100).fillna(0)
retention_trend.plot()
plt.ylabel('Retention Rate (%)')
plt.xlabel('Tenure Month')
plt.title('Customer Retention Over Tenure')
plt.show()
End-to-End Market Research Example: Actionable Churn Recommendation#
- Now combine your knowledge by running a full churn analysis and making a concrete recommendation.
- We will go from raw data to a final business insight.
- Your goal: Identify and briefly justify what customer segment needs the most urgent retention effort.
# 1. Load data and show basic churn statistics
print(df['Churn'].value_counts(normalize=True) * 100)
# 2. Churn by contract type and payment method, then find high-risk segments
risk_segments = pd.crosstab([df['Contract'], df['PaymentMethod']], df['Churn'], normalize='index').reset_index()
top_risks = risk_segments.sort_values('Yes', ascending=False).head(5)
print(top_risks[['Contract', 'PaymentMethod', 'Yes']])
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.
Related lessons
23 minLesson 22
Customer Churn Prediction: Step-by-Step Python Tutorial Using Real Data
27 minLesson 23
Python Data Analytics #23: Real Customer Churn & Retention Analysis in Python
19 minLesson 4
Building a News Data Scraping and Analysis Project Using Python
18 minLesson 11