Lesson 11 · Market Research Analytics in Python

Mastering Survey & Market Research Dataset Structures in Python | Analytics Training

Understand common structures in real-world survey and market research data Learn why dataset structure matters for business decision-making Gain insight…

⬇ Download notebook Open in Colab ↗

CourseMarket Research Analytics in Python
Lesson11 of 56
Video19 min
FormatJupyter notebook · 15 code cells

pandas NumPy Matplotlib

What you'll learn

What are Survey and Market Research Datasets?
Beginner Example 1: Loading a Customer Satisfaction Survey Dataset
Beginner Example 2: Examining Common Survey Column Types
Beginner Example 3: Loading a Marketing Campaign Dataset
Intermediate Example 1: Loading an Online Retail Behavior Dataset
Intermediate Example 2: Handling Net Promoter Score (NPS) Survey Data
Intermediate Example 3: Open-Ended Customer Feedback Data
Advanced Example 1: Understanding Customer Cohort Datasets

Data

No separate download needed — the notebook creates or downloads everything it uses.

📓 Full notebook

Download .ipynb

Structure of Survey and Market Research Datasets#

Understand common structures in real-world survey and market research data
Learn why dataset structure matters for business decision-making
Gain insight into best dataset formats for analytics
Identify the most common mistakes in organizing and analyzing survey responses
Discover how to explore, clean, and summarize typical market research datasets

import pandas as pd
import numpy as np
import openml
import warnings
warnings.filterwarnings('ignore')

What are Survey and Market Research Datasets?#

These datasets are structured tables that collect responses from people, such as customers or survey participants.
Columns may include demographics, responses to questions, ratings, open-ended feedback, or purchasing behavior.
Market research datasets often combine business outcomes (like Churn or Purchase) with survey metrics or experimental results.
Common mistakes include treating categorical answers as numeric, ignoring missing data, or misinterpreting scaled responses.
Well-structured data helps you find patterns, segment users, and make business recommendations.

Beginner Example 1: Loading a Customer Satisfaction Survey Dataset#

We start by loading a real customer satisfaction survey.
Let us preview its structure and discuss its columns.

# Load customer satisfaction survey from OpenML
dataset = openml.datasets.get_dataset(42178)
df_satis, _, _, _ = dataset.get_data(dataset_format='dataframe')
print(df_satis.shape)
print(df_satis.head(3))

(7043, 20)
   gender  SeniorCitizen Partner Dependents  tenure PhoneService  \
0  Female              0     Yes         No       1           No   
1    Male              0      No         No      34          Yes   
2    Male              0      No         No       2          Yes   

      MultipleLines InternetService OnlineSecurity OnlineBackup  \
0  No phone service             DSL             No          Yes   
1                No             DSL            Yes           No   
2                No             DSL            Yes          Yes   

  DeviceProtection TechSupport StreamingTV StreamingMovies        Contract  \
0               No          No          No              No  Month-to-month   
1              Yes          No          No              No        One year   
2               No          No          No              No  Month-to-month   

  PaperlessBilling     PaymentMethod  MonthlyCharges TotalCharges Churn  
0              Yes  Electronic check           29.85        29.85    No  
1               No      Mailed check           56.95       1889.5    No  
2              Yes      Mailed check           53.85       108.15   Yes

Beginner Example 2: Examining Common Survey Column Types#

Survey datasets often include categorical, numeric, and textual columns.
Let us look at typical columns from our customer satisfaction data.
Columns include: Gender, SeniorCitizen, Partner, Dependents, tenure, PhoneService, etc.

print('Columns:')
print(', '.join(df_satis.columns))
print('\nExample data:')
print(df_satis.iloc[0])

Columns:
gender, SeniorCitizen, Partner, Dependents, tenure, PhoneService, MultipleLines, InternetService, OnlineSecurity, OnlineBackup, DeviceProtection, TechSupport, StreamingTV, StreamingMovies, Contract, PaperlessBilling, PaymentMethod, MonthlyCharges, TotalCharges, Churn

Example data:
gender                        Female
SeniorCitizen                      0
Partner                          Yes
Dependents                        No
tenure                             1
PhoneService                      No
MultipleLines       No phone service
InternetService                  DSL
OnlineSecurity                    No
OnlineBackup                     Yes
DeviceProtection                  No
TechSupport                       No
StreamingTV                       No
StreamingMovies                   No
Contract              Month-to-month
PaperlessBilling                 Yes
PaymentMethod       Electronic check
MonthlyCharges                 29.85
TotalCharges                   29.85
Churn                             No
Name: 0, dtype: object

Beginner Example 3: Loading a Marketing Campaign Dataset#

Let us load real customer response data from a bank marketing campaign.
Notice how it is structured differently than a satisfaction survey.

dataset = openml.datasets.get_dataset(1461)
df_marketing, _, _, _ = dataset.get_data(dataset_format='dataframe')
df_marketing.columns = ['age','job','marital','education','default','balance','housing','loan','contact','day','month','duration','campaign','pdays','previous','poutcome','response']
print(df_marketing.shape)
print(df_marketing.head(3))

(45211, 17)
   age           job  marital  education default  balance housing loan  \
0   58    management  married   tertiary      no   2143.0     yes   no   
1   44    technician   single  secondary      no     29.0     yes   no   
2   33  entrepreneur  married  secondary      no      2.0     yes  yes   

   contact  day month  duration  campaign  pdays  previous poutcome response  
0  unknown    5   may     261.0         1   -1.0       0.0  unknown        1  
1  unknown    5   may     151.0         1   -1.0       0.0  unknown        1  
2  unknown    5   may      76.0         1   -1.0       0.0  unknown        1

Intermediate Example 1: Loading an Online Retail Behavior Dataset#

Survey data may be combined with purchasing or behavioral records.
Let us load and preview an online retail transactions dataset.

url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/00502/online_retail_II.xlsx'
df_retail = pd.read_excel(url, sheet_name='Year 2010-2011')
df_retail['InvoiceDate'] = pd.to_datetime(df_retail['InvoiceDate'])
print(df_retail.shape)
print(df_retail.head(3))

(541910, 8)
  Invoice StockCode                         Description  Quantity  \
0  536365    85123A  WHITE HANGING HEART T-LIGHT HOLDER         6   
1  536365     71053                 WHITE METAL LANTERN         6   
2  536365    84406B      CREAM CUPID HEARTS COAT HANGER         8   

          InvoiceDate  Price  Customer ID         Country  
0 2010-12-01 08:26:00   2.55      17850.0  United Kingdom  
1 2010-12-01 08:26:00   3.39      17850.0  United Kingdom  
2 2010-12-01 08:26:00   2.75      17850.0  United Kingdom

Intermediate Example 2: Handling Net Promoter Score (NPS) Survey Data#

NPS surveys rate customer loyalty from 0 (not likely) to 10 (very likely).
We create a sample NPS survey dataset and examine its structure.

np.random.seed(42)
df_nps = pd.DataFrame({'CustomerID': range(1,501), 'Age': np.random.randint(18,70,500), 'Region': np.random.choice(['North','South','East','West'],500), 'NPS_Score': np.random.randint(0,11,500)})
print(df_nps.shape)
print(df_nps.head(3))

(500, 4)
   CustomerID  Age Region  NPS_Score
0           1   56   West          2
1           2   69  North          0
2           3   46   East          4

Intermediate Example 3: Open-Ended Customer Feedback Data#

Some surveys include free-text responses for richer insights.
Let us inspect an example open-ended feedback dataset.

df_feedback = pd.DataFrame({'CustomerID':[1,2,3,4,5], 'Feedback':['Great service and friendly staff','Delivery was slow and packaging was poor','Excellent quality, will buy again','Customer support needs improvement','Good value for money']})
print(df_feedback.shape)
print(df_feedback.head(3))

(5, 2)
   CustomerID                                  Feedback
0           1          Great service and friendly staff
1           2  Delivery was slow and packaging was poor
2           3         Excellent quality, will buy again

Advanced Example 1: Understanding Customer Cohort Datasets#

Cohort datasets group customers by signup date or behavior period, tracking retention.
Let us generate a synthetic cohort table and discuss how it structures time-based analysis.

np.random.seed(0)
dates = pd.date_range('2021-01-01', periods=24, freq='ME')
df_cohort = pd.DataFrame({'CustomerID': np.random.randint(1000,2000,len(dates)), 'Signup_Month': dates, 'Active_Users': np.random.randint(50,300,len(dates))})
print(df_cohort.shape)
print(df_cohort.head(3))

(24, 3)
   CustomerID Signup_Month  Active_Users
0        1684   2021-01-31           138
1        1559   2021-02-28           131
2        1629   2021-03-31           215

Advanced Example 3: Multi-table Structure and Data Dictionary Creation#

Complex research projects organize datasets across multiple related tables.
Creating a data dictionary helps everyone understand variables and formats.

data_dict = pd.DataFrame({'Column': df_satis.columns, 'Type': df_satis.dtypes.astype(str), 'Example_Value': df_satis.iloc[0].astype(str).values})
print(data_dict)

                            Column     Type     Example_Value
gender                      gender   object            Female
SeniorCitizen        SeniorCitizen    uint8                 0
Partner                    Partner   object               Yes
Dependents              Dependents   object                No
tenure                      tenure    uint8                 1
PhoneService          PhoneService   object                No
MultipleLines        MultipleLines   object  No phone service
InternetService    InternetService   object               DSL
OnlineSecurity      OnlineSecurity   object                No
OnlineBackup          OnlineBackup   object               Yes
DeviceProtection  DeviceProtection   object                No
TechSupport            TechSupport   object                No
StreamingTV            StreamingTV   object                No
StreamingMovies    StreamingMovies   object                No
Contract                  Contract   object    Month-to-month
PaperlessBilling  PaperlessBilling   object               Yes
PaymentMethod        PaymentMethod   object  Electronic check
MonthlyCharges      MonthlyCharges  float64             29.85
TotalCharges          TotalCharges   object             29.85
Churn                        Churn   object                No

Error Handling Example 1: Detecting Missing Survey Responses#

Real survey datasets often contain blank or missing answers.
Let us look for missing data in the satisfaction survey.

missing_counts = df_satis.isnull().sum()
print('Columns with missing values:')
print(missing_counts[missing_counts > 0])

Columns with missing values:
Series([], dtype: int64)

Error Handling Example 2: Incorrect Grouping/Segmentation#

A common mistake is grouping by the wrong field, leading to bad insights.
Let us see what happens if we group NPS results by an unrelated column.

try:
    print(df_nps.groupby('CustomerID')['NPS_Score'].mean().head())
except Exception as e:
    print(f'Error: {e}')

CustomerID
1    2.0
2    0.0
3    4.0
4    3.0
5    9.0
Name: NPS_Score, dtype: float64

Error Handling Example 3: Misinterpreting Scaled Responses#

Likert or NPS scales are often numeric but should be analyzed as ordered categories.
Let us look at the unique survey values and summary statistics.

unique_nps = df_nps['NPS_Score'].unique()
print(sorted(unique_nps))
print(df_nps['NPS_Score'].describe())

[np.int32(0), np.int32(1), np.int32(2), np.int32(3), np.int32(4), np.int32(5), np.int32(6), np.int32(7), np.int32(8), np.int32(9), np.int32(10)]
count    500.000000
mean       4.890000
std        3.142236
min        0.000000
25%        2.000000
50%        5.000000
75%        7.000000
max       10.000000
Name: NPS_Score, dtype: float64

Best Practices: Customer Segmentation, Cross Tabulation, and Score Construction#

Segmenting customers by demographics or behavior unlocks actionable patterns.
Cross-tabulating responses with features highlights differences between groups.
Constructing indices or summary scores creates easy-to-read dashboards.
Let us segment marketing campaign responses by job type and education.

cross_tab = pd.crosstab(df_marketing['job'], df_marketing['education'])
print(cross_tab)

education      primary  secondary  tertiary  unknown
job                                                 
admin.             209       4219       572      171
blue-collar       3758       5371       149      454
entrepreneur       183        542       686       76
housemaid          627        395       173       45
management         294       1121      7801      242
retired            795        984       366      119
self-employed      130        577       833       39
services           345       3457       202      150
student             44        508       223      163
technician         158       5229      1968      242
unemployed         257        728       289       29
unknown             51         71        39      127

Best Practices: Time-based Trend Analysis with Cohorts#

Trend analysis shows if metrics improve over time after a product launch or campaign.
Let us plot active user counts by cohort signup month.

import matplotlib.pyplot as plt
plt.figure(figsize=(10,5))
plt.plot(df_cohort['Signup_Month'], df_cohort['Active_Users'], marker='o')
plt.xlabel('Signup Month')
plt.ylabel('Active Users')
plt.title('Active Users by Cohort Signup Month')
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()

No description has been provided for this image

End-to-End Example: From Survey Structure to Business Insight#

Let us walk through a simple market research workflow using survey data.
Question: Are senior citizens more likely to churn than younger customers?
Steps: Segment, summarize churn rates, and make a business recommendation.

grouped = df_satis.groupby('SeniorCitizen')['Churn'].value_counts(normalize=True).unstack().fillna(0)
print(grouped)
if 'Yes' in grouped.columns:
    senior_churn = grouped.loc[1, 'Yes']
    non_senior_churn = grouped.loc[0, 'Yes']
    print(f'Senior citizen churn rate: {senior_churn:.2%}')
    print(f'Non-senior citizen churn rate: {non_senior_churn:.2%}')
    print('Business Insight: Focus retention efforts on senior citizens if their churn rate is higher.')

Churn                No       Yes
SeniorCitizen                    
0              0.763938  0.236062
1              0.583187  0.416813
Senior citizen churn rate: 41.68%
Non-senior citizen churn rate: 23.61%
Business Insight: Focus retention efforts on senior citizens if their churn rate is higher.

Recap: Structure Enables Effective Market Research Analysis#

Well-structured survey and customer data let you segment, aggregate, and extract actionable business stories.
Always check for missing data, column types, and correct segmentation.
Start every project with clear data documentation and a simple analysis outline.

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.

☕ Buy me a coffee Other ways to help