Lesson 16 · Market Research Analytics in Python

How to Identify Missing and Invalid Survey Responses in Python | Market Research Analytics

This lesson shows you how to find missing or invalid responses in real customer or market survey data. Clean survey data is crucial for insight accuracy and…

⬇ Download notebook Open in Colab ↗

CourseMarket Research Analytics in Python
Lesson16 of 56
Video20 min
FormatJupyter notebook · 17 code cells

pandas NumPy Matplotlib

What you'll learn

Understanding Survey Data Sources and Layout
Beginner Example 1: Loading Customer Satisfaction Survey Data
Beginner Example 2: Checking for Missing Responses in the Survey
Beginner Example 3: Visualizing Missingness
Intermediate Example 1: Finding Invalid Categorical Responses
Intermediate Example 2: Detecting Out-of-Range Numerical Answers
Intermediate Example 3: Checking Missingness by Customer Segment
Advanced Example 1: Loading a Synthetic NPS Survey with Programmed Missing Values

Data

No separate download needed — the notebook creates or downloads everything it uses.

📓 Full notebook

Download .ipynb

Identifying Missing and Invalid Survey Responses#

This lesson shows you how to find missing or invalid responses in real customer or market survey data.
Clean survey data is crucial for insight accuracy and business decisions.
You will learn how to check for missing values, spot input errors, and interpret their impact on survey results.
By the end, you will be able to isolate problematic survey entries and explain their effect on marketing strategy.

import warnings
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import openml
warnings.filterwarnings('ignore')

Understanding Survey Data Sources and Layout#

Surveys capture customer demographics, ratings, and open-ended feedback.
A row is usually one respondent; columns are questions or observed traits.
Mistakes beginners make:
- Ignoring missing values (they bias averages and group summaries).
- Not validating if responses are within expected ranges.
- Misreading text data, like misclassifying blanks as valid answers.

Beginner Example 1: Loading Customer Satisfaction Survey Data#

We use a real customer satisfaction survey from OpenML.
First, inspect the data for completeness.

dataset = openml.datasets.get_dataset(42178)
df, _, _, _ = dataset.get_data(dataset_format='dataframe')
print(df.shape)
print(df.head(3))

(7043, 20)
   gender  SeniorCitizen Partner Dependents  tenure PhoneService  \
0  Female              0     Yes         No       1           No   
1    Male              0      No         No      34          Yes   
2    Male              0      No         No       2          Yes   

      MultipleLines InternetService OnlineSecurity OnlineBackup  \
0  No phone service             DSL             No          Yes   
1                No             DSL            Yes           No   
2                No             DSL            Yes          Yes   

  DeviceProtection TechSupport StreamingTV StreamingMovies        Contract  \
0               No          No          No              No  Month-to-month   
1              Yes          No          No              No        One year   
2               No          No          No              No  Month-to-month   

  PaperlessBilling     PaymentMethod  MonthlyCharges TotalCharges Churn  
0              Yes  Electronic check           29.85        29.85    No  
1               No      Mailed check           56.95       1889.5    No  
2              Yes      Mailed check           53.85       108.15   Yes

Beginner Example 2: Checking for Missing Responses in the Survey#

Sometimes survey respondents leave answers blank.
We will identify where information is missing.

missing_count = df.isnull().sum()
print(missing_count.sort_values(ascending=False))

gender              0
SeniorCitizen       0
Partner             0
Dependents          0
tenure              0
PhoneService        0
MultipleLines       0
InternetService     0
OnlineSecurity      0
OnlineBackup        0
DeviceProtection    0
TechSupport         0
StreamingTV         0
StreamingMovies     0
Contract            0
PaperlessBilling    0
PaymentMethod       0
MonthlyCharges      0
TotalCharges        0
Churn               0
dtype: int64

Beginner Example 3: Visualizing Missingness#

Seeing where missing responses cluster helps find problematic questions.
We will create a simple bar plot of missing values per column.

# Check and visualize missing values safely
missing = df.isnull().sum().sort_values(ascending=False)
missing_nonzero = missing[missing > 0]
if len(missing_nonzero) > 0:
    plt.figure(figsize=(10,4))
    missing_nonzero.plot(kind='bar', color='salmon')
    plt.title('Missing Value Count by Column')
    plt.xlabel('Column')
    plt.ylabel('Count')
    plt.show()
else:
    print('No missing values found in the dataset.')

No missing values found in the dataset.

Intermediate Example 1: Finding Invalid Categorical Responses#

Some survey questions have allowed answer options (e.g., Male/Female).
We check if any responses fall outside of these allowed categories.

allowed_genders = ['Male', 'Female']
invalid_gender = ~df['gender'].isin(allowed_genders)
print('Invalid gender responses:', df[invalid_gender]['gender'].unique())

Invalid gender responses: []

Intermediate Example 2: Detecting Out-of-Range Numerical Answers#

Survey scales often ask respondents to give rates in a certain range (e.g., 1-5).
Mistakes or data entry errors may create responses outside this range.

invalid_tenure = (df['tenure'] < 0) | (df['tenure'] > 72)
print('Number of invalid tenure entries:', invalid_tenure.sum())
print('Example invalid tenure values:', df.loc[invalid_tenure, 'tenure'].unique())

Number of invalid tenure entries: 0
Example invalid tenure values: []

Intermediate Example 3: Checking Missingness by Customer Segment#

Missing responses are often more common in certain demographic groups.
Segmenting missingness reveals hidden survey bias.

seg_missing = df.groupby('SeniorCitizen')['TotalCharges'].apply(lambda x: x.isnull().mean())
print(seg_missing)

SeniorCitizen
0    0.0
1    0.0
Name: TotalCharges, dtype: float64

Advanced Example 1: Loading a Synthetic NPS Survey with Programmed Missing Values#

We create a synthetic Net Promoter Score (NPS) survey with known missing values.
This helps us test missingness detection and repair tools.

np.random.seed(42)
nps_df = pd.DataFrame({'CustomerID': range(1,501), 'Age': np.random.randint(18,70,500), 'Region': np.random.choice(['North','South','East','West'],500), 'NPS_Score': np.random.randint(0,11,500)})
# Intentionally set 10% of NPS_Score as missing
idx = np.random.choice(nps_df.index, size=50, replace=False)
nps_df.loc[idx, 'NPS_Score'] = np.nan
print(nps_df.head())

   CustomerID  Age Region  NPS_Score
0           1   56   West        2.0
1           2   69  North        0.0
2           3   46   East        4.0
3           4   32   West        3.0
4           5   60   East        9.0

Advanced Example 2: Quantifying Impact of Missingness on Customer Metrics#

How do missing survey responses change your business conclusions?
We compare average NPS before and after dropping incomplete responses.

avg_with_missing = nps_df['NPS_Score'].mean()
avg_no_missing = nps_df['NPS_Score'].dropna().mean()
print(f'Average NPS (with missing as NaN): {avg_with_missing:.2f}')
print(f'Average NPS (excluding missing): {avg_no_missing:.2f}')

Average NPS (with missing as NaN): 4.89
Average NPS (excluding missing): 4.89

Advanced Example 3: Identifying and Handling Impossible Values in NPS#

NPS scores must be in range 0 to 10, inclusive.
We simulate and catch any impossible scores.

nps_df.loc[5,'NPS_Score'] = 15  # Simulate invalid score
invalid_idx = (nps_df['NPS_Score'] < 0) | (nps_df['NPS_Score'] > 10)
print('Indexes of impossible NPS scores:', nps_df[invalid_idx].index.tolist())
nps_df.loc[invalid_idx, 'NPS_Score'] = np.nan
print('Impossible values replaced with NaN.')

Indexes of impossible NPS scores: [5]
Impossible values replaced with NaN.

Error Handling Example: Common Mistakes with Missing Data#

Not all missing values are marked as NaN - sometimes blanks or special codes are used.
We will search for common representations of missing data beyond just NaN.

custom_missing = nps_df['Region'].isin(['', 'NA', 'Unknown', None])
print('Number of non-standard missing Region entries:', custom_missing.sum())

Number of non-standard missing Region entries: 0

Error Handling Example: Incorrect Grouping When Summarizing Survey Data#

A common analytics mistake is grouping by a column with too many missing values.
We will see how dropping missing groups changes the group average.

grouped = nps_df.groupby('Region')['NPS_Score'].mean()
print(grouped)
grouped_dropna = nps_df.dropna(subset=['Region']).groupby('Region')['NPS_Score'].mean()
print('Grouped means after dropping missing:', grouped_dropna)

Region
East     4.611650
North    5.158273
South    4.700935
West     4.980000
Name: NPS_Score, dtype: float64
Grouped means after dropping missing: Region
East     4.611650
North    5.158273
South    4.700935
West     4.980000
Name: NPS_Score, dtype: float64

Error Handling Example: Misreading Likert Scale or NPS Values#

Sometimes, analysts forget that 0 on an NPS or Likert scale is valid (not missing or 'bad').
We examine a summary to check if zeros are present and treated properly.

print('NPS value counts (including 0):')
print(nps_df['NPS_Score'].value_counts(dropna=False).sort_index())

NPS value counts (including 0):
NPS_Score
0.0     44
1.0     36
2.0     44
3.0     42
4.0     42
5.0     44
6.0     41
7.0     48
8.0     34
9.0     39
10.0    35
NaN     51
Name: count, dtype: int64

Best Practices: Survey Data Segmentation and Cross-Tab Analysis#

Segmenting respondents helps spot missingness patterns and improve survey targeting.
Cross-tabulation of missing data can uncover which groups tend to skip questions.
Let us analyze missing NPS by age group and region.

nps_df['AgeGroup'] = pd.cut(nps_df['Age'], bins=[17,30,40,50,60,80], labels=['18-30','31-40','41-50','51-60','61+'])
missing_xtab = pd.crosstab(nps_df['AgeGroup'], nps_df['Region'], values=nps_df['NPS_Score'].isnull(), aggfunc='sum', dropna=False)
print(missing_xtab)

Region    East  North  South  West
AgeGroup                          
18-30        0      3      3     5
31-40        1      2      4     2
41-50        3      1      0     4
51-60        2      3      4     3
61+          4      1      3     3

Best Practices: Constructing Missingness Indices and Reporting Trends#

Track the proportion of missing answers over time, region, or cohort.
A rising trend could reveal survey fatigue or disengaged segments.

# Simulate a survey timestamp column
nps_df['ResponseMonth'] = pd.to_datetime('2022-01-01') + pd.to_timedelta(np.random.randint(0,180, len(nps_df)), unit='D')
nps_df['Month'] = nps_df['ResponseMonth'].dt.to_period('M')
monthly_missing = nps_df.groupby('Month')['NPS_Score'].apply(lambda x: x.isnull().mean())
monthly_missing.plot(marker='o')
plt.ylabel('Fraction Missing')
plt.title('Monthly Survey NPS Missingness')
plt.show()

No description has been provided for this image

End-to-End Example: Flagging and Reporting on a Real Survey's Data Quality#

We will summarize missing or invalid NPS responses by customer region.
The final table makes it easy for executives to spot data issues by group.

report = nps_df.groupby('Region').agg(
    responses=('NPS_Score', 'count'),
    missing_count=('NPS_Score', lambda x: x.isnull().sum()),
    invalid_count=('NPS_Score', lambda x: ((x<0)|(x>10)).sum())
)
report['missing_pct'] = (report['missing_count'] / (report['responses'] + report['missing_count'])) * 100
print(report.sort_values('missing_pct', ascending=False))

        responses  missing_count  invalid_count  missing_pct
Region                                                      
West          100             17              0    14.529915
South         107             14              0    11.570248
East          103             10              0     8.849558
North         139             10              0     6.711409

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.

☕ Buy me a coffee Other ways to help