Mathew K AnalyticsLesson 41 · Market Research Analytics in Python
Cleaning Open-Ended Survey Text Data for Market Research in Python
Open-ended customer feedback is valuable for understanding real experiences. Businesses need to clean and organize text data to extract meaningful insights.…
- CourseMarket Research Analytics in Python
- Lesson41 of 56
- Video23 min
- FormatJupyter notebook · 18 code cells
What you'll learn
- Understanding Customer Feedback Data in Market Research
- Loading an Example Open-Ended Customer Feedback Dataset
- Beginner Example: Lowercasing All Feedback for Consistency
- Beginner Example: Removing Extra Spaces
- Beginner Example: Removing Punctuation from Feedback
- Intermediate Example: Removing Common Stop Words
- Intermediate Example: Identifying Duplicate Feedback
- Intermediate Example: Handling Missing or Blank Feedback
Data
No separate download needed — the notebook creates or downloads everything it uses.
📓 Full notebook
Download .ipynbCleaning Open-Ended Survey Text Data for Market Research#
- Open-ended customer feedback is valuable for understanding real experiences.
- Businesses need to clean and organize text data to extract meaningful insights.
- In this lesson, we will practice cleaning, standardizing, and analyzing text responses from customer surveys.
- You will learn practical methods to prepare messy feedback for sentiment, keyword, and topic analysis.
import pandas as pd
import numpy as np
import re
import warnings
warnings.filterwarnings('ignore')
Understanding Customer Feedback Data in Market Research#
- Survey data captures customer thoughts and opinions, often through free-text feedback.
- Each response can be messy: typos, slang, and varied ways of expressing ideas.
- Failure to clean text can result in missed insights or misleading analysis.
- Beginners often forget to standardize entries, remove irrelevant content, or check for empty or duplicate responses.
Loading an Example Open-Ended Customer Feedback Dataset#
- We will use a small dataset of customer feedback for hands-on practice.
- Each row is an individual response to an open-ended question about their experience.
- This data is realistic and highlights common issues found in survey free-text fields.
df = pd.DataFrame({'CustomerID':[1,2,3,4,5],
'Feedback':['Great service and friendly staff',
'Delivery was slow and packaging was poor',
'Excellent quality, will buy again',
'Customer support needs improvement',
'Good value for money']})
print(df.shape)
print(df.head(3))
Beginner Example: Lowercasing All Feedback for Consistency#
- Text responses are entered in many formats; lowercasing helps standardize for analysis.
- This is often the very first step before searching or matching keywords.
df['Feedback_lower'] = df['Feedback'].str.lower()
print(df[['Feedback', 'Feedback_lower']])
Beginner Example: Removing Extra Spaces#
- Customers sometimes use extra spaces, which can hurt text matching.
- Cleaning up whitespace early creates more reliable downstream analysis.
df['Feedback_stripped'] = df['Feedback_lower'].str.strip().replace(r'\s+', ' ', regex=True)
print(df[['Feedback_lower','Feedback_stripped']])
Beginner Example: Removing Punctuation from Feedback#
- Unnecessary punctuation can break keyword matching and noise up analysis.
- Standardizing text by removing such characters is essential.
df['Feedback_clean'] = df['Feedback_stripped'].str.replace(r'[^a-zA-Z0-9 ]', '', regex=True)
print(df[['Feedback_stripped','Feedback_clean']])
Intermediate Example: Removing Common Stop Words#
- Words like 'the', 'and', 'is' appear often but rarely add useful meaning.
- By removing stop words, businesses focus on the terms that carry the most feedback value.
stop_words = set(['and','was','the','is','for','to','will','with','again','needs'])
df['Feedback_nostop'] = df['Feedback_clean'].apply(lambda x: ' '.join([word for word in x.split() if word not in stop_words]))
print(df[['Feedback_clean','Feedback_nostop']])
Intermediate Example: Identifying Duplicate Feedback#
- Sometimes respondents may submit identical comments, either as a mistake or due to limited options.
- Duplicates can overstate particular trends, so identifying them is important for fair analysis.
# Insert an intentional duplicate to demonstrate
df.loc[5] = [6, 'Good value for money', 'good value for money', 'good value for money', 'good value for money', 'good value money']
duplicates = df.duplicated('Feedback_clean', keep=False)
print(df.loc[duplicates, ['CustomerID','Feedback_clean']])
Intermediate Example: Handling Missing or Blank Feedback#
- Open-ended responses are often skipped, resulting in empty or missing values.
- Deciding how to handle these is crucial to avoid skewing your analysis.
df.loc[7] = [8, None, None, None, None, None] # Simulate a customer who skipped feedback
num_missing = df['Feedback'].isnull().sum()
print(f'Missing feedback responses: {num_missing}')
Advanced Example: Removing Spelling Errors with Simple Mapping#
- Spelling mistakes in feedback can hurt pattern recognition and metric accuracy.
- While advanced spelling correction uses libraries, sometimes a manual mapping fixes the top issues quickly.
common_typos = {'recieve':'receive','qualty':'quality','servce':'service','delievry':'delivery'}
def fix_typos(text):
if pd.isnull(text): return text
words = text.split()
return ' '.join([common_typos.get(word, word) for word in words])
df['Feedback_fixed'] = df['Feedback_nostop'].apply(fix_typos)
print(df[['Feedback_nostop','Feedback_fixed']])
Advanced Example: Keyword Tagging for Business Themes#
- Tagging feedback by keywords or business themes allows quick aggregation and reporting.
- Here we tag feedback by the presence of 'delivery', 'service', or 'support' related comments.
themes = {'delivery':'delivery','service':'service','support':'support'}
def assign_themes(text):
if pd.isnull(text): return []
words = set(text.split())
return [theme for theme in themes if theme in words]
df['Themes'] = df['Feedback_fixed'].apply(assign_themes)
print(df[['Feedback_fixed','Themes']])
Advanced Example: Simple Sentiment Analysis with Lexicon#
- Assigning positive or negative sentiment helps prioritize issues and celebrate successes.
- We use a rule-based approach as an easy introduction.
positive = set(['great','excellent','good','friendly','value','quality'])
negative = set(['poor','slow','needs','improvement'])
def get_sentiment(text):
if pd.isnull(text): return None
tokens = set(text.split())
if tokens & positive: return 'positive'
if tokens & negative: return 'negative'
return 'neutral'
df['Sentiment'] = df['Feedback_fixed'].apply(get_sentiment)
print(df[['Feedback_fixed','Sentiment']])
Error Handling Example: Excluding Feedback with Only Stop Words or Empty Comments#
- Some responses may only have unhelpful words or be totally blank.
- Analyses should remove or flag these to protect result quality.
df['OnlyStop'] = df['Feedback_nostop'].apply(lambda x: True if (pd.isnull(x) or len(str(x).strip())==0) else False)
excl = df[df['OnlyStop']]
print(excl[['CustomerID','Feedback','Feedback_nostop','OnlyStop']])
Error Handling Example: Detecting Unintended Numeric Feedback#
- Sometimes customers enter numbers instead of text, either by mistake or because of a flawed survey design.
- It is best practice to check and correct these before extracting insights.
# Demonstration: Insert numeric-only feedback for validation testing
df.loc[9] = [10, '12345', '54321', '11111', '22222', '33333', '44444', '55555', '66666', '77777']
numeric_rows = df['Feedback_clean'].str.fullmatch(r'\d+', na=False)
print(df.loc[numeric_rows, ['CustomerID','Feedback','Feedback_clean']])
Error Handling Example: Handling Inconsistent Groupings or Aggregations#
- Beginners sometimes group text data by mismatched or uncleaned variants, underestimating shared feedback themes.
- Consistent cleaning ensures groupings represent true business topics.
# Group by raw vs cleaned theme tags to see the impact
theme_counts_raw = df.groupby('Feedback').size().reset_index(name='RawCount')
theme_counts_clean = df.groupby('Feedback_fixed').size().reset_index(name='CleanCount')
print(theme_counts_raw.head(3))
print(theme_counts_clean.head(3))
Best Practices for Market Research Text Analytics#
- Always check for blank, duplicate, or obviously irrelevant responses before analysis.
- Clean and standardize text before searching for business themes or aggregating counts.
- Keep the raw feedback for audit, but use cleaned versions for reporting.
- Use keyword tagging and sentiment rules to prioritize team focus.
- Combine open-ended feedback with segmentation to discover hidden trends.
Best Practice Example: Segmenting Feedback by Sentiment#
- Analyzing feedback separately for positive and negative sentiment gives clearer action plans.
- Segmenting enables personalized follow-up and performance tracking.
segment_counts = df.groupby('Sentiment').size().reset_index(name='Count')
print(segment_counts)
Best Practice Example: Cross-Tabulation of Themes and Sentiments#
- Cross-tabulating sentiment and business theme shows which issues are most positive or negative.
- This pinpoints what needs urgent action or is performing especially well.
import itertools
records = []
for _, row in df.iterrows():
for theme in row['Themes']:
records.append({'Theme': theme, 'Sentiment': row['Sentiment']})
theme_sentiment_df = pd.DataFrame(records)
ct = pd.crosstab(theme_sentiment_df.Theme, theme_sentiment_df.Sentiment)
print(ct)
End-to-End MARKET RESEARCH Example: Calculating the Top Negative Theme and Suggested Next Step#
- Let us use our cleaned feedback data to find the main source of dissatisfaction and offer a solution.
- This practice links raw customer feedback analysis to real business improvement recommendations.
# Find the theme with the highest negative feedback
neg_theme_counts = theme_sentiment_df[theme_sentiment_df.Sentiment=='negative'].Theme.value_counts()
if not neg_theme_counts.empty:
top_neg_theme = neg_theme_counts.idxmax()
print(f'The most negative feedback theme is: {top_neg_theme}. Recommendation: Launch a focused improvement project in this area.')
else:
print('No clear negative themes found; continue to monitor for emerging issues.')
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.
Related lessons
14 minLesson 15
Comprehensive Guide to String Operations and Text Cleaning with Pandas in Python
20 minLesson 11
Effective Methods for Cleaning and Managing Missing Data in Banking Datasets Using Python
14 minLesson 6
Master Data Cleaning and Preparation with Pandas in Python for Effective Bible Study Analysis
29 minLesson 3