Mathew K Analytics

Lesson 33 · Market Research Analytics in Python

Comparing Groups Using t-Tests: A Practical Guide for Market Research Analytics in Python

In this lesson, we will compare two or more customer groups using t-tests to find if their averages are significantly different. This analysis helps…

⬇ Download notebook🧠 Quiz & exercises

What you'll learn

Data

No separate download needed — the notebook creates or downloads everything it uses.

📓 Full notebook

Download .ipynb

Comparing Customer Groups with t-Tests in Market Research#

  • In this lesson, we will compare two or more customer groups using t-tests to find if their averages are significantly different.
  • This analysis helps businesses make data-driven decisions, such as targeting different demographics or evaluating satisfaction based on customer segments.
  • We will use real datasets to understand t-tests in customer analytics.
  • By the end, you will be able to identify when differences between groups are real or just due to random chance.
import pandas as pd
import numpy as np
import openml
import scipy.stats as stats
import warnings
warnings.filterwarnings('ignore')

Understanding Customer Data in Market Research#

  • Customer data can include demographics (like gender, age, region), ratings (like satisfaction scores), and behaviors (like retention).
  • Surveys may have missing responses or misunderstood questions, which can affect analysis.
  • Beginners sometimes group data incorrectly or do not account for scales (like treating categories as numbers).
  • It is important to clearly define groups (like male/female, churned/stayed, campaign/controls) before analysis.
# Beginner Example 1: Load a customer satisfaction dataset from OpenML
dataset = openml.datasets.get_dataset(42178)
df, _, _, _ = dataset.get_data(dataset_format='dataframe')
print(df.shape)
print(df.head(3))
(7043, 20)
   gender  SeniorCitizen Partner Dependents  tenure PhoneService  \
0  Female              0     Yes         No       1           No   
1    Male              0      No         No      34          Yes   
2    Male              0      No         No       2          Yes   

      MultipleLines InternetService OnlineSecurity OnlineBackup  \
0  No phone service             DSL             No          Yes   
1                No             DSL            Yes           No   
2                No             DSL            Yes          Yes   

  DeviceProtection TechSupport StreamingTV StreamingMovies        Contract  \
0               No          No          No              No  Month-to-month   
1              Yes          No          No              No        One year   
2               No          No          No              No  Month-to-month   

  PaperlessBilling     PaymentMethod  MonthlyCharges TotalCharges Churn  
0              Yes  Electronic check           29.85        29.85    No  
1               No      Mailed check           56.95       1889.5    No  
2              Yes      Mailed check           53.85       108.15   Yes  
# Beginner Example 2: Check for missing values
missing = df.isnull().sum()
print(missing[missing > 0])
Series([], dtype: int64)
# Beginner Example 3: Compare Monthly Charges by Gender using t-test
male = df[df['gender'] == 'Male']['MonthlyCharges'].dropna()
female = df[df['gender'] == 'Female']['MonthlyCharges'].dropna()
t_stat, p_value = stats.ttest_ind(male, female, equal_var=False)
print('t-statistic:', t_stat)
print('p-value:', p_value)
t-statistic: -1.2226726068634024
p-value: 0.22149431343928863
# Beginner Example 4: Visualize Monthly Charges Distribution by Gender
import matplotlib.pyplot as plt
plt.hist(male, bins=30, alpha=0.6, label='Male')
plt.hist(female, bins=30, alpha=0.6, label='Female')
plt.xlabel('Monthly Charges')
plt.ylabel('Number of Customers')
plt.title('Distribution of Monthly Charges by Gender')
plt.legend()
plt.show()
No description has been provided for this image
# Beginner Example 5: Summarize mean and standard deviation for groups
print('Male monthly charges: Mean = {:.2f}, SD = {:.2f}'.format(male.mean(), male.std()))
print('Female monthly charges: Mean = {:.2f}, SD = {:.2f}'.format(female.mean(), female.std()))
Male monthly charges: Mean = 64.33, SD = 30.12
Female monthly charges: Mean = 65.20, SD = 30.06
# Intermediate Example 1: Load marketing campaign dataset and prepare group array
dataset2 = openml.datasets.get_dataset(1461)
df2, _, _, _ = dataset2.get_data(dataset_format='dataframe')
df2.columns = ['age','job','marital','education','default','balance','housing','loan','contact','day','month','duration','campaign','pdays','previous','poutcome','response']
print(df2[['age','duration','response']].head())
   age  duration response
0   58     261.0        1
1   44     151.0        1
2   33      76.0        1
3   47      92.0        1
4   33     198.0        1
# Intermediate Example 2: Test if campaign responders spent longer on calls
yes = df2[df2['response'] == 'yes']['duration']
no = df2[df2['response'] == 'no']['duration']
t_stat2, p_value2 = stats.ttest_ind(yes, no, equal_var=False)
print('t-statistic:', t_stat2)
print('p-value:', p_value2)
t-statistic: nan
p-value: nan
# Intermediate Example 3: Calculate effect size (Cohen's d) for business impact
def cohens_d(a, b):
    return (a.mean() - b.mean()) / np.sqrt((a.std() ** 2 + b.std() ** 2) / 2)
effect_size = cohens_d(yes, no)
print('Cohen\'s d effect size:', effect_size)
Cohen's d effect size: nan
# Intermediate Example 4: Compare age between responders and non-responders
age_yes = df2[df2['response'] == 'yes']['age']
age_no = df2[df2['response'] == 'no']['age']
t_stat_age, p_value_age = stats.ttest_ind(age_yes, age_no, equal_var=False)
print('Age t-statistic:', t_stat_age)
print('Age p-value:', p_value_age)
Age t-statistic: nan
Age p-value: nan
# Intermediate Example 5: Visualize campaign response rates by education
import seaborn as sns
sns.countplot(y='education', hue='response', data=df2)
plt.title('Campaign Response by Education Level')
plt.xlabel('Number of Customers')
plt.ylabel('Education Level')
plt.show()
No description has been provided for this image
# Intermediate Example 6: Analyze cross-tab of campaign response and job
cross = pd.crosstab(df2['job'], df2['response'])
print(cross)
response          1     2
job                      
admin.         4540   631
blue-collar    9024   708
entrepreneur   1364   123
housemaid      1131   109
management     8157  1301
retired        1748   516
self-employed  1392   187
services       3785   369
student         669   269
technician     6757   840
unemployed     1101   202
unknown         254    34
# Advanced Example 1: Using NPS survey to compare regions (synthetic data)
np.random.seed(42)
df_nps = pd.DataFrame({
    'CustomerID': range(1,501),
    'Age': np.random.randint(18,70,500),
    'Region': np.random.choice(['North','South','East','West'],500),
    'NPS_Score': np.random.randint(0,11,500)
})
region1 = df_nps[df_nps['Region'] == 'North']['NPS_Score']
region2 = df_nps[df_nps['Region'] == 'South']['NPS_Score']
t_stat_nps, p_value_nps = stats.ttest_ind(region1, region2, equal_var=False)
print('NPS t-statistic:', t_stat_nps)
print('NPS p-value:', p_value_nps)
NPS t-statistic: 1.3137700571086297
NPS p-value: 0.19009510997777748
# Advanced Example 2: Compare NPS categories (Promoters vs. Detractors)
promoters = df_nps[df_nps['NPS_Score'] >= 9]
detractors = df_nps[df_nps['NPS_Score'] <= 6]
age_promoters = promoters['Age']
age_detractors = detractors['Age']
t_stat_agecat, p_value_agecat = stats.ttest_ind(age_promoters, age_detractors, equal_var=False)
print('Age t-statistic between Promoters and Detractors:', t_stat_agecat)
print('p-value:', p_value_agecat)
Age t-statistic between Promoters and Detractors: 0.6293693677078811
p-value: 0.5302590146459578
# Advanced Example 3: Handle multiple group t-tests with Bonferroni correction
regions = df_nps['Region'].unique()
results = []
for i in range(len(regions)):
    for j in range(i+1, len(regions)):
        group1 = df_nps[df_nps['Region'] == regions[i]]['NPS_Score']
        group2 = df_nps[df_nps['Region'] == regions[j]]['NPS_Score']
        t_stat, p_val = stats.ttest_ind(group1, group2, equal_var=False)
        results.append({'regionA': regions[i], 'regionB': regions[j], 'p_value': p_val})
# Bonferroni-corrected alpha for all pairwise tests
alpha = 0.05 / len(results)
sig_results = [r for r in results if r['p_value'] < alpha]
print('Significant differences after correction:', sig_results)
Significant differences after correction: []
# Error Handling Example 1: What if there are missing NPS values?
df_nps_nan = df_nps.copy()
df_nps_nan.loc[0:9, 'NPS_Score'] = np.nan  # Set first 10 NPS_Score as missing
clean_nps = df_nps_nan['NPS_Score'].dropna()
print('Clean NPS shape after dropping NaN:', clean_nps.shape)
Clean NPS shape after dropping NaN: (490,)
# Error Handling Example 2: What happens if groups are empty?
empty_group = df_nps[df_nps['Region'] == 'Central']['NPS_Score'] if 'Central' in df_nps['Region'].unique() else pd.Series([])
if empty_group.empty:
    print('No customers in this region (Central). Test cannot be performed.')
No customers in this region (Central). Test cannot be performed.
# Error Handling Example 3: Incorrect grouping (comparing continuous rather than categorical)
try:
    stats.ttest_ind(df_nps['NPS_Score'], df_nps['Age'])
except Exception as e:
    print('Error:', e)
# Error Handling Example 4: Misinterpreting Likert scales as interval
df_nps['Likert'] = np.random.choice(['Strongly Disagree', 'Disagree', 'Neutral', 'Agree', 'Strongly Agree'], 500)
try:
    stats.ttest_ind(df_nps[df_nps['Likert'] == 'Strongly Agree']['NPS_Score'],
                   df_nps[df_nps['Likert'] == 'Strongly Disagree']['NPS_Score'])
    print('T-test completed, but be cautious: Likert scale is ordinal, not interval.')
except Exception as e:
    print('Error:', e)
T-test completed, but be cautious: Likert scale is ordinal, not interval.

Best Practices in t-Test Market Analytics#

  • Define your business groups (segments) before running tests.
  • Clean and check for missing data, as it can bias results.
  • Use effect size to explain practical impact, not just p-values.
  • Use cross-tabs for categorical comparisons, not t-tests.
  • Respect scale types: use t-tests for interval/ratio scales only.
  • Segment and report your findings visually for business audiences.

Mini End-to-End Example: Are Customers with 'Tech Support' Paying More?#

  • We will use the customer satisfaction survey dataset.
  • We compare average MonthlyCharges for two groups: those with and without tech support.
  • This simulates a real-world business question: Does providing tech support drive higher charges or reflect higher-spend customers?
  • We clean, group, test, and interpret a result for management recommendation.
# Step 1: Prepare groups and clean numeric column
with_tech = df[df['TechSupport'] == 'Yes']['MonthlyCharges'].dropna()
without_tech = df[df['TechSupport'] == 'No']['MonthlyCharges'].dropna()
print('Sample sizes:', len(with_tech), len(without_tech))
Sample sizes: 2044 3473
# Step 2: Run the t-test and interpret result
t_stat_final, p_value_final = stats.ttest_ind(with_tech, without_tech, equal_var=False)
print('t-statistic:', t_stat_final)
print('p-value:', p_value_final)
if p_value_final < 0.05:
    print('Result: There is a significant difference in charges. Business may explore pricing strategies!')
else:
    print('Result: No significant difference detected. Keep tech support pricing model stable.')
t-statistic: 9.963543273660985
p-value: 3.9913000753750244e-23
Result: There is a significant difference in charges. Business may explore pricing strategies!
 

🧠 Test yourself

Pick an answer to check it. Your best score is saved in this browser.

  1. What does equal_var=False do in stats.ttest_ind(a, b, equal_var=False)?

  2. A t-test comparing MonthlyCharges between two groups gives p = 0.003. What is the correct reading at alpha = 0.05?

  3. Why does the lesson compute Cohen's d on top of the p-value?

  4. With 4 regions there are 6 pairwise t-tests. What Bonferroni-corrected alpha does the lesson use?

  5. Why does the lesson warn about running a t-test on Likert answers like 'Agree' / 'Disagree'?

💻 Practice exercises

Try each one in the notebook or in your own editor before opening the solution.

  1. Create two lists of 10 monthly charge values (male and female customers). Run Welch's t-test, print the t-statistic and p-value, and print each group's mean and standard deviation.

    💡 Hint

    Use scipy.stats.ttest_ind(a, b, equal_var=False).

    ✅ Show solution
    import pandas as pd
    from scipy import stats
    
    male = pd.Series([70.5, 65.0, 80.2, 55.5, 90.1, 60.0, 75.3, 68.4, 85.0, 72.2])
    female = pd.Series([72.0, 66.5, 78.0, 58.0, 88.5, 63.0, 74.1, 70.0, 83.2, 71.0])
    t_stat, p_value = stats.ttest_ind(male, female, equal_var=False)
    print('t-statistic:', round(t_stat, 3))
    print('p-value:', round(p_value, 3))
    print('Male: mean = {:.2f}, SD = {:.2f}'.format(male.mean(), male.std()))
    print('Female: mean = {:.2f}, SD = {:.2f}'.format(female.mean(), female.std()))
    
  2. Create a small campaign DataFrame (16 rows) with response ('yes'/'no') and call duration. Write a cohens_d(a, b) function, test whether responders talked longer with Welch's t-test, and print the p-value, Cohen's d, and a plain-English verdict.

    💡 Hint

    Cohen's d = (mean_a - mean_b) / sqrt((sd_a^2 + sd_b^2) / 2).

    ✅ Show solution
    import numpy as np
    import pandas as pd
    from scipy import stats
    
    df = pd.DataFrame({
        'response': ['yes'] * 6 + ['no'] * 10,
        'duration': [420, 380, 510, 450, 390, 470, 180, 220, 150, 260, 200, 240, 170, 210, 190, 230],
    })
    yes = df[df['response'] == 'yes']['duration']
    no = df[df['response'] == 'no']['duration']
    
    def cohens_d(a, b):
        return (a.mean() - b.mean()) / np.sqrt((a.std() ** 2 + b.std() ** 2) / 2)
    
    t_stat, p_value = stats.ttest_ind(yes, no, equal_var=False)
    d = cohens_d(yes, no)
    print('p-value:', round(p_value, 5))
    print("Cohen's d:", round(d, 2))
    if p_value < 0.05 and abs(d) >= 0.8:
        print('Responders talk significantly longer, and the effect is large.')
    elif p_value < 0.05:
        print('Significant, but the effect is modest.')
    else:
        print('No significant difference.')
    
  3. Create a synthetic NPS-style survey of 200 customers across four regions (use a seeded random generator and make 'North' slightly higher on average). Run Welch t-tests for every pair of regions, apply a Bonferroni-corrected alpha, and print the pairs that remain significant.

    💡 Hint

    Use itertools.combinations and alpha = 0.05 / number_of_tests.

    ✅ Show solution
    import numpy as np
    import pandas as pd
    from itertools import combinations
    from scipy import stats
    
    rng = np.random.default_rng(42)
    regions = ['North', 'South', 'East', 'West']
    df = pd.DataFrame({'Region': np.repeat(regions, 50)})
    means = {'North': 8.0, 'South': 6.0, 'East': 6.2, 'West': 6.1}
    df['Score'] = [np.clip(rng.normal(means[r], 1.5), 0, 10) for r in df['Region']]
    
    pairs = list(combinations(regions, 2))
    alpha = 0.05 / len(pairs)
    print('Corrected alpha:', round(alpha, 4))
    significant = []
    for a, b in pairs:
        t, p = stats.ttest_ind(df[df['Region'] == a]['Score'], df[df['Region'] == b]['Score'], equal_var=False)
        if p < alpha:
            significant.append((a, b, round(p, 6)))
    print('Significant after correction:', significant)
    

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.