Mathew K Analytics

Lesson 24 · Real-World Data Analytics

A/B Testing in Python: Real Mobile Game Experiment (Cookie Cats) | Data Analytics #24

Analyse a real A/B test in Python: a mobile game moved its first gate from level 30 to level 40. Did it hurt player retention? We check group balance,…

What you'll learn

Datasets used in this lesson

Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.

📓 Full notebook

Download .ipynb

Data Analytics 100, Video 24: Marketing Campaign A/B Test Analysis#

  • Video twenty-four of the hundred-video real-world data analytics series.
  • A real mobile game genuinely A/B tested moving a paywall gate from level thirty to level forty.
  • Let's get into it.

This lesson is for learners who can summarise data with pandas and want to analyse a controlled experiment properly. An A/B test randomly splits users into two groups, shows each group a different version of a product, and compares outcomes. Because the split is random, a real difference in outcomes can be credited to the change itself.

The business question comes from a mobile game that tested moving a paywall gate, a point where players must wait or pay to continue, from level 30 to level 40. Does the later gate keep more players coming back? The data is loaded from ab_test_cookie_cats_raw.csv, with 2,256 players. The key columns are version (gate_30 or gate_40), sum_gamerounds (rounds played), and retention_1 and retention_7, which record whether a player returned 1 day and 7 days after installing.

In this lesson you will:

  • compare retention and engagement between the two groups
  • test the gaps with chi-square and Mann-Whitney tests and a confidence interval
  • separate statistical significance from effect size and business impact

Prerequisites: you should be comfortable with pandas groupby and have a basic idea of what a p-value is. Earlier lessons in the series cover both.

Part 1: A Real Mobile Game A/B Test#

Every experiment analysis starts by loading the data and checking its size. This step imports pandas, numpy, matplotlib and the stats module from scipy, which holds the statistical tests used later. It reads the test results into ab_test and checks .shape. The output (2256, 5) means 2,256 players and five columns. That is enough to spot large effects, but small differences of one or two percentage points may be hard to separate from chance with a sample this size.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats
ab_test = pd.read_csv('ab_test_cookie_cats_raw.csv')
ab_test.shape
(2256, 5)

Part 2: Real Data Preview#

Checking data types early prevents subtle calculation errors. The code calls head(5) and then dtypes, but Jupyter only displays the last expression, so you see the types rather than the rows. userid and sum_gamerounds are whole numbers, version is text (shown as object), and both retention columns are bool, meaning True or False. Booleans are handy: taking .mean() of a True/False column gives the share of Trues, which is exactly a retention rate.

ab_test.head(5)
ab_test.dtypes
userid             int64
version           object
sum_gamerounds     int64
retention_1         bool
retention_7         bool
dtype: object

Part 3: The Real Two Groups#

You need to know how many players landed in each version before comparing them. value_counts() counts rows per version. The output shows 1,135 players in gate_40 and 1,121 in gate_30. The groups are very close in size, which is what you would expect from a random split. A large imbalance would be a warning sign that something went wrong with how players were assigned.

group_counts = ab_test['version'].value_counts()
group_counts
version
gate_40    1135
gate_30    1121
Name: count, dtype: int64

Part 4: Real Group Balance Check#

A sample ratio check confirms that the split matches what was planned, here roughly 50/50. Dividing the counts by the total number of rows and multiplying by 100 gives each group's share. The output shows 50.31% in gate_40 and 49.69% in gate_30. That small difference is well within what chance would produce. If you ever see a split like 55/45 in a test meant to be 50/50, investigate before trusting any results.

group_share = (group_counts / len(ab_test) * 100).round(2)
group_share
version
gate_40    50.31
gate_30    49.69
Name: count, dtype: float64

Part 5: Real One-Day Retention by Group#

One-day retention measures whether the change affects players' early experience. groupby('version')['retention_1'].mean() gives the share of players who came back the next day in each group; .round(4) * 100 turns it into a percentage. The output shows 43.44% for gate_30 and 43.00% for gate_40. The difference is under half a percentage point. That makes sense: most players would not reach level 30 or 40 on their first day, so the gate position should barely matter yet.

retention_1_by_group = ab_test.groupby('version')['retention_1'].mean().round(4) * 100
retention_1_by_group
version
gate_30    43.44
gate_40    43.00
Name: retention_1, dtype: float64

Part 6: Real Seven-Day Retention by Group#

Seven-day retention is the more important metric here, because by then more players will have reached the gate. The same groupby and .mean() approach on retention_7 shows 19.18% for gate_30 against 17.71% for gate_40. So the earlier gate kept slightly more players returning after a week, a gap of 1.47 percentage points. The next question is whether that gap is real or just noise from random assignment.

retention_7_by_group = ab_test.groupby('version')['retention_7'].mean().round(4) * 100
retention_7_by_group
version
gate_30    19.18
gate_40    17.71
Name: retention_7, dtype: float64

Part 7: Visualizing Real Retention by Group#

Side-by-side charts help stakeholders compare the two metrics at a glance. plt.subplots(1, 2) creates two panels in one figure, and each retention series is drawn into its own panel by passing ax=axes[0] or ax=axes[1] to .plot. The figure is saved to retention_by_group.png and closed, so there is no output here. In the image, expect the two 1-day bars to look almost identical and the 7-day bars to show a small gap, with both 7-day bars much lower than the 1-day ones.

fig, axes = plt.subplots(1, 2, figsize=(11, 5))
retention_1_by_group.plot(kind='bar', ax=axes[0], color=['steelblue', 'darkorange'])
axes[0].set_title('Real 1-Day Retention (%)')
axes[0].set_ylabel('Real Retention Rate (%)')
retention_7_by_group.plot(kind='bar', ax=axes[1], color=['steelblue', 'darkorange'])
axes[1].set_title('Real 7-Day Retention (%)')
plt.tight_layout()
plt.savefig('retention_by_group.png', dpi=120)
plt.close()

Part 8: Real Statistical Significance, One-Day Retention#

A difference between groups always appears in a sample, even when the change does nothing, so you need a test to judge it. pd.crosstab builds a contingency table: counts of retained and not retained players per version. stats.chi2_contingency runs a chi-square test, which checks whether the retention pattern differs between groups more than chance would explain. The p-value is 0.8633. That is very high, meaning a gap this size would be entirely normal if both versions performed the same.

contingency_1 = pd.crosstab(ab_test['version'], ab_test['retention_1'])
chi2_1, p_value_1, dof_1, expected_1 = stats.chi2_contingency(contingency_1)
round(p_value_1, 4)
np.float64(0.8633)

Part 9: Real Statistical Significance, Seven-Day Retention#

The same chi-square test is applied to the more important 7-day metric. A new contingency table counts 7-day retention by version, and chi2_contingency returns the p-value. The output is 0.3976. A p-value is the probability of seeing a gap at least this large if there were truly no difference. About a 40% chance means the 1.47-point gap could quite easily be random noise in a sample of this size.

contingency_7 = pd.crosstab(ab_test['version'], ab_test['retention_7'])
chi2_7, p_value_7, dof_7, expected_7 = stats.chi2_contingency(contingency_7)
round(p_value_7, 4)
np.float64(0.3976)

Part 10: Real Honest Interpretation#

Analysts decide in advance what counts as significant so they cannot move the goalposts after seeing results. The code sets the usual threshold alpha = 0.05 and checks whether each p-value is below it. Both results come back False. So neither the 1-day nor the 7-day difference is statistically significant in this sample. That does not prove the versions are equal; it means this data cannot confidently tell them apart.

alpha = 0.05
significant_1 = p_value_1 < alpha
significant_7 = p_value_7 < alpha
significant_1, significant_7
(np.False_, np.False_)

Part 11: Real Engagement, Total Game Rounds#

Retention is not the only outcome; engagement shows how much people play. groupby('version')['sum_gamerounds'].mean() gives the average rounds per player. The output shows 54.83 for gate_30 and 44.70 for gate_40, a gap of about 10 rounds. That looks large, but averages of game activity are easily pulled up by a few very heavy players, so this result needs checking with more robust measures before you trust it.

gamerounds_by_group = ab_test.groupby('version')['sum_gamerounds'].mean().round(2)
gamerounds_by_group
version
gate_30    54.83
gate_40    44.70
Name: sum_gamerounds, dtype: float64

Part 12: Real Median Engagement, Outlier-Resistant#

The median is the middle value when players are sorted, so a handful of extreme players cannot drag it around. groupby(...)['sum_gamerounds'].median() shows exactly 15.0 rounds for both groups. So the typical player in each version played the same amount, and the 10-round gap in averages must come from a small number of heavy players. When mean and median disagree like this, the data is skewed and the median usually gives the fairer picture.

gamerounds_median_by_group = ab_test.groupby('version')['sum_gamerounds'].median()
gamerounds_median_by_group
version
gate_30    15.0
gate_40    15.0
Name: sum_gamerounds, dtype: float64

Part 13: Real Outlier Check#

Extreme values explain why the mean and median disagreed. The code finds the highest number of rounds with .max() and counts players who played zero rounds, using a comparison and .sum(). The output shows one player with 1,906 rounds and 97 players who installed but never played. A single player at 1,906 can lift a group's average noticeably. Zero-round players are worth noting too, as they never had any chance to meet the gate.

max_rounds = ab_test['sum_gamerounds'].max()
zero_rounds = (ab_test['sum_gamerounds'] == 0).sum()
max_rounds, zero_rounds
(np.int64(1906), np.int64(97))

Part 14: Real Mann-Whitney Test on Engagement#

Because game rounds are so skewed, a test based on averages would be misleading. The Mann-Whitney U test compares two groups using the ranks of values rather than the values themselves, so a few extreme players have little influence. It is called a non-parametric test because it does not assume a bell-shaped distribution. alternative='two-sided' tests for a difference in either direction. The p-value is 0.487, so there is no evidence of a real difference in engagement between versions.

gate_30_rounds = ab_test.loc[ab_test['version'] == 'gate_30', 'sum_gamerounds']
gate_40_rounds = ab_test.loc[ab_test['version'] == 'gate_40', 'sum_gamerounds']
mw_stat, mw_p = stats.mannwhitneyu(gate_30_rounds, gate_40_rounds, alternative='two-sided')
round(mw_p, 4)
np.float64(0.487)

Part 15: Real Distribution of Game Rounds#

A histogram shows the full shape of engagement, which a single number cannot. The code filters to players with fewer than 200 rounds, so a few extreme players do not squash the chart, then passes both groups to one plt.hist call so their bars sit side by side across 30 bins. The chart is saved to gamerounds_distribution.png and closed, so nothing displays. In the image, look for most players bunched at the low end with a long tail to the right, and for the two groups' shapes being very similar.

capped_rounds = ab_test[ab_test['sum_gamerounds'] < 200]
plt.figure(figsize=(9, 5))
plt.hist([capped_rounds.loc[capped_rounds['version']=='gate_30','sum_gamerounds'], capped_rounds.loc[capped_rounds['version']=='gate_40','sum_gamerounds']], bins=30, label=['gate_30','gate_40'], color=['steelblue','darkorange'], alpha=0.7)
plt.xlabel('Real Game Rounds Played (capped at 200)')
plt.ylabel('Real Number of Players')
plt.title('Real Distribution of Game Rounds by Gate Version')
plt.legend()
plt.tight_layout()
plt.savefig('gamerounds_distribution.png', dpi=120)
plt.close()

Part 16: Real Retention Funnel, Both Milestones#

A player who returns on both day 1 and day 7 is more clearly engaged than one who returns only once. The & operator combines the two True/False columns, giving True only when both are True. Averaging by version shows 14.63% of gate_30 players and 13.39% of gate_40 players returned on both days. The pattern matches the 7-day result: gate_30 slightly ahead, by a little over one percentage point.

ab_test['retained_both'] = ab_test['retention_1'] & ab_test['retention_7']
retained_both_by_group = ab_test.groupby('version')['retained_both'].mean().round(4) * 100
retained_both_by_group
version
gate_30    14.63
gate_40    13.39
Name: retained_both, dtype: float64

Part 17: Real Conditional Retention#

Conditional retention asks a narrower question: among players who came back on day 1, how many were still around on day 7? The code filters to day-1 returners, then averages retention_7 by version. The output shows 33.68% for gate_30 and 31.15% for gate_40. This is useful because it removes players who left immediately, before the gate could matter. The gap is about 2.5 points, but with a smaller group, it carries even more uncertainty.

day1_returners = ab_test[ab_test['retention_1'] == True]
conditional_retention = day1_returners.groupby('version')['retention_7'].mean().round(4) * 100
conditional_retention
version
gate_30    33.68
gate_40    31.15
Name: retention_7, dtype: float64

Part 18: Real Effect Size, Not Just P-Values#

A p-value tells you whether a gap is likely real; an effect size tells you how big it is. The code calculates two versions. The percentage-point gap is a simple subtraction: 1.47 points. The relative lift divides one rate by the other, giving 8.3%: gate_30 retention is about 8% higher relative to gate_40. Both describe the same effect. Relative lift sounds more impressive, so always report the point gap alongside it to avoid overstating results.

retention_7_gap_pct_points = round(retention_7_by_group['gate_30'] - retention_7_by_group['gate_40'], 2)
retention_7_relative_lift = round((retention_7_by_group['gate_30'] / retention_7_by_group['gate_40'] - 1) * 100, 2)
retention_7_gap_pct_points, retention_7_relative_lift
(np.float64(1.47), np.float64(8.3))

Part 19: Real Sample Size Sensitivity Check#

Sample size controls how precisely you can measure a difference, so it is worth stating it clearly. The code counts players in each version with a comparison and .sum(), then adds them. The output shows 1,121 in gate_30, 1,135 in gate_40, and 2,256 in total. With around 1,100 players per group, small gaps like 1.47 points carry a lot of uncertainty. Detecting gaps that small reliably usually needs many thousands of players per group.

n_gate_30 = (ab_test['version'] == 'gate_30').sum()
n_gate_40 = (ab_test['version'] == 'gate_40').sum()
n_gate_30, n_gate_40, n_gate_30 + n_gate_40
(np.int64(1121), np.int64(1135), np.int64(2256))

Part 20: Real Confidence Interval on the Gap#

A confidence interval gives a range of plausible values for the true gap, which is often more useful than a p-value. The code converts both 7-day rates back to proportions, calculates the standard error of the difference with the usual formula sqrt(p1(1-p1)/n1 + p2(1-p2)/n2), and adds and subtracts 1.96 times it for a 95% interval. The result runs from -1.73 to 4.67 percentage points. Because that range includes zero, gate_40 could even be slightly better, which matches the non-significant chi-square test.

p1 = retention_7_by_group['gate_30'] / 100
p2 = retention_7_by_group['gate_40'] / 100
se = np.sqrt(p1*(1-p1)/n_gate_30 + p2*(1-p2)/n_gate_40)
ci_low = round((p1 - p2 - 1.96*se) * 100, 2)
ci_high = round((p1 - p2 + 1.96*se) * 100, 2)
ci_low, ci_high
(np.float64(-1.73), np.float64(4.67))

Part 21: Real Recommendation Logic#

Turning results into a decision rule is part of the job, but the rule must be read with care. The code simply checks whether gate_30 has higher 7-day retention, and the output is True. This comparison ignores uncertainty, so on its own it is not a strong recommendation. A fair reading is that gate_30 performed slightly better in this sample, the evidence is not conclusive, and keeping the gate at 30 is a reasonable default rather than a proven win.

recommend_gate_30 = retention_7_by_group['gate_30'] > retention_7_by_group['gate_40']
recommend_gate_30
np.True_

Part 22: Real Correlation, Engagement and Retention#

It is worth checking that engagement and retention are linked, because that supports using game rounds as an early signal. .astype(int) turns True and False in retention_7 into 1 and 0 so a correlation can be calculated, and .corr() gives the Pearson correlation, a number from -1 to +1 for how closely two columns move together. The output is 0.527, a moderately strong positive link: players who play more rounds are much more likely to return after a week.

engagement_retention_corr = ab_test['sum_gamerounds'].corr(ab_test['retention_7'].astype(int))
round(engagement_retention_corr, 3)
np.float64(0.527)

Part 23: Real High-Engagement Segment#

An effect can be hidden or concentrated in a particular group of users. .quantile(0.75) finds the round count that 75% of players fall at or below, and the filter keeps only players above it, the top quarter by engagement. Their 7-day retention is 60.58% for gate_30 against 52.46% for gate_40, a gap of about 8 points. That is much larger than the overall gap, which makes sense: heavy players are the ones most likely to hit the gate. This group is smaller, so the gap is still uncertain.

high_engagement = ab_test[ab_test['sum_gamerounds'] > ab_test['sum_gamerounds'].quantile(0.75)]
high_engagement_retention = high_engagement.groupby('version')['retention_7'].mean().round(4) * 100
high_engagement_retention
version
gate_30    60.58
gate_40    52.46
Name: retention_7, dtype: float64

Part 24: Real Low-Engagement Segment#

The opposite segment shows whether the effect holds for lightly engaged players. .quantile(0.25) finds the bottom quarter boundary, and <= keeps players at or below it. Their 7-day retention is tiny in both versions: 1.63% for gate_30 and 1.52% for gate_40. These players almost never reach the gate, so the version makes no practical difference to them. Together with the previous step, this suggests any real effect is concentrated among engaged players.

low_engagement = ab_test[ab_test['sum_gamerounds'] <= ab_test['sum_gamerounds'].quantile(0.25)]
low_engagement_retention = low_engagement.groupby('version')['retention_7'].mean().round(4) * 100
low_engagement_retention
version
gate_30    1.63
gate_40    1.52
Name: retention_7, dtype: float64

Part 25: Real Summary Table#

A compact summary table is what usually goes into a report or slide. pd.DataFrame is built from a dictionary of the series already calculated, so the versions line up as rows automatically because the series share the same index. The table shows gate_30 slightly ahead on both retention measures and on average rounds (54.83 against 44.70), while both versions have the same median of 15 rounds. Showing mean and median side by side makes the skew obvious to readers.

summary_table = pd.DataFrame({'retention_1_pct': retention_1_by_group, 'retention_7_pct': retention_7_by_group, 'avg_gamerounds': gamerounds_by_group, 'median_gamerounds': gamerounds_median_by_group}).round(2)
summary_table
retention_1_pct retention_7_pct avg_gamerounds median_gamerounds
version
gate_30 43.44 19.18 54.83 15.0
gate_40 43.00 17.71 44.70 15.0

Part 26: Saving the Real Summary Table#

Saving the summary lets you share it or reload it later without re-running the analysis. This time to_csv keeps the index, because the version names are stored there, and read_csv(..., index_col=0) restores that first column as the index. .equals then checks whether the reloaded table matches the original exactly, values and labels. The output is True. Forgetting index_col=0 is a common mistake that would make the tables differ.

summary_table.to_csv('ab_test_summary.csv')
reloaded_summary = pd.read_csv('ab_test_summary.csv', index_col=0)
reloaded_summary.equals(summary_table)
True

Part 27: Real Sanity Check, Percentages in Range#

Retention percentages must fall between 0 and 100; a value outside that range would mean a bug, such as multiplying by 100 twice. The code builds a True/False series for each limit, combines them with &, and all() checks every group passes. Joining the two retention checks with and gives one overall answer. The output is True, so every percentage is valid. Small guards like this are cheap insurance before numbers reach a report.

all((0 <= retention_1_by_group) & (retention_1_by_group <= 100)) and all((0 <= retention_7_by_group) & (retention_7_by_group <= 100))
True

Part 28: Real Sanity Check, Group Sizes Sum Correctly#

Every player should belong to exactly one group. If the two group counts do not add up to the total, some rows have a missing or unexpected version value. The code adds the two counts and compares them with len(ab_test). The output is True, so the 1,121 and 1,135 players account for all 2,256 rows. In messier data, this check often catches typos such as Gate_30 creating a third group.

(n_gate_30 + n_gate_40) == len(ab_test)
np.True_

Part 29: Real Business Framing#

Executives respond to players and revenue more than to percentages, so the effect is translated into a business figure. The code assumes 1,000,000 weekly active players, a made-up round number for illustration, and multiplies by the 1.47-point gap. The output is 14,700 extra players retained after a week. Treat this carefully: the confidence interval included zero, so the true effect could be smaller or even reversed. Present it as a possible upside, not a forecast.

weekly_active_estimate = 1_000_000
extra_retained_players = round(weekly_active_estimate * (retention_7_gap_pct_points / 100))
extra_retained_players
14700

Part 30: Real Recap Print#

The recap sentence pulls the key numbers together, built from variables so it stays correct if the data changes. The output reports 2,256 players, 19.18% seven-day retention for gate_30 against 17.71% for gate_40, a 1.47-point gap, and a chi-square p-value of 0.3976. Note the phrase "in this sample": it is an honest reminder that the gap was not statistically significant, and wording like that keeps readers from overreading results.

print(f'Across {len(ab_test)} real players, gate_30 held {retention_7_by_group["gate_30"]}% seven-day retention versus {retention_7_by_group["gate_40"]}% for gate_40, a real {retention_7_gap_pct_points} point gap with a chi-square p-value of {round(p_value_7,4)} in this sample.')
Across 2256 real players, gate_30 held 19.18% seven-day retention versus 17.71% for gate_40, a real 1.47 point gap with a chi-square p-value of 0.3976 in this sample.

Wrap-Up: What You Learned#

  • A real randomized controlled experiment, not just an observational comparison, is what makes an A/B test genuinely trustworthy.
  • Chi-square tests check whether a real categorical outcome gap could plausibly be explained by real random chance alone.
  • Real skewed engagement data, like game rounds played, needs medians and non-parametric tests, not just a real raw average.
  • A real percentage-point gap and a real relative lift tell genuinely different stories about the same real effect, both matter for real decisions.
  • Segmenting a real A/B test by engagement level can reveal whether a real effect is consistent or concentrated in one real group.
  • Next video: real customer funnel and conversion rate analysis, tracking how real users drop off at each real stage.

Practice on your own

  • Repeat the confidence interval calculation for 1-day retention using retention_1_by_group and the same group sizes, and check whether that interval also includes zero.
  • Remove players with zero game rounds, then re-run the 7-day retention comparison and chi-square test. Note whether the gap or the p-value changes.
  • Split players into four engagement bands with pd.qcut(ab_test['sum_gamerounds'], 4, duplicates='drop') and compare 7-day retention by version within each band.

The next video moves on to customer funnel and conversion rate analysis, tracking how users drop off at each stage. Watch the video walkthrough to see the reasoning behind each test explained live.

🧠 Test yourself

Pick an answer to check it. Your best score is saved in this browser.

  1. Why does the lesson use a chi-square test on pd.crosstab(version, retention_7)?

  2. A chi-square test on seven-day retention gives p = 0.07 with alpha = 0.05. What is the honest conclusion?

  3. Why does the lesson look at medians and a Mann-Whitney test for sum_gamerounds?

  4. gate_30 has 25% seven-day retention and gate_40 has 20%. How does the lesson describe the difference?

  5. What does this return?

    pd.crosstab(ab_test['version'], ab_test['retention_7'])
    

💻 Practice exercises

Try each one in the notebook or in your own editor before opening the solution.

  1. Create a 16-row DataFrame of A/B test players with columns version ('gate_30' or 'gate_40'), retention_1 and retention_7 (booleans) and sum_gamerounds. Print group sizes, group shares in %, and the 1-day and 7-day retention rate (%) per version.

    💡 Hint

    The mean of a boolean column is the retention rate.

    ✅ Show solution
    import pandas as pd
    
    ab_test = pd.DataFrame({
        'version': ['gate_30']*8 + ['gate_40']*8,
        'retention_1': [True, True, False, True, False, True, False, True, True, False, False, True, False, False, True, False],
        'retention_7': [True, False, False, True, False, False, False, True, False, False, False, True, False, False, False, False],
        'sum_gamerounds': [120, 45, 3, 200, 0, 31, 5, 88, 60, 2, 0, 95, 4, 1, 40, 0],
    })
    counts = ab_test['version'].value_counts()
    print(counts)
    print((counts / len(ab_test) * 100).round(2))
    print((ab_test.groupby('version')['retention_1'].mean() * 100).round(1))
    print((ab_test.groupby('version')['retention_7'].mean() * 100).round(1))
    
  2. Build an A/B dataset of 200 players (100 per group) where 20 of gate_30 and 35 of gate_40 were retained after 7 days. Run scipy.stats.chi2_contingency on the crosstab and print the p-value and whether it is significant at alpha = 0.05.

    💡 Hint

    Create the retention list with [True]*20 + [False]*80 for each group.

    ✅ Show solution
    import pandas as pd
    from scipy import stats
    
    ab_test = pd.DataFrame({
        'version': ['gate_30'] * 100 + ['gate_40'] * 100,
        'retention_7': [True] * 20 + [False] * 80 + [True] * 35 + [False] * 65,
    })
    contingency = pd.crosstab(ab_test['version'], ab_test['retention_7'])
    print(contingency)
    chi2, p_value, dof, expected = stats.chi2_contingency(contingency)
    alpha = 0.05
    print('p-value:', round(p_value, 4))
    print('Significant at 0.05:', p_value < alpha)
    
  3. Given 7-day retention of 44.2% for gate_30 (n = 4400) and 43.3% for gate_40 (n = 4600), compute the percentage-point gap, the relative lift, and a 95% confidence interval for the gap using the normal approximation. Then, for two small skewed lists of game rounds, print medians and the Mann-Whitney p-value.

    💡 Hint

    Standard error = sqrt(p1(1-p1)/n1 + p2(1-p2)/n2); CI = gap +/- 1.96 * SE.

    ✅ Show solution
    import numpy as np
    from scipy import stats
    
    p1, n1 = 0.442, 4400
    p2, n2 = 0.433, 4600
    gap_pp = round((p1 - p2) * 100, 2)
    relative_lift = round((p1 / p2 - 1) * 100, 2)
    se = np.sqrt(p1 * (1 - p1) / n1 + p2 * (1 - p2) / n2)
    ci_low = round((p1 - p2 - 1.96 * se) * 100, 2)
    ci_high = round((p1 - p2 + 1.96 * se) * 100, 2)
    print('Gap (percentage points):', gap_pp)
    print('Relative lift (%):', relative_lift)
    print('95% CI for the gap:', (ci_low, ci_high))
    
    rounds_30 = [0, 1, 3, 5, 8, 12, 20, 35, 60, 500]
    rounds_40 = [0, 0, 2, 4, 7, 10, 18, 30, 55, 2000]
    print('Medians:', np.median(rounds_30), np.median(rounds_40))
    stat, p = stats.mannwhitneyu(rounds_30, rounds_40, alternative='two-sided')
    print('Mann-Whitney p-value:', round(p, 4))
    

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.