Lesson 15 · Statistics for data analysts

Statistics Capstone: A Full Real-World Analysis | Statistics #15

The final video of the 15-part series: one complete real statistical analysis, start to finish, combining nearly everything covered so far. Real Sample…

⬇ Download notebook ⬇ Notebook + data (.zip)Open in Colab ↗

CourseStatistics for data analysts
Lesson15 of 15
Video15 min
FormatJupyter notebook · 11 code cells
Data1 dataset

statsmodels pandas NumPy SciPy Matplotlib

Datasets used in this lesson

Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.

superstore_full.csv1.2 MB

📓 Full notebook

Download .ipynb

Statistics for Data Analysts, Video 15: Capstone Statistical Analysis#

The final video of the 15-part series: one complete real statistical analysis, start to finish, combining nearly everything covered so far.
Real Sample Superstore order data, including real Discount figures this time.
The business question: is our discounting strategy hurting real profitability, and where should we focus? Let's find out, honestly.

Before You Start#

Open a new Jupyter Notebook in VS Code and select your Python interpreter as the kernel.
You'll need pandas, NumPy, SciPy, Matplotlib, and statsmodels.
Place superstore_full.csv in the same folder as this notebook.

import pandas as pd
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
import statsmodels.formula.api as smf
import statsmodels.api as sm
from statsmodels.stats.outliers_influence import variance_inflation_factor

df = pd.read_csv('superstore_full.csv')
print(df.shape)
print(df[['Sales', 'Discount', 'Quantity', 'Profit']].describe())

(9994, 14)
              Sales     Discount     Quantity       Profit
count   9994.000000  9994.000000  9994.000000  9994.000000
mean     229.858001     0.156203     3.789574    28.656896
std      623.245101     0.206452     2.225110   234.260108
min        0.444000     0.000000     1.000000 -6599.978000
25%       17.280000     0.000000     2.000000     1.728750
50%       54.490000     0.200000     3.000000     8.666500
75%      209.940000     0.200000     5.000000    29.364000
max    22638.480000     0.800000    14.000000  8399.976000

Part 1: Framing the Business Question#

df['margin'] = df['Profit'] / df['Sales']
df['high_discount'] = (df['Discount'] >= 0.2).astype(int)
print(f'Real orders with discount >= 20%: {df["high_discount"].sum()} of {len(df)}')

Real orders with discount >= 20%: 5050 of 9994

Part 2: The Shape of Real Profit#

print(f'Real Profit skewness: {stats.skew(df["Profit"]):.2f}')
print(f'Real Profit range: {df["Profit"].min():.2f} to {df["Profit"].max():.2f}')
print(f'Real orders with a loss (Profit < 0): {(df["Profit"] < 0).sum()}')

Real Profit skewness: 7.56
Real Profit range: -6599.98 to 8399.98
Real orders with a loss (Profit < 0): 1871

plt.figure(figsize=(9, 4))
plt.hist(df['margin'].clip(-1, 1), bins=60, color='steelblue', edgecolor='white')
plt.axvline(0, color='darkred', linestyle='--', label='Break-even')
plt.title('Real Profit Margin Distribution (clipped for display)')
plt.xlabel('Profit Margin')
plt.legend()
plt.show()

No description has been provided for this image

Part 3: A Confidence Interval for Average Margin#

n = len(df)
mean_margin = df['margin'].mean()
se_margin = df['margin'].std(ddof=1) / np.sqrt(n)
t_crit = stats.t.ppf(0.975, df=n - 1)
ci_low, ci_high = mean_margin - t_crit * se_margin, mean_margin + t_crit * se_margin
print(f'Real average margin: {mean_margin:.4f}')
print(f'Real 95% CI: ({ci_low:.4f}, {ci_high:.4f})')

Real average margin: 0.1203
Real 95% CI: (0.1112, 0.1295)

Part 4: Does Discounting Actually Hurt Margin?#

high = df.loc[df['high_discount'] == 1, 'margin']
low = df.loc[df['high_discount'] == 0, 'margin']
print(f'Real high-discount orders (n={len(high)}): mean margin = {high.mean():.4f}')
print(f'Real low-discount orders (n={len(low)}): mean margin = {low.mean():.4f}')

Real high-discount orders (n=5050): mean margin = -0.0883
Real low-discount orders (n=4944): mean margin = 0.3334

u_stat, p_value = stats.mannwhitneyu(high, low, alternative='two-sided')
print(f'Real Mann-Whitney U p-value: {p_value:.2e}')

Real Mann-Whitney U p-value: 0.00e+00

Part 5: Quantifying the Effect with Regression#

model = smf.ols('Profit ~ Sales + Discount + Quantity', data=df).fit()
print(model.summary())

                            OLS Regression Results                            
==============================================================================
Dep. Variable:                 Profit   R-squared:                       0.273
Model:                            OLS   Adj. R-squared:                  0.273
Method:                 Least Squares   F-statistic:                     1249.
Date:                Sun, 16 Aug 2026   Prob (F-statistic):               0.00
Time:                        20:45:36   Log-Likelihood:                -67121.
No. Observations:                9994   AIC:                         1.342e+05
Df Residuals:                    9990   BIC:                         1.343e+05
Df Model:                           3                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
Intercept     34.9721      4.218      8.291      0.000      26.704      43.240
Sales          0.1800      0.003     54.961      0.000       0.174       0.186
Discount    -233.4570      9.686    -24.101      0.000    -252.444    -214.470
Quantity      -2.9622      0.917     -3.230      0.001      -4.760      -1.165
==============================================================================
Omnibus:                    14925.586   Durbin-Watson:                   1.996
Prob(Omnibus):                  0.000   Jarque-Bera (JB):         75963879.502
Skew:                          -8.185   Prob(JB):                         0.00
Kurtosis:                     429.796   Cond. No.                     3.26e+03
==============================================================================

Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 3.26e+03. This might indicate that there are
strong multicollinearity or other numerical problems.

X = df[['Sales', 'Discount', 'Quantity']]
X_const = sm.add_constant(X)
vif_data = pd.DataFrame({'feature': X_const.columns, 'VIF': [variance_inflation_factor(X_const.values, i) for i in range(X_const.shape[1])]})
print(vif_data)
discount_effect = model.params['Discount']
print(f'Real effect: each 10-point rise in discount predicts ${abs(discount_effect) * 0.10:.2f} less real profit per order, holding Sales and Quantity fixed')

    feature       VIF
0     const  4.453873
1     Sales  1.042986
2  Discount  1.001008
3  Quantity  1.042234
Real effect: each 10-point rise in discount predicts $23.35 less real profit per order, holding Sales and Quantity fixed

Part 6: Where Is the Discounting Concentrated?#

category_table = pd.crosstab(df['Category'], df['high_discount'])
chi2_stat, p_chi, dof, expected = stats.chi2_contingency(category_table)
cramers_v = np.sqrt(chi2_stat / (len(df) * (min(category_table.shape) - 1)))
print(f'Real chi-square p-value: {p_chi:.2e}')
print(f'Real Cramer\'s V: {cramers_v:.4f}')
print(df.groupby('Category')['high_discount'].mean().round(3))

Real chi-square p-value: 1.72e-10
Real Cramer's V: 0.0671
Category
Furniture          0.545
Office Supplies    0.478
Technology         0.548
Name: high_discount, dtype: float64

Part 7: Executive Summary#

print('EXECUTIVE SUMMARY: Discounting and Profitability')
print('=' * 55)
print(f'- Real average profit margin across all orders: {mean_margin:.1%} (95% CI: {ci_low:.1%} to {ci_high:.1%})')
print(f'- Real orders discounted 20% or more average a {high.mean():.1%} margin, i.e. a genuine loss on average')
print(f'- Real orders discounted under 20% average a {low.mean():.1%} margin, solidly profitable')
print(f'- This gap is real and statistically robust (Mann-Whitney p < 0.001), not sampling noise')
print(f'- Regression estimates each 10-point rise in discount costs about ${abs(discount_effect) * 0.10:.2f} in real profit per order, holding order size and quantity fixed')
print(f'- Heavy discounting is only weakly associated with product category (Cramer\'s V={cramers_v:.3f}), so this is a broad real pricing issue, not one confined to a specific category')
print('RECOMMENDATION: Review discount approval thresholds above 20% across the board, rather than targeting a single category')

EXECUTIVE SUMMARY: Discounting and Profitability
=======================================================
- Real average profit margin across all orders: 12.0% (95% CI: 11.1% to 12.9%)
- Real orders discounted 20% or more average a -8.8% margin, i.e. a genuine loss on average
- Real orders discounted under 20% average a 33.3% margin, solidly profitable
- This gap is real and statistically robust (Mann-Whitney p < 0.001), not sampling noise
- Regression estimates each 10-point rise in discount costs about $23.35 in real profit per order, holding order size and quantity fixed
- Heavy discounting is only weakly associated with product category (Cramer's V=0.067), so this is a broad real pricing issue, not one confined to a specific category
RECOMMENDATION: Review discount approval thresholds above 20% across the board, rather than targeting a single category

Wrap-Up: The Full 15-Part Series#

This capstone combined distribution checks, confidence intervals, non-parametric hypothesis testing, multiple regression with a VIF sanity check, and a chi-square test with an honest effect size, into one coherent real analysis with an actionable conclusion.
Across all 15 videos: probability and simulation, discrete and continuous distributions, the Central Limit Theorem, sampling and bias, confidence intervals, hypothesis testing, a full synthetic A/B test with power analysis, non-parametric tests, multiple comparisons, simple and multiple regression, categorical data analysis, and Bayesian statistics.
Every honest finding in this series, including the real misses, the real disagreements between tests, and the real cases where statistical and practical significance diverged, was reported as it actually came out, not smoothed over.
That's the whole series. Thanks for following along through all fifteen videos, real data and all. Subscribe if you haven't, and go run these notebooks yourself. See you in the next series.

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.

☕ Buy me a coffee Other ways to help