Lesson 15 · Real-World Data Analytics

Python Data Analytics #15: Correlation & Diversification Across Real Assets

Video fifteen of the hundred-video real-world data analytics series. Bringing in three more real stocks and a real market index to actually test what…

⬇ Download notebook ⬇ Notebook + data (.zip)Open in Colab ↗

CourseReal-World Data Analytics
Lesson15 of 26
Video30 min
FormatJupyter notebook · 30 code cells
Data1 dataset

pandas NumPy Matplotlib

Datasets used in this lesson

Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.

multi_stock_2007.csv55.7 KB

📓 Full notebook

Download .ipynb

Data Analytics 100, Video 15: Correlation and Diversification Across Real Assets#

Video fifteen of the hundred-video real-world data analytics series.
Bringing in three more real stocks and a real market index to actually test what diversification does to real risk.
Let's get into it.

Part 1: Beyond Just Two Assets#

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
raw = pd.read_csv('multi_stock_2007.csv', parse_dates=['Date'])
raw['stock'].unique()

array(['MSFT', 'IBM', 'SBUX', 'AAPL', 'GSPC'], dtype=object)

Part 2: Real Long-to-Wide Reshape#

wide = raw.pivot(index='Date', columns='stock', values='value')
wide = wide.sort_index()
wide.shape

(220, 5)

Part 3: Real Missing-Day Cleanup#

wide.isna().sum()
wide_clean = wide.dropna()
wide_clean.shape

(219, 5)

Part 4: Real Daily Returns for Every Asset#

returns = wide_clean.pct_change().dropna()
returns.shape
returns.head(3).round(4)

stock	AAPL	GSPC	IBM	MSFT	SBUX
Date
2007-01-04	0.0222	0.0012	0.0107	-0.0017	0.0011
2007-01-05	-0.0071	-0.0061	-0.0091	-0.0057	-0.0043
2007-01-08	0.0049	0.0022	0.0152	0.0098	-0.0037

Part 5: Real Individual Volatility Comparison#

annualized_vol = returns.std() * np.sqrt(252)
annualized_vol.round(3).sort_values(ascending=False)

stock
AAPL    0.389
MSFT    0.227
SBUX    0.222
IBM     0.199
GSPC    0.153
dtype: float64

Part 6: Real Correlation Matrix#

corr_matrix = returns.corr()
corr_matrix.round(3)

stock	AAPL	GSPC	IBM	MSFT	SBUX
stock
AAPL	1.000	0.490	0.455	0.388	0.305
GSPC	0.490	1.000	0.552	0.644	0.583
IBM	0.455	0.552	1.000	0.438	0.481
MSFT	0.388	0.644	0.438	1.000	0.469
SBUX	0.305	0.583	0.481	0.469	1.000

Part 7: Visualizing the Real Correlation Matrix#

plt.figure(figsize=(7, 6))
im = plt.imshow(corr_matrix, cmap='coolwarm', vmin=-1, vmax=1)
plt.xticks(range(len(corr_matrix.columns)), corr_matrix.columns, rotation=45)
plt.yticks(range(len(corr_matrix.columns)), corr_matrix.columns)
plt.colorbar(im, label='Real Correlation')
plt.title('Real Correlation Matrix, Five Real Assets')
plt.tight_layout()
plt.savefig('correlation_heatmap.png', dpi=120)
plt.close()

Part 8: Real Most and Real Least Correlated Pair#

corr_pairs = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(bool)).stack()
corr_pairs.idxmax(), round(corr_pairs.max(), 3)
corr_pairs.idxmin(), round(corr_pairs.min(), 3)

(('AAPL', 'SBUX'), np.float64(0.305))

Part 9: Real Market Beta for Each Stock#

market_var = returns['GSPC'].var()
betas = {}
for ticker in ['AAPL', 'MSFT', 'IBM', 'SBUX']:
    cov_with_market = returns[ticker].cov(returns['GSPC'])
    betas[ticker] = cov_with_market / market_var
pd.Series(betas).round(3).sort_values(ascending=False)

AAPL    1.243
MSFT    0.951
SBUX    0.843
IBM     0.715
dtype: float64

Part 10: Interpreting Real Beta Values#

high_beta_stock = pd.Series(betas).idxmax()
low_beta_stock = pd.Series(betas).idxmin()
high_beta_stock, low_beta_stock

('AAPL', 'IBM')

Part 11: Building a Real Equal-Weight Portfolio#

stock_cols = ['AAPL', 'MSFT', 'IBM', 'SBUX']
portfolio_returns = returns[stock_cols].mean(axis=1)
portfolio_returns.head(3).round(4)

Date
2007-01-04    0.0081
2007-01-05   -0.0065
2007-01-08    0.0066
dtype: float64

Part 12: Real Portfolio Volatility vs Real Individual Volatility#

portfolio_vol = portfolio_returns.std() * np.sqrt(252)
individual_vols = returns[stock_cols].std() * np.sqrt(252)
round(portfolio_vol, 3), individual_vols.round(3).to_dict()

(np.float64(0.196),
 {'AAPL': 0.389, 'MSFT': 0.227, 'IBM': 0.199, 'SBUX': 0.222})

Part 13: Quantifying the Real Diversification Benefit#

average_individual_vol = individual_vols.mean()
diversification_benefit_pct = (1 - portfolio_vol / average_individual_vol) * 100
round(diversification_benefit_pct, 1)

np.float64(24.5)

Part 14: Visualizing Real Portfolio vs Real Individual Growth#

cumulative_portfolio = (1 + portfolio_returns).cumprod()
cumulative_individual = (1 + returns[stock_cols]).cumprod()
plt.figure(figsize=(11, 6))
for ticker in stock_cols:
    plt.plot(cumulative_individual.index, cumulative_individual[ticker], alpha=0.5, label=f'Real {ticker}')
plt.plot(cumulative_portfolio.index, cumulative_portfolio, color='black', linewidth=2.5, label='Real Equal-Weight Portfolio')
plt.xlabel('Real Date')
plt.ylabel('Real Growth of $1 Invested')
plt.title('Real Diversified Portfolio vs Real Individual Stocks')
plt.legend()
plt.tight_layout()
plt.savefig('diversification_growth.png', dpi=120)
plt.close()

Part 15: Real Portfolio vs Real Market Index#

cumulative_market = (1 + returns['GSPC']).cumprod()
round(cumulative_portfolio.iloc[-1], 3), round(cumulative_market.iloc[-1], 3)

(np.float64(1.171), np.float64(1.045))

Part 16: Real Sharpe Ratio Comparison#

portfolio_sharpe = (portfolio_returns.mean() * 252) / portfolio_vol
market_sharpe = (returns['GSPC'].mean() * 252) / (returns['GSPC'].std() * np.sqrt(252))
round(portfolio_sharpe, 3), round(market_sharpe, 3)

(np.float64(1.033), np.float64(0.412))

Part 17: Real Rolling Correlation with the Market#

rolling_corr_aapl = returns['AAPL'].rolling(30).corr(returns['GSPC'])
rolling_corr_ibm = returns['IBM'].rolling(30).corr(returns['GSPC'])
rolling_corr_aapl.dropna().describe().round(3)

count    189.000
mean       0.419
std        0.221
min        0.040
25%        0.211
50%        0.415
75%        0.617
max        0.785
dtype: float64

Part 18: Visualizing Real Rolling Market Correlation#

plt.figure(figsize=(11, 5))
plt.plot(rolling_corr_aapl.index, rolling_corr_aapl, label='Real Apple vs Market')
plt.plot(rolling_corr_ibm.index, rolling_corr_ibm, label='Real IBM vs Market')
plt.axhline(0, color='gray', linestyle='--')
plt.xlabel('Real Date')
plt.ylabel('Real 30-Day Rolling Correlation')
plt.title('Real Rolling Correlation with the Market Index')
plt.legend()
plt.tight_layout()
plt.savefig('rolling_market_correlation.png', dpi=120)
plt.close()

Part 19: Real Weighted Portfolio Variance, More Than Two Assets#

weights = np.array([0.25, 0.25, 0.25, 0.25])
cov_matrix = returns[stock_cols].cov() * 252
portfolio_variance = weights @ cov_matrix.values @ weights
round(np.sqrt(portfolio_variance), 3) == round(portfolio_vol, 3)

np.True_

Part 20: Real Minimum-Correlation Two-Stock Pair#

stock_corr = returns[stock_cols].corr()
stock_corr_pairs = stock_corr.where(np.triu(np.ones(stock_corr.shape), k=1).astype(bool)).stack()
best_pair = stock_corr_pairs.idxmin()
best_pair, round(stock_corr_pairs.min(), 3)

(('AAPL', 'SBUX'), np.float64(0.305))

Part 21: Real Two-Stock Minimum-Correlation Portfolio#

pair_returns = returns[list(best_pair)].mean(axis=1)
pair_vol = pair_returns.std() * np.sqrt(252)
round(pair_vol, 3)

np.float64(0.252)

Part 22: Real Number of Assets vs Real Risk Reduction#

one_stock_vol = individual_vols.mean()
two_stock_vol = pair_vol
four_stock_vol = portfolio_vol
one_stock_vol, two_stock_vol, four_stock_vol

(np.float64(0.2590001111751411),
 np.float64(0.25156577751854164),
 np.float64(0.1955076567662535))

Part 23: Saving the Real Multi-Asset Summary Table#

summary = pd.DataFrame({'AnnualizedVol': annualized_vol, 'Beta': pd.Series(betas).reindex(annualized_vol.index)})
summary.round(3).to_csv('multi_asset_summary.csv')
reloaded = pd.read_csv('multi_asset_summary.csv', index_col=0)
reloaded.shape[0] == summary.shape[0]

True

Part 24: Real Correlation Matrix Export#

corr_matrix.round(4).to_csv('correlation_matrix.csv')
reloaded_corr = pd.read_csv('correlation_matrix.csv', index_col=0)
reloaded_corr.shape == corr_matrix.shape

True

Part 25: Real Sanity Check on Correlation Bounds#

(corr_matrix.values >= -1.0001).all() and (corr_matrix.values <= 1.0001).all()
np.allclose(np.diag(corr_matrix.values), 1.0)

True

Part 26: Real Recap Print#

print(f'Combining {len(stock_cols)} real stocks cut annualized volatility from {round(average_individual_vol*100,1)}% average to {round(portfolio_vol*100,1)}%, a real {round(diversification_benefit_pct,1)}% risk reduction.')

Combining 4 real stocks cut annualized volatility from 25.9% average to 19.6%, a real 24.5% risk reduction.

Part 27: Real Portfolio Drawdown vs Real Market Drawdown#

def max_drawdown(cumulative):
    running_peak = cumulative.cummax()
    drawdown = (cumulative - running_peak) / running_peak
    return drawdown.min()
round(max_drawdown(cumulative_portfolio) * 100, 2), round(max_drawdown(cumulative_market) * 100, 2)

(np.float64(-13.46), np.float64(-9.43))

Part 28: Real Skewness and Kurtosis of Returns#

returns[stock_cols].skew().round(3)
returns[stock_cols].kurtosis().round(3)

stock
AAPL    2.184
MSFT    8.262
IBM     2.988
SBUX    0.615
dtype: float64

Part 29: Real Days Each Stock Beat the Market#

beat_market = {ticker: (returns[ticker] > returns['GSPC']).sum() for ticker in stock_cols}
pd.Series(beat_market).sort_values(ascending=False)

AAPL    119
IBM     117
MSFT    108
SBUX     75
dtype: int64

Part 30: Real Best and Worst Single Day for the Portfolio#

port_best_day = portfolio_returns.idxmax()
port_worst_day = portfolio_returns.idxmin()
port_best_pct = round(portfolio_returns.max() * 100, 2)
port_worst_pct = round(portfolio_returns.min() * 100, 2)
port_best_day, port_best_pct, port_worst_day, port_worst_pct

(Timestamp('2007-11-13 00:00:00'),
 np.float64(5.39),
 Timestamp('2007-11-09 00:00:00'),
 np.float64(-4.43))

Wrap-Up: What You Learned#

Correlation measures how real assets genuinely move together, and low correlation is exactly what makes diversification work.
Beta measures how much a real stock swings relative to the real broader market, a different lens on risk than volatility alone.
Combining real assets into a portfolio reduces total volatility below the real average of the individual pieces, purely from imperfect correlation.
The real matrix form of portfolio variance generalizes cleanly beyond just two assets to any real number of holdings.
More real assets is not automatically better, the real correlation between them matters just as much as the real count.
Next video: turning these real indicators and real risk metrics into an actual backtested real trading strategy.

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.

☕ Buy me a coffee Other ways to help