Lesson 34 · Real-World Data Analytics
Panel Data in Pandas: Life Expectancy Trends 1952–2007 | Python Data Analytics #34
A real, classic dataset tracking real life expectancy, population, and income across real countries and decades. Let's get into it. This lesson is for…
- CourseReal-World Data Analytics
- Lesson34 of 100
- Video16 min
- FormatJupyter notebook · 30 code cells
- Data1 dataset
What you'll learn
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
📓 Full notebook
Download .ipynbData Analytics 100, Video 34: Real Global Life Expectancy Analysis#
- Video thirty-four of the hundred-video real-world data analytics series.
- A real, classic dataset tracking real life expectancy, population, and income across real countries and decades.
- Let's get into it.
This lesson is for learners who are comfortable summarising a single table and want to work with data that changes over time. The question is a big one: how much did life expectancy improve around the world between 1952 and 2007, which countries gained most and least, and how closely is a country's income linked to how long its people live?
The data is a classic Gapminder-style dataset loaded from gapminder_life_expectancy_raw.csv. It has 1,380 rows: 115 countries, each observed at 12 five-year snapshots from 1952 to 2007. The columns that matter are country, continent, year, lifeExp (life expectancy at birth, in years), pop (population) and gdpPercap (GDP per capita, a measure of average income per person). It covers 115 countries, not every country in the world, and the lesson is upfront about that.
In this lesson you will:
- compare averages across years and continents
- reshape long data into a wide table with
pivot - calculate a population-weighted average and see why it differs
- find countries that beat or miss what their income would predict
Prerequisites: you should be comfortable with pandas groupby and boolean filtering; earlier lessons in the series cover both.
Part 1: A Real Global Health Dataset#
As always, start by checking the size of the data. The code imports pandas and matplotlib, reads gapminder_life_expectancy_raw.csv with pd.read_csv, and checks .shape. The output is (1380, 6): 1,380 rows and 6 columns. In this dataset each row is one country in one year, not one country, so the row count alone does not tell you how many countries you have. That is what the next step works out.
import pandas as pd
import matplotlib.pyplot as plt
life = pd.read_csv('gapminder_life_expectancy_raw.csv')
life.shape
Part 2: Real Data Preview#
With data that repeats each country over several years, you need to know how many distinct countries and years there are. The code runs head(5) and then nunique() on country and year; only the last expression is displayed, so the output is the pair of counts. It shows 115 countries and 12 years. Since 115 multiplied by 12 is 1,380, every country appears in every year, which means the panel is complete and year-to-year comparisons are fair.
life.head(5)
life['country'].nunique(), life['year'].nunique()
Part 3: Real Coverage, Honestly Stated#
Before making any global claims, you should be honest about what the data covers. value_counts() on continent counts rows per continent. The output shows 504 rows for Africa, 312 for Asia, 288 for Europe, 252 for the Americas and only 24 for Oceania. Dividing by 12 years gives 42 African countries but just 2 in Oceania. Any Oceania average therefore rests on two countries, so treat it with caution.
life['continent'].value_counts()
Part 4: Real Life Expectancy Statistics#
A quick summary of lifeExp tells you the range you are working with across all countries and years. describe() returns count, mean, standard deviation, minimum, quartiles and maximum, rounded to two decimals. The output shows life expectancy ranging from 23.60 to 82.60 years, with a mean of 59.24 and a median of 60.40. Remember that this mixes 1952 and 2007 values together, so it describes the whole period rather than any single year.
life['lifeExp'].describe().round(2)
Part 5: Real Global Change, 1952 vs 2007#
Comparing the first and last years gives you the headline change. The code filters rows for 1952 and for 2007 with a boolean condition, then averages lifeExp for each. The output shows an average of 48.62 years in 1952 and 67.13 in 2007, a rise of about 18.5 years. Note that this is a simple average across countries, so a small country counts as much as a large one. A later step shows how weighting by population changes the picture.
life_1952 = round(life[life['year'] == 1952]['lifeExp'].mean(), 2)
life_2007 = round(life[life['year'] == 2007]['lifeExp'].mean(), 2)
life_1952, life_2007
Part 6: Visualizing the Real Global Trend#
A line chart shows whether improvement was steady or came in bursts. The code groups by year, averages lifeExp, and plots the result with .plot(kind='line', marker='o'), where the markers highlight each five-year snapshot. The figure is saved and closed, so there is no inline output. In the image, look for a line that climbs at every snapshot from about 48.6 to 67.1, and check whether the slope flattens in any period.
yearly_avg = life.groupby('year')['lifeExp'].mean().round(2)
plt.figure(figsize=(8, 5))
yearly_avg.plot(kind='line', marker='o', color='steelblue')
plt.xlabel('Real Year')
plt.ylabel('Real Average Life Expectancy')
plt.title('Real Global Average Life Expectancy Over Time')
plt.tight_layout()
plt.savefig('global_lifeexp_trend.png', dpi=120)
plt.close()
Part 7: Real Life Expectancy by Continent, 2007#
Global averages hide large regional differences. The code filters to 2007, groups by continent, averages lifeExp, rounds to one decimal and sorts from highest to lowest with sort_values(ascending=False). The output shows Oceania at 80.7, Europe at 77.4, the Americas at 73.4, Asia at 70.3 and Africa at 55.5. Africa sits about 15 years below Asia, a far bigger gap than between any other pair. Remember the Oceania figure covers only two countries.
continent_2007 = life[life['year'] == 2007].groupby('continent')['lifeExp'].mean().round(1).sort_values(ascending=False)
continent_2007
Part 8: Visualizing Real Life Expectancy by Continent#
A bar chart makes the continent gap easy to present. The code plots the sorted 2007 averages with .plot(kind='bar') and uses plt.xticks(rotation=30) to tilt the labels so they do not overlap. The figure is saved and closed, so there is no inline output. In the image, look for four bars of fairly similar height and one much shorter bar for Africa. Sorting before plotting, as here, makes rankings much easier to read.
plt.figure(figsize=(7, 5))
continent_2007.plot(kind='bar', color='darkorange')
plt.ylabel('Real Average Life Expectancy')
plt.title('Real Life Expectancy by Continent, 2007')
plt.xticks(rotation=30)
plt.tight_layout()
plt.savefig('lifeexp_by_continent.png', dpi=120)
plt.close()
Part 9: Real Income and Life Expectancy#
A key question in global health is whether richer countries live longer. The code filters to 2007 and uses Series.corr() to compute the Pearson correlation between gdpPercap and lifeExp. Correlation runs from -1 to 1, where values near 1 mean the two rise together. The output is 0.684, a fairly strong positive link. It does not prove that income causes longer lives, as other factors such as healthcare and education also rise with income.
gdp_lifeexp_corr = round(life[life['year'] == 2007]['gdpPercap'].corr(life[life['year'] == 2007]['lifeExp']), 3)
gdp_lifeexp_corr
Part 10: Visualizing Real Income vs Life Expectancy#
A scatter plot shows the shape of the income relationship, which a single correlation number cannot. The code saves the 2007 rows as data_2007 for reuse, then plots gdpPercap against lifeExp with alpha=0.6 so overlapping points stay visible. The figure is saved and closed, so there is no inline output. In the image, look for a steep rise at low incomes that flattens out at higher incomes. That curve means extra income matters most for the poorest countries.
data_2007 = life[life['year'] == 2007]
plt.figure(figsize=(8, 6))
plt.scatter(data_2007['gdpPercap'], data_2007['lifeExp'], alpha=0.6, color='seagreen')
plt.xlabel('Real GDP Per Capita ($)')
plt.ylabel('Real Life Expectancy')
plt.title('Real Income vs Life Expectancy, 2007')
plt.tight_layout()
plt.savefig('gdp_vs_lifeexp.png', dpi=120)
plt.close()
Part 11: Real Top Five Countries by Life Expectancy#
Looking at the leaders makes the averages concrete. nlargest(5, 'lifeExp') returns the five rows with the highest life expectancy, and the column list keeps country, continent and lifeExp. The output shows Japan at 82.603, Hong Kong, China at 82.208, Iceland at 81.757, Australia at 81.235 and Israel at 80.745. Three of the five are in Asia, which shows that the Asia continental average of 70.3 hides a very wide range.
top_five_2007 = data_2007.nlargest(5, 'lifeExp')[['country', 'continent', 'lifeExp']]
top_five_2007
Part 12: Real Bottom Five Countries by Life Expectancy#
The countries at the bottom are just as important to name. nsmallest(5, 'lifeExp') returns the five lowest 2007 values. The output shows Mozambique at 42.082, Sierra Leone at 42.568, Lesotho at 42.592, Angola at 42.731 and Afghanistan at 43.828. Four of the five are in Africa. The gap between Japan at the top and Mozambique at the bottom is about 40 years, which is a powerful figure for any report.
bottom_five_2007 = data_2007.nsmallest(5, 'lifeExp')[['country', 'continent', 'lifeExp']]
bottom_five_2007
Part 13: Real Reshaping for Year-Over-Year Comparison#
Data stored with one row per country per year is called long format. To compare years side by side, you reshape it to wide format, with one row per country and one column per year. pivot(index='country', columns='year', values='lifeExp') does this. The output shows the first five countries with their 1952 and 2007 values, for example Afghanistan rising from 28.801 to 43.828. pivot fails if any country-year pair appears twice, so it doubles as a duplicate check.
life_pivot = life.pivot(index='country', columns='year', values='lifeExp')
life_pivot[[1952, 2007]].head(5)
Part 14: Real Biggest Gains, 1952 to 2007#
With years as columns, measuring each country's improvement is a single subtraction. The code subtracts the 1952 column from the 2007 column and sorts the gains from largest to smallest. head(5) shows the top five, rounded to one decimal. Oman gained the most at 38.1 years, followed by Indonesia at 33.2, Saudi Arabia at 32.9, and Libya and Korea, Rep. at 31.2 each. Several of these are countries that grew rapidly richer over the period.
life_gain = (life_pivot[2007] - life_pivot[1952]).sort_values(ascending=False)
life_gain.head(5).round(1)
Part 15: Real Smallest Gains and Real Declines#
The other end of the ranking shows where progress stalled. Because the gains are already sorted from largest to smallest, tail(5) returns the five smallest. The output shows Congo, Dem. Rep. at 7.3 years, Liberia at 7.2, Rwanda at 6.2, Botswana at 3.1 and Lesotho at just 0.5. No country shows a negative value, so none ended lower than it started, though Lesotho barely moved in 55 years. Something held these countries back, which is worth investigating further.
life_gain.tail(5).round(1)
Part 16: Visualizing Real Biggest Gains#
A horizontal bar chart is a good choice when the labels are country names, because long names fit along the side. The code plots the top five gains with kind='barh', then calls plt.gca().invert_yaxis() so the largest gain appears at the top rather than the bottom. The figure is saved and closed, so there is no inline output. In the image, look for Oman's bar clearly longer than the rest.
plt.figure(figsize=(8, 5))
life_gain.head(5).plot(kind='barh', color='seagreen')
plt.xlabel('Real Life Expectancy Gain, 1952-2007')
plt.title('Real Biggest Life Expectancy Gains')
plt.gca().invert_yaxis()
plt.tight_layout()
plt.savefig('biggest_lifeexp_gains.png', dpi=120)
plt.close()
Part 17: Real Africa Trend Deep Dive#
Since Africa has the lowest life expectancy, it deserves a closer look over time. The code filters to African rows, groups by year and averages lifeExp. The output rises steadily by about 2 years per snapshot from 38.62 in 1952 to 51.04 in 1982, then nearly stalls: 53.37 in 1992, 53.88 in 1997 and 53.96 in 2002, before reaching 55.52 in 2007. A flat stretch like this in a time series is a signal to look for a cause.
africa_trend = life[life['continent'] == 'Africa'].groupby('year')['lifeExp'].mean().round(2)
africa_trend
Part 18: Real Population-Weighted vs Simple Average#
A simple average treats every country equally, so a small country counts as much as a very large one. A population-weighted average gives each country weight in proportion to its population, which better reflects the average person. The code multiplies lifeExp by pop, sums it, and divides by total population. The output shows a simple 2007 average of 67.13 and a weighted average of 68.53. The weighted figure is higher, which suggests large-population countries have above-average life expectancy.
simple_avg_2007 = round(data_2007['lifeExp'].mean(), 2)
weighted_avg_2007 = round((data_2007['lifeExp'] * data_2007['pop']).sum() / data_2007['pop'].sum(), 2)
simple_avg_2007, weighted_avg_2007
Part 19: Real Fastest-Growing Populations#
Population change is another lens on how countries developed. The code pivots pop into a wide table in the same way as before, then divides 2007 by 1952 to get a growth multiple, and sorts it. The output shows Kuwait's population grew 15.7 times, followed by Jordan at 10.0, Djibouti at 7.9, Saudi Arabia at 6.9 and Oman at 6.3. A ratio like this is easier to compare across countries of very different sizes than a raw change in headcount.
pop_pivot = life.pivot(index='country', columns='year', values='pop')
pop_growth = (pop_pivot[2007] / pop_pivot[1952]).sort_values(ascending=False)
pop_growth.head(5).round(1)
Part 20: Real Correlation Matrix#
A correlation matrix shows how every pair of numeric columns relates. The code computes corr(numeric_only=True) for lifeExp, gdpPercap, pop and year. The output shows lifeExp correlates 0.561 with income and 0.444 with year, but only 0.048 with population. The income correlation here is lower than the 0.684 for 2007 alone, because this matrix mixes all years together. Population size has almost no link to life expectancy.
numeric_corr = life[['lifeExp', 'gdpPercap', 'pop', 'year']].corr(numeric_only=True)
numeric_corr.round(3)
Part 21: Real Simple Income Threshold Rule#
A simple rule tests how useful income is as a predictor on its own. The code flags 2007 countries with GDP per capita above 10,000 dollars as predicted high, and separately flags countries with life expectancy above 70 as actually high, using .astype(int) for both. It then counts matches with &. The output shows 41 true positives (rich and long-lived) and 3 false positives (rich but not above 70). So almost every rich country also has high life expectancy.
predicted_high = (data_2007['gdpPercap'] > 10000).astype(int)
actual_high = (data_2007['lifeExp'] > 70).astype(int)
true_positive = ((predicted_high == 1) & (actual_high == 1)).sum()
false_positive = ((predicted_high == 1) & (actual_high == 0)).sum()
true_positive, false_positive
Part 22: Real Precision of the Threshold Rule#
Precision measures how often the rule is right when it predicts high life expectancy. It is calculated as true positives divided by all positive predictions, here 41 out of 44. The output is 0.932, so about 93% of countries with income above 10,000 dollars also live past 70 on average. Precision alone does not tell you how many long-lived countries the rule missed; that would need recall, which counts false negatives as well.
threshold_precision = round(true_positive / (true_positive + false_positive), 3)
threshold_precision
Part 23: Real Countries Beating Their Income Bracket#
Countries that live longer than their income would suggest are often the most interesting to study. The code filters 2007 data for GDP per capita below 5,000 dollars and life expectancy above 70, combining the two conditions with &. The output lists eight countries, including China at 72.961, Nicaragua at 72.899 with an income of about 2,749 dollars, and Jordan at 72.535. These countries achieved high life expectancy on modest incomes, which suggests factors beyond wealth are at work.
overachievers = data_2007[(data_2007['gdpPercap'] < 5000) & (data_2007['lifeExp'] > 70)][['country', 'gdpPercap', 'lifeExp']]
overachievers
Part 24: Real Countries Underperforming Their Income Bracket#
The reverse case, rich countries with low life expectancy, is equally revealing. The code filters for GDP per capita above 10,000 dollars and life expectancy below 65. The output lists three countries: Botswana at 50.728, Equatorial Guinea at 51.579 and Gabon at 56.735, all with incomes above 12,000 dollars. These are three of the false positives from the threshold rule. Income on its own can hide how widely that wealth is shared.
underachievers = data_2007[(data_2007['gdpPercap'] > 10000) & (data_2007['lifeExp'] < 65)][['country', 'gdpPercap', 'lifeExp']]
underachievers
Part 25: Real Decade-by-Decade Africa vs Europe Gap#
Tracking a gap over time shows whether it is closing. The code filters to Africa and Europe with isin, groups by both year and continent, and uses unstack() to turn continents into columns. It then adds a gap column as Europe minus Africa. The output shows the gap at 25.9 years in 1952, narrowing to 20.8 in 1987, then widening again to 22.4 in 2002 and ending at 21.9 in 2007. Progress in closing the gap stalled from the late 1980s.
africa_vs_europe = life[life['continent'].isin(['Africa', 'Europe'])].groupby(['year', 'continent'])['lifeExp'].mean().unstack().round(1)
africa_vs_europe['gap'] = (africa_vs_europe['Europe'] - africa_vs_europe['Africa']).round(1)
africa_vs_europe
Part 26: Saving the Real 2007 Snapshot#
Saving a single-year snapshot gives you a small, tidy file that is easy to share or use in a dashboard. to_csv writes the 2007 rows and index=False leaves out the row index. The code then reloads the file and compares .shape. The output is True, so the saved file matches the snapshot exactly. Saving focused subsets like this is good practice when others only need one slice of a larger dataset.
data_2007.to_csv('life_expectancy_2007_snapshot.csv', index=False)
reloaded = pd.read_csv('life_expectancy_2007_snapshot.csv')
reloaded.shape == data_2007.shape
Part 27: Real Sanity Check, Global Gain#
This sanity check protects the headline finding. The code subtracts the 1952 average from the 2007 average, rounds it, and checks that the gain is greater than 15 years. The output is True. Using a threshold rather than an exact value means the check still passes if minor data corrections shift the figure slightly, while still catching a serious error such as filtering the wrong year.
round(life_2007 - life_1952, 1) > 15
Part 28: Real Sanity Check, Row Count Preserved#
This check confirms the dataset is still a complete grid of countries and years. It compares len(life) with 115 multiplied by 12, and the output is True: 1,380 rows, one for each country in each year. If a row were lost or duplicated, this check would fail, and pivots and year-over-year comparisons could quietly give wrong answers.
len(life) == 115 * 12
Part 29: Real Headline Finding#
A headline finding should be short and backed by the numbers. The code prints an f-string that inserts the 1952 and 2007 averages calculated earlier. The output reads: global average life expectancy rose from 48.62 to 67.13 years between 1952 and 2007, though the gap between Africa and Europe remained wide throughout. Pairing good news with its main caveat in one sentence gives a more honest summary than either alone.
print(f'Global average life expectancy rose from {life_1952} to {life_2007} years between 1952 and 2007, though the real gap between Africa and Europe remained wide throughout.')
Part 30: Real Recap Print#
The recap sentence states the scope of the analysis and its main conclusion. The code uses nunique() inside an f-string to report the number of countries and snapshots, so the sentence stays accurate if the data changes. The output reads: across 115 countries and 12 five-year snapshots, income explained a meaningful share of the life expectancy gap, but plenty of countries beat or missed what their income alone would predict. Stating coverage up front is a mark of honest analysis.
print(f'Across {life["country"].nunique()} real countries and {life["year"].nunique()} real five-year snapshots, income explained a meaningful share of the real life expectancy gap, but plenty of real countries beat or missed what their income alone would predict.')
Wrap-Up: What You Learned#
- Being real upfront about a real dataset's actual coverage, one hundred fifteen countries rather than the full historical set, matters more than pretending it's complete.
- Real global average life expectancy rose by nearly two decades between 1952 and 2007, one of the real great achievements of that half-century.
- Real income correlated with life expectancy, but plenty of individual real countries beat or missed what income alone would predict.
- Reshaping real long data into a real wide pivot table made a direct real year-over-year comparison dramatically easier.
- Real population-weighting a global average can shift the real headline number, always check whether your average should be weighted.
- Next video: a real chronic kidney disease risk factor analysis, using a real classic clinical dataset from a hospital in Tamil Nadu, India.
Practice on your own
- Repeat the decade-by-decade gap table for Asia versus the Americas, using
isin,groupby(['year', 'continent'])andunstack(), and check whether that gap closed faster. - Calculate the population-weighted average life expectancy for 1952 and compare it with the simple 1952 average of 48.62.
- Change the overachiever thresholds to income below 3,000 dollars and life expectancy above 65, and see which countries join or leave the list.
The next video moves on to a chronic kidney disease risk factor analysis, using a classic clinical dataset from a hospital in Tamil Nadu, India.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.



