Data Analytics 100, Video 33: Real Medical Insurance Cost Analysis#

• Video thirty-three of the hundred-video real-world data analytics series.
• A real, widely used dataset of real individual medical insurance charges, hunting for exactly what drives real healthcare costs up.
• Let's get into it.

This lesson is for learners who can already group and summarise data in pandas and want to apply it to a cost question. The business question is the one every insurer and health planner asks: what actually drives individual medical insurance charges up, and which groups of people cost the most?

The data is a widely used dataset of 1,338 individual insurance beneficiaries, loaded from insurance_raw.csv. Each row is one person, with age, sex, bmi (body mass index), children (number of dependants covered), smoker (yes or no), region (four US regions) and charges, the medical costs billed in dollars.

In this lesson you will:

• compare average charges across smoking status, sex, region, BMI band and age group
• see why mean and median tell different stories for skewed cost data
• isolate a compounded high-cost group and compare it with everyone else
• pull out the most and least expensive individuals with nlargest and nsmallest

Prerequisites: you should be comfortable with pandas groupby, boolean filtering and pd.cut. The two previous lessons on diabetes and heart disease use the same patterns, so revisit them if any of this feels new.

Part 1: Real Individual Insurance Charges#

The first thing to confirm with any new dataset is how big it is. The code imports pandas and matplotlib, loads insurance_raw.csv with pd.read_csv, and checks .shape. The output is (1338, 7): 1,338 beneficiaries and 7 columns. With only seven columns, every field can be examined in turn, which is exactly how this lesson is structured. The row count also matters later, when you judge whether a small group, such as people with five children, is large enough to trust.

Part 2: Real Data Preview#

Knowing each column's data type tells you which ones you can average and which ones you need to group by. The code runs head(5) and then dtypes, but Jupyter only displays the last expression, so the output is the list of types. age and children are whole numbers (int64), bmi and charges are decimals (float64), and sex, smoker and region are object, meaning text. The text columns are your grouping variables. Run head(5) on its own if you also want to see sample rows.

Part 3: Real Overall Charge Statistics#

Before comparing groups, get a feel for the overall spread of charges. describe() on a single column returns count, mean, standard deviation, minimum, quartiles and maximum, and .round(2) tidies it. The output shows a mean of 13,270.42 but a median (the 50% row) of only 9,382.03, and a maximum of 63,770.43. When the mean sits well above the median, a small number of very large values are pulling it up. This is called right skew, and it is typical of cost data.

Part 4: Real Smoking Status#

Smoking is the most obvious candidate cost driver, so first check how many smokers there are. value_counts() counts each answer, and (insurance['smoker'] == 'yes').mean() gives the share of smokers, because the mean of True and False values is the proportion of Trues. Returning both as a tuple shows both results. The output shows 1,064 non-smokers and 274 smokers, a smoker rate of 20.5%. So roughly one person in five smokes, a group big enough to compare reliably.

Part 5: Real Charges by Smoking Status#

Now you can test whether smokers really cost more. groupby('smoker')['charges'].mean() averages charges for each group, and .round(2) keeps two decimal places. The output shows average charges of 8,434.27 for non-smokers and 32,050.23 for smokers, almost four times higher. This is by far the largest gap you will see in this lesson, and it sets the benchmark that every other factor is measured against.

Part 6: Visualizing Real Charges by Smoking Status#

A bar chart makes a gap of this size instantly clear to any audience. The code calls .plot(kind='bar') on the averages with a green and a red bar, labels the y-axis in dollars, and keeps the labels horizontal with rotation=0. The figure is saved with plt.savefig and closed, so there is no inline output. When you open the image, look for the smoker bar standing nearly four times taller than the non-smoker bar.

Part 7: Real Charge Distributions by Smoking Status#

Averages hide how values are spread, so it is worth seeing the full distribution of charges for each group. The code overlays two plt.hist calls, one for non-smokers and one for smokers, with bins=30 and alpha=0.6 so the overlap stays visible. The figure is saved and closed, so there is no inline output. In the image, look for the non-smoker bars piled up at low charges with a long thin tail, and the smoker bars spread much further to the right.

Part 8: Real Charges by Sex#

Sex is a standard demographic split, and checking it helps you rule factors in or out. The code averages charges by sex. The output shows 12,569.58 for women and 13,956.75 for men, a gap of under 1,400 dollars. Compared with the smoking gap of over 23,000 dollars, this is small. It could also partly reflect other differences between the groups, such as smoking rates, so treat it as a weak signal rather than a cause.

Part 9: Real Charges by Region#

Region could matter if healthcare prices or habits differ across the country. The code counts people per region with value_counts() and averages charges per region with groupby, returning both together. The output shows the regions are close to equal in size (324 to 364 people), and averages range from 12,346.94 in the southwest to 14,735.41 in the southeast. The southeast is highest, but the spread across regions is modest compared with the smoking effect.

Part 10: Real BMI Categories#

BMI is easier to discuss in standard health bands than as raw numbers. pd.cut assigns each value to a band using edges 0, 18.5, 25, 30 and 100 with the labels Underweight, Normal, Overweight and Obese. By default the right edge is included, so a BMI of exactly 30 counts as Overweight. value_counts() shows 705 people in the obese band, 386 overweight, 226 normal and only 21 underweight. More than half the beneficiaries fall in the obese band.

Part 11: Real Charges by BMI Category#

With BMI bands in place, you can see how charges change as weight rises. The code groups by bmi_category and averages charges. observed=True tells pandas to include only categories that actually appear, avoiding a warning for categorical columns. The output rises from 8,657.62 for underweight to 10,435.44 for normal, 10,997.80 for overweight, then jumps to 15,560.93 for obese. The big step happens at the obese band, not gradually.

Part 12: Visualizing Real Charges by BMI Category#

Charting the BMI averages shows the step pattern more clearly than a table. The code uses .plot(kind='bar') on the averages with one colour and horizontal labels. The figure is saved and closed, so there is no inline output. In the image, look for three bars of fairly similar height followed by a clearly taller obese bar. Keep the categories in their natural order, as they are here, so the chart reads from lightest to heaviest.

Part 13: Real Compounded High-Cost Group#

Two risk factors together can push costs far beyond either alone. The code filters for people who are both in the obese band and smokers, combining the two conditions with & and wrapping each in brackets. It returns the group size and its average charge. The output shows 144 people averaging 41,692.81 dollars, well above the 32,050.23 for all smokers and 15,560.93 for all obese beneficiaries. This small group is where costs concentrate most.

Part 14: Real Everyone Else, for Comparison#

A high average only means something when you compare it with everyone else. The code builds the same combined condition and flips it with ~, which means NOT, so it selects every person who is not an obese smoker. It then averages their charges. The output is 9,842.6 dollars. The obese smoker group therefore costs a little over four times as much on average as the rest. Remember to wrap the whole condition in brackets before applying ~.

Part 15: Real Charges by Number of Children#

Family size could plausibly raise costs, so it is worth checking. The code counts people by number of children with value_counts().sort_index() and averages charges for each count. The output shows averages rising from 12,365.98 with no children to 15,355.32 with three, then falling to 8,786.04 with five. However, only 25 people have four children and 18 have five, so those averages rest on very small groups. Always check group sizes before reading too much into a pattern.

Part 16: Real Age Groups#

Age groups show whether charges climb steadily with age. pd.cut uses edges 17, 30, 40, 50 and 64, and because the right edge is included by default, the first band covers ages 18 to 30 and the last covers 51 to 64. The output shows averages rising at every step: 9,397.55 for 18-29, 11,639.31 for 30-39, 14,782.04 for 40-49 and 18,084.99 for 50-64. Unlike children, age shows a steady, consistent increase.

Part 17: Visualizing Real Charges by Age Group#

A bar chart of the age averages makes the steady climb easy to see. The code plots the age group Series with .plot(kind='bar'), labels the y-axis in dollars and keeps the labels horizontal. The figure is saved and closed, so there is no inline output. In the image, look for a staircase pattern, with each bar taller than the one before. That consistency is what makes age a reliable cost factor, even though its effect is smaller than smoking.

Part 18: Real Correlation with Charges#

Correlation puts the numeric columns on a single scale from -1 to 1, where 0 means no straight-line relationship. The code selects age, bmi, children and charges, computes corr(numeric_only=True), keeps the charges column and sorts it. The output shows age at 0.299, bmi at 0.198 and children at only 0.068. Note that smoker is not included because it is text. Correlation only works on numbers, so the biggest driver is missing from this ranking.

Part 19: Real Scatter, Age vs Charges by Smoking Status#

A scatter plot can show two effects at once. The code draws two plt.scatter calls, plotting age against charges for non-smokers in green and smokers in red, with alpha=0.5 so overlapping points remain visible. The figure is saved and closed, so there is no inline output. In the image, look for bands of points that all rise with age, and notice that the smoker points sit in higher bands than the non-smoker points at every age.

Part 20: Real Median vs Mean Charges#

With skewed data, the mean and median can give very different answers, and you need to know which one you are reporting. The code calculates both for charges. The output shows a mean of 13,270.42 and a median of 9,382.03. The median is the middle value, so half of beneficiaries are charged less than about 9,400 dollars. The mean is pulled up by a smaller number of very expensive cases. For a typical person, quote the median; for total budget planning, the mean is more useful.

Part 21: Real Top Ten Most Expensive Beneficiaries#

Looking at the individual records behind the highest costs often reveals a pattern. nlargest(10, 'charges') returns the ten rows with the highest charges, and the column list keeps only the fields of interest. The output shows charges from 63,770.43 down to 48,885.14 dollars. Every one of these people is a smoker, and every BMI is above 30. Ages range widely, from 28 to 64, so age alone does not explain these extreme cases.

Part 22: Real Smoker Share Among Top Ten#

Putting a number on what you saw in the top ten makes the finding easy to quote. The code checks whether each smoker value equals 'yes' and takes .mean() of those booleans, giving the share of smokers, then multiplies by 100. The output is 100.0: all ten of the most expensive beneficiaries are smokers. Compared with a smoker rate of 20.5% across the whole dataset, this is a striking concentration.

Part 23: Real Cheapest Ten Beneficiaries#

The cheapest records are just as informative as the most expensive ones. nsmallest(10, 'charges') returns the ten rows with the lowest charges. The output shows charges between about 1,121.87 and 1,163.46 dollars, and every one of these people is an 18-year-old male non-smoker. Their BMI values vary widely, from 23.21 to 53.13, which suggests that at the youngest age, BMI makes little difference to cost when someone does not smoke.

Part 24: Real Simple Threshold Cost Rule#

A simple cost threshold lets you ask how concentrated high costs are. The code flags anyone with charges above 20,000 dollars, stores the flag as 1 or 0 with .astype(int), and takes its mean to get the share of high-cost people. It then filters to the flagged rows and uses .eq('yes').mean() to find the share who smoke. The output shows 20.4% of beneficiaries are high-cost, and 77.7% of those are smokers, even though smokers are only 20.5% of everyone.

Part 25: Real Non-Smoker High-Cost Outliers#

Not every high-cost case is a smoker, and those exceptions are worth a look. The code filters for non-smokers with charges above 20,000 dollars. It calls len() first, but only the last expression is displayed, so the output shows the first five rows from head(5). These include a 33-year-old with a normal BMI of 22.705 and charges of 21,984.47, and a 64-year-old charged 30,166.62. These cases cannot be explained by smoking, BMI or age alone. Print len(nonsmoker_high_cost) separately to see how many there are.

Part 26: Saving the Real Categorized Dataset#

Saving the data with its new category columns means later work can reuse the bands without rebuilding them. to_csv writes the file and index=False stops pandas adding the row index as an extra column. The code then reads the file back and compares .shape. The output is True, so the saved file has the same rows and columns, including bmi_category and age_group. Note that when reloaded, these become plain text columns rather than ordered categories.

Part 27: Real Sanity Check, Smoker Rate#

A sanity check confirms that a headline number has not changed since you first calculated it. The code compares smoker_rate with 20.5 using ==, and the output is True. This exact comparison works because the rate was rounded to one decimal place. For values that have not been rounded, use np.isclose instead, because tiny floating-point differences can make == return False.

Part 28: Real Sanity Check, Smoker Charge Multiple#

This check turns the main finding into a test: smokers should cost more than three times as much as non-smokers. The code divides the smoker average by the non-smoker average using label lookups such as charges_by_smoker['yes'], rounds to two decimals and checks whether the result is above 3. The output is True. Writing checks around your key claims, rather than only around counts, protects the conclusions you are going to present.

Part 29: Real Top Cost Driver Recommendation#

A recommendation should be concrete, so this step turns the smoking comparison into a single sentence. The f-string uses the format code :,.0f, which adds thousands separators and drops the decimals, to present money neatly. The output reads: smoking status alone drives average charges from $8,434 to $32,050, a 3.8x multiple. Phrasing a result as a multiple like this is often easier for decision-makers to grasp than two raw averages.

Part 30: Real Recap Print#

A one-line recap pulls the whole analysis together for a report or slide. The code prints an f-string using the row count and the overall mean, again formatted with :,.0f. The output reads: across 1338 beneficiaries averaging $13,270 in charges, smoking status proved the single strongest individual cost driver, well ahead of BMI, age or region. Because the numbers come from variables, the sentence stays accurate if the data is updated.

Wrap-Up: What You Learned#

• Real smoking status alone produced a real more than three-times difference in average charges, dwarfing every other single field in this dataset.
• Real right-skewed cost data means the real mean and real median tell genuinely different stories, always check both.
• Combining real obesity with real smoking compounded real cost far beyond either factor examined alone.
• A real simple cost threshold, cross-tabulated against smoking status, revealed just how concentrated real high-cost cases actually are.
• Even among real non-smokers, a real handful of high-cost outliers remind you that real no single field ever explains everything.
• Next video: a real global life expectancy and health-spending analysis, zooming out from individual bills to entire real countries.

Practice on your own

1. Recalculate the charges by smoking status and by BMI band using .median() instead of .mean(), and see how much the gaps shrink once the skew is reduced.
2. Build a two-way table of average charges with groupby(['age_group', 'smoker']) and check whether the smoker gap grows or stays the same as age rises.
3. Count how many non-smokers have charges above 20,000 dollars with len(), then compare their average age and BMI with all non-smokers to see what sets them apart.

The next video zooms out from individual bills to entire countries, with a global life expectancy and health-spending analysis.