Data Analytics 100, Video 32: Real Heart Disease Risk Factor Analysis#

• Video thirty-two of the hundred-video real-world data analytics series.
• A real, classic clinical dataset used to hunt for the real factors most strongly tied to a real heart disease diagnosis.
• Let's get into it.

This lesson is for learners who have worked through a first health dataset and want more practice with messy clinical data. The business question is: which patient measurements are most strongly linked to a heart disease diagnosis, and which simple combinations of symptoms flag the highest risk?

The data is a classic clinical dataset from the Cleveland Clinic, loaded from heart_disease_raw.csv, with 303 patients. Columns use short clinical codes: age, sex, cp (chest pain type), trestbps (resting blood pressure), chol (cholesterol), thalach (maximum heart rate reached during exercise), exang (exercise-induced chest pain), oldpeak (a heart trace measure from exercise testing), ca (number of major vessels seen on a fluoroscopy scan) and diagnosis, a severity score from 0 to 4.

In this lesson you will:

• fix untidy column names and a hidden ? missing-value marker
• turn a severity scale into a simple yes or no disease flag
• compare disease rates across sex, chest pain type and age group
• score a one-line threshold rule with precision and recall

Prerequisites: you should be comfortable with pandas groupby, boolean filtering and pd.cut. The previous lesson on diabetes risk factors uses the same patterns.

Part 1: A Real Clinical Dataset from Cleveland Clinic#

Start by checking the size of the data so you know what you are working with. The code imports pandas, numpy and matplotlib, loads heart_disease_raw.csv with pd.read_csv, and checks .shape. The output is (303, 14): 303 patients and 14 columns. That is a small dataset, so each patient carries real weight in every percentage you calculate. A group of 30 patients, for example, is only about 10% of the data, and its rates will move a lot with a few cases.

Part 2: A Real Messy Header#

Column names in real CSV files often contain stray spaces, which break code like heart['age'] with a confusing KeyError. The code uses a list comprehension with c.strip() to remove leading and trailing whitespace from every name, then lists them again. Only the final tolist() is displayed, so the output shows the cleaned names: age, sex, cp, trestbps, chol, fbs, restecg, thalach, exang, oldpeak, slope, ca, thal and diagnosis. Stripping names right after loading is a good habit for any new file.

Part 3: Real Data Preview#

Looking at real rows shows you how values are coded before you start comparing groups. head(5) displays the first five patients. Notice that most columns are plain numbers, but categorical fields such as sex, cp and exang are also stored as numeric codes, for example sex is 1 or 0. Also look at ca and thal, which show values like 0.0 and 6.0. They look numeric, but as the next steps reveal, pandas has actually read them as text because of a hidden marker.

Part 4: Real Diagnosis Severity Scale#

The target column needs to be understood before you can measure anything against it. value_counts() counts patients for each diagnosis value and .sort_index() puts them in order from 0 to 4 rather than by size. The output shows 164 patients with a score of 0 (no disease), then 55, 36, 35 and 13 for levels 1 to 4. The higher levels are small groups, so analysing each one separately would give unstable rates. That is the reason for the next step.

Part 5: Real Binary Diagnosis Flag#

Merging the severity levels into a single yes or no flag gives you groups large enough to compare reliably. heart['diagnosis'] > 0 returns True for any level of disease, and .astype(int) turns True and False into 1 and 0, stored in a new has_disease column. Because the flag is 0 or 1, .mean() gives the share of patients with disease. The output is 45.9, so nearly half of the patients in this dataset have some heart disease. This is the baseline for every comparison that follows.

Part 6: A Real Hidden Missing-Value Marker#

Missing values do not always appear as blanks. In this file, a literal question mark stands in for an unknown value, which forces pandas to treat the whole column as text. The code compares ca and thal to the string '?' and uses .sum() to count the matches. The output shows 4 question marks in ca and 2 in thal. If you missed these, any attempt to average the columns would either fail or give misleading results.

Part 7: Real Cleaning, Question Marks to Missing#

Now the question marks need to become real missing values and the columns need to become numbers. pd.to_numeric converts text to numbers, and errors='coerce' tells it to turn anything it cannot convert, such as '?', into NaN instead of raising an error. isna().sum() then counts the missing values. The output shows 4 for ca and 2 for thal, exactly matching the question mark counts, so nothing else was lost. Always compare counts before and after a coerce step like this.

Part 8: Real Descriptive Statistics#

With every column numeric, describe() can finally summarise the whole table. It returns count, mean, standard deviation, minimum, quartiles and maximum, and .round(2) tidies the numbers. The count row confirms that ca has 299 values and thal 301, reflecting the missing entries. The average age is 54.44, and the mean of sex is 0.68, meaning about 68% of patients are male. Note that describe() also summarises coded categories like cp, where a mean has little meaning.

Part 9: Real Disease Rate by Sex#

Sex is a known risk factor for heart disease, so it is a natural first comparison. The code uses .map() with a dictionary to turn the codes 1 and 0 into readable labels, Male and Female, then groups by the new label and averages has_disease to get a rate. The output shows 25.8% for women and 55.3% for men, more than double. Since most patients are men, this also helps explain why the overall rate of 45.9% is so high.

Part 10: Visualizing Real Disease Rate by Sex#

A simple bar chart turns the two sex rates into something you can show to a non-technical audience. The code calls .plot(kind='bar') on the rate Series with two colours, adds a y-axis label and title, and uses plt.xticks(rotation=0) to keep the labels horizontal. The figure is saved to a file and closed, so there is no inline output. When you open it, look for the male bar standing roughly twice as tall as the female bar.

Part 11: Real Age by Diagnosis#

Age is one of the most common risk factors, so it is worth checking how much it differs between the two groups. The code groups by has_disease and averages age. The output shows 52.6 years for patients without disease and 56.6 for those with it, a gap of four years. That is a real difference, but a modest one, and it matches the fairly weak correlation you will see for age later in the lesson.

Part 12: Real Cholesterol by Diagnosis#

Cholesterol is often the first thing people associate with heart disease, so it is useful to test that assumption against the data. The code averages chol within each diagnosis group. The output shows 242.6 for patients without disease and 251.5 for those with it, a difference of only about 9 units on values that range from 126 to 564. In this dataset, cholesterol separates the groups much less than you might expect, a reminder that data can challenge common beliefs.

Part 13: Real Max Heart Rate by Diagnosis#

Maximum heart rate during an exercise test reflects how well the heart copes with effort. The code averages thalach by diagnosis. The output shows 158.4 for patients without disease and 139.3 for those with it, a gap of about 19 beats per minute. Here the direction is reversed: patients with heart disease reach a lower peak heart rate. Expect this column to show a negative correlation with the diagnosis, which is just as useful as a positive one.

Part 14: Real Resting Blood Pressure by Diagnosis#

Resting blood pressure is another standard clinical measurement worth comparing. The code averages trestbps within each diagnosis group. The output shows 129.2 for patients without disease and 134.6 for those with it, a difference of about 5 units. That is a smaller gap than for maximum heart rate. When comparing gaps across columns, remember they are on different scales, which is why correlation, a scale-free measure, is used in the next steps.

Part 15: Visualizing Real Max Heart Rate Distributions#

A histogram shows how much the two groups overlap, which a pair of averages cannot. The code draws two plt.hist calls on the same axes, one for each value of has_disease, with bins=20 and alpha=0.6 so overlapping bars stay visible. The figure is saved and closed, so there is no inline output. In the image, look for the red disease bars sitting further left, at lower heart rates, and note the wide shared middle range, which means heart rate alone cannot perfectly separate the groups.

Part 16: Real Correlation with Diagnosis#

Correlation ranks all the numeric measurements on one scale from -1 to 1, where 0 means no straight-line relationship. The code selects a list of columns, computes corr(numeric_only=True), keeps the has_disease column and sorts it. The output puts ca first at 0.460 and oldpeak at 0.425, then age at 0.223. thalach sits at the bottom at -0.417, a strong negative link. Cholesterol is close to zero at 0.085. Remember to judge strength by size, ignoring the sign.

Part 17: Visualizing the Real Correlation Matrix#

A heatmap shows the full correlation matrix, including how the measurements relate to each other. The code builds the matrix for the same columns, draws it with plt.imshow using the coolwarm colour map, and sets vmin=-1, vmax=1 so blue always means negative and red always means positive. The chart is saved and closed, so there is no inline output. In the image, look at the has_disease row: blue for thalach and warmer colours for ca and oldpeak.

Part 18: Real Chest Pain Type Distribution#

Chest pain type is a categorical code from 1 to 4, so you count patients per type rather than averaging. value_counts().sort_index() lists each code in order. The output shows 23 patients with type 1, 50 with type 2, 86 with type 3 and 144 with type 4. Type 4 covers nearly half the patients. In this dataset, type 4 is asymptomatic, meaning no typical chest pain was reported. Counting first tells you which rates in the next step rest on small groups.

Part 19: Real Disease Rate by Chest Pain Type#

Now you can see how disease rates differ across the chest pain types. The code groups by cp and averages has_disease, then converts to a percentage. The output shows 72.9% for type 4, far above types 1 (30.4%), 2 (18.0%) and 3 (20.9%). This is counter-intuitive: the group with no typical chest pain has the highest rate. It is a good example of why you should let the data speak, and check category meanings carefully, before drawing conclusions.

Part 20: Visualizing Real Disease Rate by Chest Pain Type#

A bar chart makes the outlier category stand out immediately. The code plots the chest pain rates with .plot(kind='bar'), labels both axes, and keeps the type codes horizontal with rotation=0. The figure is saved and closed, so there is no inline output. In the image, look for the type 4 bar standing far above the other three. If you present this to others, replace the numeric codes with their names so readers do not need a lookup table.

Part 21: Real Exercise-Induced Angina#

Exercise-induced angina means chest pain that appears during physical effort, recorded as 1 in exang. The code groups by exang and averages has_disease to get the disease rate for each value. The output shows 30.9% for patients without exercise-induced angina and 76.8% for those with it, about two and a half times higher. This makes exang one of the clearest single indicators in the dataset.

Part 22: Real Age Groups#

Banding age into groups lets you see whether risk rises steadily. pd.cut assigns each age to a bin with edges 20, 40, 50, 60 and 100. By default the right edge is included, so a patient aged exactly 40 lands in Under 40 and one aged 50 lands in 40-49; set right=False if you want the labels to match exactly. The output shows 33.3% under 40, 30.3% for 40-49, then a jump to 50.8% for 50-59 and 55.7% for 60+.

Part 23: Real Compounded High-Risk Group#

Combining risk indicators often identifies a much riskier group than any single factor. The code filters for patients with chest pain type 4 and exercise-induced angina, joining the two conditions with & and wrapping each in brackets. It returns the size of the group and its disease rate. The output shows 80 patients with a rate of 87.5%, higher than the 72.9% for chest pain type 4 alone and the 76.8% for angina alone, and almost double the overall rate of 45.9%.

Part 24: Real Simple Threshold Risk Rule#

A single threshold rule is a useful benchmark before trying any model. The code flags patients whose maximum heart rate is below 140, using .astype(int) to store the flag as 1 or 0, and then counts three outcomes by combining conditions with &. The output shows 61 true positives (flagged and diseased), 24 false positives (flagged but healthy) and 78 false negatives (diseased but not flagged). The large number of missed cases is already a warning sign about this rule.

Part 25: Real Precision and Recall#

Precision and recall summarise how good the rule is. Precision is the share of flagged patients who really have disease: true positives divided by all flagged. Recall is the share of all diseased patients the rule catches: true positives divided by true positives plus false negatives. The output gives a precision of 0.718 and a recall of 0.439. So when the rule flags someone it is right about 72% of the time, but it misses more than half of the patients with disease.

Part 26: Real Missing Data Impact Check#

Before trusting conclusions about a column, check how much of it is missing. isna() returns True for each missing value, and .mean() on those booleans gives the share; multiplying by 100 turns it into a percentage. The output shows 1.3% of ca values are missing, which is only 4 patients. That is small enough that it is unlikely to change the conclusions, which is reassuring since ca turns out to be the strongest correlate.

Part 27: Saving the Real Cleaned Dataset#

Saving the cleaned data lets you, or a colleague, pick up from the corrected version without repeating the fixes. to_csv writes the file and index=False stops pandas writing the row numbers as an extra column. The code reads the file straight back and compares .shape. The output is True, so the saved file has the same rows and columns as the cleaned DataFrame, including the new has_disease, sex_label and age_group columns.

Part 28: Real Sanity Check, Disease Rate#

A sanity check locks in a key result so that later edits cannot quietly change it. The code compares the stored disease_rate with 45.9 using ==, and the output is True. This works because the rate was rounded to one decimal place when it was calculated. For unrounded floating-point numbers, prefer np.isclose, because tiny rounding differences can make an exact comparison return False even when the values are effectively equal.

Part 29: Real Top Risk Factor Recommendation#

A clear headline finding is what stakeholders remember. The code drops has_disease from the correlations, applies .abs() so strong negative links count as much as positive ones, and uses .idxmax() to find the strongest. It then looks up the original signed value. The output names ca at 0.46. Without .abs(), a strong negative correlate like thalach at -0.417 could never be picked, so this is an important detail when ranking correlations.

Part 30: Real Recap Print#

A short recap sentence is the kind of summary you would put at the top of a report. The code prints an f-string that pulls the patient count, the disease rate and the top factor from variables calculated earlier. The output reads: across 303 patients, 45.9% had a confirmed heart disease diagnosis, and ca stood out as the strongest individual risk signal. In a real report, translate codes like ca into plain language, such as number of major vessels seen on the scan.

Wrap-Up: What You Learned#

• Real-world CSVs carry real formatting quirks, from stray whitespace in column names to a real literal question mark standing in for a real missing value.
• Collapsing a real multi-level severity scale into a real clean binary flag made this real entire analysis dramatically simpler without losing the real core question.
• Real max heart rate and real number of vessels visible on fluoroscopy were the real strongest individual correlates of diagnosis in this dataset, well ahead of cholesterol.
• Combining two real risk indicators, asymptomatic chest pain and exercise-induced angina, compounded real risk well beyond either factor alone.
• Even a real single-threshold rule, with no modeling at all, can be honestly scored with real precision and real recall.
• Next video: a real medical insurance cost analysis, digging into what actually drives individual healthcare charges.

Practice on your own

1. Repeat the disease rate comparison for fbs (fasting blood sugar flag) and for thal, using groupby and .mean() on has_disease, and compare the results with the exang rates.
2. Try maximum heart rate thresholds of 130, 150 and 160 in the simple risk rule and record precision and recall for each. Which threshold gives the best balance?
3. Re-run the age banding with right=False and check how the rates for Under 40 and 40-49 change once 40-year-olds move into the correct group.

The next video moves on to a medical insurance cost analysis, digging into what actually drives individual healthcare charges.