Lesson 35 · Real-World Data Analytics
Handle '?' and Impossible Values in Pandas: Kidney Disease Data | Python Data Analytics #35
A real, classic clinical dataset from a hospital in Tamil Nadu, India, hunting for exactly what separates real kidney disease patients from real healthy ones.
- CourseReal-World Data Analytics
- Lesson35 of 100
- Video16 min
- FormatJupyter notebook · 30 code cells
- Data1 dataset
What you'll learn
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
- ckd_raw.csv preview41.8 KB
📓 Full notebook
Download .ipynbData Analytics 100, Video 35: Real Chronic Kidney Disease Risk Factor Analysis#
- Video thirty-five of the hundred-video real-world data analytics series.
- A real, classic clinical dataset from a hospital in Tamil Nadu, India, hunting for exactly what separates real kidney disease patients from real healthy ones.
- Let's get into it.
This lesson is for learners who have tackled the earlier health datasets in the series and are ready for a messier one. The question is: what separates patients with chronic kidney disease (CKD, a long-term loss of kidney function) from those without it, and which single measurement signals it most clearly?
The data is a classic clinical dataset from a hospital in Tamil Nadu, India, loaded from ckd_raw.csv. It has 400 patients and 25 columns. Many use short medical codes: lab results such as hemo (haemoglobin), pcv (packed cell volume), sc (serum creatinine), bgr (random blood glucose) and bp (blood pressure), yes or no fields such as htn (hypertension), dm (diabetes), ane (anaemia) and pe (pedal oedema, swelling of the feet), and the target class, which is ckd or notckd.
In this lesson you will:
- load data with a custom missing-value marker and audit what is missing
- compare lab values and condition rates between diagnosis groups
- rank risk factors by correlation, including negative ones
- check whether missing data itself differs between groups
Prerequisites: you should be comfortable with pandas groupby, boolean filtering and basic matplotlib. If lambda functions are new to you, a quick refresher will help.
Part 1: A Real Clinical Dataset#
Loading the file correctly from the start saves cleaning later. In this dataset a question mark stands in for unknown values, so the code passes na_values='?' to pd.read_csv, which turns every ? into a proper missing value as the file is read. It then checks .shape. The output is (400, 25): 400 patients and 25 columns. That is a lot of columns for so few patients, so expect some fields to be sparsely filled.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
ckd = pd.read_csv('ckd_raw.csv', na_values='?')
ckd.shape
Part 2: Real Data Preview#
Before comparing anything, you need to know how the diagnosis is recorded and how balanced the groups are. The code runs head(5) and then value_counts() on class; only the last expression is displayed, so the output is the class counts. It shows 250 patients with ckd and 150 with notckd. Both groups are large enough to compare. Run head(5) on its own to see what the coded columns look like.
ckd.head(5)
ckd['class'].value_counts()
Part 3: Real Overall Prevalence#
The overall prevalence is the baseline for everything that follows. The code compares class with 'ckd', which gives True or False for each patient, and .mean() on those booleans gives the share that are True. Multiplying by 100 and rounding gives 62.5%. Remember this is a hospital sample, not the general population, so the rate is far higher than you would see in the community. Do not quote it as a population prevalence.
ckd_rate = round((ckd['class'] == 'ckd').mean() * 100, 1)
ckd_rate
Part 4: Real Missing Data Audit#
Clinical data is often incomplete, and you need to know where the gaps are before you trust any average. isna().sum() counts missing values per column, sort_values(ascending=False) puts the worst first, and head(10) shows the top ten. The output shows rbc (red blood cell appearance) missing for 152 patients, rbcc (red blood cell count) for 131 and wbcc (white blood cell count) for 106. With 400 patients, rbc is missing for 38% of them, so results on that column need extra care.
missing_counts = ckd.isna().sum().sort_values(ascending=False)
missing_counts.head(10)
Part 5: Real Numeric Conversion#
Lab columns must be numeric before you can average or correlate them. Stray text in a single cell can make pandas store a whole column as text. The code lists the 14 numeric fields and uses .apply(pd.to_numeric, errors='coerce'), which converts each column and turns anything that cannot be read as a number into a missing value instead of raising an error. The output of dtypes confirms all 14 columns are now float64, so they are ready for calculations.
numeric_cols = ['age', 'bp', 'sg', 'al', 'su', 'bgr', 'bu', 'sc', 'sod', 'pot', 'hemo', 'pcv', 'wbcc', 'rbcc']
ckd[numeric_cols] = ckd[numeric_cols].apply(pd.to_numeric, errors='coerce')
ckd[numeric_cols].dtypes
Part 6: Real Hemoglobin by Diagnosis#
Haemoglobin is the protein in red blood cells that carries oxygen, and healthy kidneys help the body make red blood cells. The code groups by class and averages hemo, skipping missing values automatically. The output shows 10.65 for CKD patients and 15.19 for those without CKD, a gap of over 4 units. Lower haemoglobin in the CKD group fits what you would expect clinically, and it points to hemo as a strong candidate signal.
hemo_by_class = ckd.groupby('class')['hemo'].mean().round(2)
hemo_by_class
Part 7: Visualizing Real Hemoglobin Distributions#
A histogram shows whether the two haemoglobin groups overlap or sit apart. The code draws two plt.hist calls on the same axes, one per class, each with .dropna() to remove missing values, bins=20 and alpha=0.6 so overlap stays visible. The figure is saved and closed, so there is no inline output. In the image, look for the red CKD bars concentrated at lower values and the green bars at higher values, and note how small the overlap region is.
plt.figure(figsize=(8, 5))
plt.hist(ckd[ckd['class']=='ckd']['hemo'].dropna(), bins=20, alpha=0.6, label='CKD', color='crimson')
plt.hist(ckd[ckd['class']=='notckd']['hemo'].dropna(), bins=20, alpha=0.6, label='No CKD', color='seagreen')
plt.xlabel('Real Hemoglobin (g/dL)')
plt.ylabel('Real Patient Count')
plt.title('Real Hemoglobin by Diagnosis')
plt.legend()
plt.tight_layout()
plt.savefig('hemoglobin_by_diagnosis.png', dpi=120)
plt.close()
Part 8: Real Blood Pressure by Diagnosis#
High blood pressure both damages the kidneys and results from kidney damage, so it is a natural comparison. The code averages bp within each class. The output shows 79.62 for CKD patients and 71.35 for those without CKD, a gap of about 8 units. That is a clear difference, but smaller relative to the scale than the haemoglobin gap, which you will see confirmed in the correlation ranking later.
bp_by_class = ckd.groupby('class')['bp'].mean().round(2)
bp_by_class
Part 9: Real Serum Creatinine by Diagnosis#
Serum creatinine is a waste product that healthy kidneys filter out of the blood, so high levels suggest the kidneys are not working well. The code averages sc by class. The output shows 4.41 for CKD patients and only 0.87 for those without CKD, about five times higher. Be aware that a very large average gap can be driven by a few extreme values, which is why the correlation for sc later turns out more modest than this gap suggests.
sc_by_class = ckd.groupby('class')['sc'].mean().round(2)
sc_by_class
Part 10: Real Blood Glucose by Diagnosis#
Diabetes is a leading cause of kidney disease, so blood glucose is worth checking. The code averages bgr, a random blood glucose reading taken without fasting, within each class. The output shows 175.42 for CKD patients and 107.72 for those without CKD, a gap of almost 68 units. This lines up with the high diabetes rate in the CKD group shown a little later.
bgr_by_class = ckd.groupby('class')['bgr'].mean().round(2)
bgr_by_class
Part 11: Real Age by Diagnosis#
Age can explain many other differences between groups, so it is worth knowing how far apart the groups are. The code averages age by class. The output shows 54.54 years for CKD patients and 46.52 for those without CKD, a gap of about 8 years. That is a real difference, but older patients also tend to have higher blood pressure and glucose, so some of the gaps seen so far may partly reflect age.
age_by_class = ckd.groupby('class')['age'].mean().round(2)
age_by_class
Part 12: Real Hypertension Prevalence#
Hypertension is stored as text, yes or no, so you cannot simply average it. The code groups by class, then uses .apply() with a lambda (a small one-line function) that checks s == 'yes' and takes the mean, giving the share of yes answers in each group. The output shows 58.8% of CKD patients have hypertension, compared with 0.0% of those without CKD. Note that missing values count as not yes in this calculation, so the CKD rate may be slightly understated.
htn_by_class = (ckd.groupby('class')['htn'].apply(lambda s: (s == 'yes').mean()) * 100).round(1)
htn_by_class
Part 13: Real Diabetes Prevalence#
Diabetes is the other major condition linked to kidney disease. The code uses the same groupby and lambda pattern on the dm column, calculating the share of yes answers per class. The output shows 54.8% of CKD patients have diabetes, and again 0.0% of those without CKD. A perfect zero in the healthy group is unusual in real data and suggests the control patients may have been selected to be free of these conditions.
dm_by_class = (ckd.groupby('class')['dm'].apply(lambda s: (s == 'yes').mean()) * 100).round(1)
dm_by_class
Part 14: Visualizing Real Comorbidity Rates#
Showing the two condition rates side by side makes the comparison easy to present. The code combines the hypertension and diabetes rates into a DataFrame and calls .plot(kind='bar') to draw grouped bars. The stored output shows <Figure size 700x500 with 0 Axes>, an empty figure. This appears because DataFrame.plot opens its own new figure, leaving the one from plt.figure blank. In the saved image, look for two tall bars for CKD and none for patients without CKD.
comorbidity_table = pd.DataFrame({'Hypertension': htn_by_class, 'Diabetes': dm_by_class})
plt.figure(figsize=(7, 5))
comorbidity_table.plot(kind='bar')
plt.ylabel('Real Prevalence (%)')
plt.title('Real Comorbidity Rates by Diagnosis')
plt.xticks(rotation=0)
plt.tight_layout()
plt.savefig('comorbidity_rates.png', dpi=120)
plt.close()
Part 15: Real Compounded Risk#
Checking patients with both conditions shows how risk compounds. The code filters for rows where htn and dm are both yes, combining the conditions with &, then calculates the share with CKD. The output shows 106 patients, all of them with CKD, a rate of 100.0%. Read this carefully: since no patient without CKD had either condition, any patient with both was bound to be in the CKD group. The result reflects how the sample was built as much as true risk.
both_conditions = ckd[(ckd['htn'] == 'yes') & (ckd['dm'] == 'yes')]
both_rate = round((both_conditions['class'] == 'ckd').mean() * 100, 1)
len(both_conditions), both_rate
Part 16: Real Correlation Ranking#
Correlation ranks all the lab values on one scale from -1 to 1. The code first creates ckd_flag, a 1 or 0 version of class, because correlation needs numbers. It then computes corr(numeric_only=True), keeps the ckd_flag column, removes its self-correlation with .drop, and sorts. The output shows strong negative links for hemo at -0.769, pcv at -0.741 and rbcc at -0.699, while the strongest positive is bgr at 0.420. Judge strength by size, so negative values count just as much.
ckd['ckd_flag'] = (ckd['class'] == 'ckd').astype(int)
corr_cols = ['age', 'bp', 'bgr', 'bu', 'sc', 'sod', 'pot', 'hemo', 'pcv', 'wbcc', 'rbcc']
diag_corr = ckd[corr_cols + ['ckd_flag']].corr(numeric_only=True)['ckd_flag'].drop('ckd_flag').sort_values()
diag_corr.round(3)
Part 17: Visualizing the Real Correlation Ranking#
A horizontal bar chart is a clear way to show a ranked list with both positive and negative values. The code plots the sorted correlations with kind='barh'. The figure is saved and closed, so there is no inline output. In the image, look for the long bars extending to the left for hemo, pcv and rbcc, and the shorter bars to the right for bgr, bu and sc. Bars pointing in opposite directions make it obvious which measurements fall and which rise with CKD.
plt.figure(figsize=(8, 6))
diag_corr.plot(kind='barh', color='darkorange')
plt.xlabel('Real Correlation with CKD Diagnosis')
plt.title('Real Risk Factor Correlation Ranking')
plt.tight_layout()
plt.savefig('risk_factor_correlation.png', dpi=120)
plt.close()
Part 18: Real Anemia by Diagnosis#
Anaemia means having too few healthy red blood cells, and it often accompanies kidney disease. The code uses the groupby and lambda pattern to find the share of yes answers in ane for each class. The output shows 24.0% of CKD patients are recorded as anaemic, compared with 0.0% of those without CKD. This fits with the low haemoglobin found earlier, though the recorded rate is lower than the haemoglobin gap alone might suggest.
ane_by_class = (ckd.groupby('class')['ane'].apply(lambda s: (s == 'yes').mean()) * 100).round(1)
ane_by_class
Part 19: Real Appetite by Diagnosis#
Poor appetite is a common symptom as waste products build up in the blood. Here the code checks for the value 'poor' rather than 'yes', because appet records good or poor. Always check what values a text column actually contains before writing the condition. The output shows 32.8% of CKD patients report poor appetite, compared with 0.0% of those without CKD. Symptoms like this are less specific than lab results but are easy to observe.
appet_by_class = (ckd.groupby('class')['appet'].apply(lambda s: (s == 'poor').mean()) * 100).round(1)
appet_by_class
Part 20: Real Pedal Edema by Diagnosis#
Pedal oedema is swelling in the feet and ankles caused by fluid build-up, which can happen when the kidneys cannot remove excess fluid. The code calculates the share of yes values in pe within each class. The output shows 30.4% of CKD patients have pedal oedema, compared with 0.0% of those without CKD. Like appetite, it is a visible symptom that a clinician can spot without any lab test.
pe_by_class = (ckd.groupby('class')['pe'].apply(lambda s: (s == 'yes').mean()) * 100).round(1)
pe_by_class
Part 21: Real Abnormal Red Blood Cells#
The rbc column records whether red blood cells in a urine sample look normal or abnormal. The code checks for 'abnormal' within each class. The output shows 18.8% of CKD patients have abnormal red blood cells, compared with 0.0% of those without CKD. Be cautious with this figure: rbc is the most incomplete column in the dataset, and missing values count as not abnormal here, so the true CKD rate among patients actually tested is likely higher.
rbc_by_class = (ckd.groupby('class')['rbc'].apply(lambda s: (s == 'abnormal').mean()) * 100).round(1)
rbc_by_class
Part 22: Real Packed Cell Volume by Diagnosis#
Packed cell volume is the percentage of blood made up of red blood cells, so it is closely linked to haemoglobin. The code averages pcv by class. The output shows 32.94 for CKD patients and 46.34 for those without CKD, a gap of over 13 points. This matches its strong negative correlation of -0.741. When two measurements are this closely related, they largely tell the same story, so adding both to a model adds little new information.
pcv_by_class = ckd.groupby('class')['pcv'].mean().round(2)
pcv_by_class
Part 23: Real Simple Threshold Risk Rule#
A one-line rule shows how far a single lab value can take you. The code flags patients with hemo below 12 as at risk, then counts true positives and false positives. valid_rows uses notna() to make sure only patients with a recorded haemoglobin are counted. The output shows 147 true positives (flagged and with CKD) and 0 false positives (flagged but without CKD). The code does not count false negatives, so you cannot yet tell how many CKD patients the rule misses.
risk_flag = (ckd['hemo'] < 12).astype(int)
actual = ckd['ckd_flag']
valid_rows = ckd['hemo'].notna()
true_positive = ((risk_flag == 1) & (actual == 1) & valid_rows).sum()
false_positive = ((risk_flag == 1) & (actual == 0) & valid_rows).sum()
true_positive, false_positive
Part 24: Real Precision of the Threshold Rule#
Precision answers the question: when the rule flags someone, how often is it right? It is true positives divided by all flagged patients, here 147 out of 147. The output is 1.0, meaning every flagged patient has CKD. That sounds perfect, but precision alone ignores missed cases, so also calculate recall (true positives divided by all CKD patients) before calling the rule useful. A rule can have perfect precision and still miss many patients.
threshold_precision = round(true_positive / (true_positive + false_positive), 3)
threshold_precision
Part 25: Real Missing Data Impact by Diagnosis#
Missing data is not always random, and when it differs between groups it can bias your results. The code groups by class and uses a lambda that applies isna().mean() to rbc, giving the share missing in each group. The output shows rbc is missing for 57.2% of CKD patients but only 6.0% of those without CKD. So the abnormal red blood cell rate for CKD patients rests on fewer than half of them, which is an important caveat to state.
missing_by_class = ckd.groupby('class')['rbc'].apply(lambda s: s.isna().mean() * 100).round(1)
missing_by_class
Part 26: Saving the Real Cleaned Dataset#
Saving the cleaned data lets later work start from numeric columns and the new ckd_flag, without repeating the conversion. to_csv writes the file and index=False leaves out the row index. The code reloads it and compares .shape. The output is True, so the saved file matches the cleaned DataFrame in rows and columns. When you reload it later, the question marks are already gone, since they were saved as empty cells.
ckd.to_csv('ckd_cleaned.csv', index=False)
reloaded = pd.read_csv('ckd_cleaned.csv')
reloaded.shape == ckd.shape
Part 27: Real Sanity Check, Prevalence#
This sanity check confirms the headline prevalence has not changed during the analysis. The code compares ckd_rate with 62.5 using ==, and the output is True. The exact comparison works because the rate was rounded to one decimal place. For unrounded floating-point values, use np.isclose, since tiny differences can make == return False even when two numbers are effectively equal.
ckd_rate == 62.5
Part 28: Real Sanity Check, Row Count Preserved#
Cleaning steps such as converting columns should never remove patients. This check compares len(ckd) with 400, and the output is True, so every patient is still present. Because the conversion used errors='coerce' rather than dropping bad rows, unreadable values became missing cells instead of lost records. Keeping all rows makes later comparisons, such as the missing-data check, possible.
len(ckd) == 400
Part 29: Real Top Risk Factor Recommendation#
A clear headline is what decision-makers will remember. The code applies .abs() to the correlations so negative and positive values are ranked by strength, uses .idxmax() to find the strongest, and then looks up the original signed value. The output names hemo at -0.769. Without .abs(), the code would have picked bgr at 0.420, missing the strongest signal entirely, so always rank correlations by absolute value.
top_factor = diag_corr.abs().idxmax()
top_factor_corr = round(diag_corr[top_factor], 3)
print(f'{top_factor} is the real strongest single correlate of a real chronic kidney disease diagnosis in this dataset, at a real correlation of {top_factor_corr}.')
Part 30: Real Recap Print#
A one-sentence recap brings the analysis together for a report. The f-string pulls the patient count, the CKD rate and the top factor from variables calculated earlier. The output reads: across 400 patients, 62.5% carried a confirmed kidney disease diagnosis, and hemo stood out as the strongest individual risk signal. In a written report, replace hemo with haemoglobin and note that lower values go with disease.
print(f'Across {len(ckd)} real patients, {ckd_rate}% carried a confirmed kidney disease diagnosis, and {top_factor} stood out as the real strongest individual risk signal.')
Wrap-Up: What You Learned#
- Real clinical data is genuinely messy, this dataset had real missing values scattered across nearly every lab field, sometimes over a third of all patients.
- Real hemoglobin and real packed cell volume were the real strongest individual correlates of diagnosis, both real negative, lower blood counts tracked with real kidney disease.
- Cross-tabulating real comorbidities like hypertension and diabetes against diagnosis surfaced a real compounded high-risk group.
- Even a real single-threshold rule, using only real hemoglobin, can be honestly scored with real precision against actual outcomes.
- Checking whether real missingness itself differs by group is a real easy, honest step most analyses skip entirely.
- Next video: a real global vaccination and public health snapshot, wrapping up this domain with a genuinely global view.
Practice on your own
- Extend the threshold rule by counting false negatives (CKD patients with a recorded
hemoof 12 or more), then calculate recall and compare it with the precision of 1.0. - Repeat the missing-data check by class for
wbcc,rbccandpot, and note which columns are missing far more often for one group. - Recalculate the hypertension rate using only patients with a recorded
htnvalue (filter withnotna()first) and see whether the CKD rate of 58.8% changes.
The next video wraps up this healthcare domain with a global vaccination and public health snapshot.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.



