Mathew K AnalyticsLesson 5 · Pandas Full Course
Pandas DataFrame Basics: Add, Drop, Rename and Summarise Columns | Pandas Course #5
In this lesson you'll be able to: Read a table's key attributes: shape, size, columns, index, dtypes Add new columns with fast, whole-column arithmetic Drop…
- CoursePandas Full Course
- Lesson5 of 65
- Video24 min
- FormatJupyter notebook · 45 code cells
- Data1 dataset
What you'll learn
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
- titanic.csv preview58.9 KB
📓 Full notebook
Download .ipynbPandas 05: Basic DataFrame Operations and Attributes#
In this lesson you'll be able to:
- Read a table's key attributes: shape, size, columns, index, dtypes
- Add new columns with fast, whole-column arithmetic
- Drop and rename columns without losing your original data
- Summarise down columns or across rows with the axis argument
- Change column types safely, missing values included Dataset: the Titanic passenger list (data/pandas/titanic.csv)
Warm-up: load and inspect#
Lesson 04 loaded and inspected a file. Same moves, new dataset.
import pandas as pd
df = pd.read_csv("data/pandas/titanic.csv")
cols = ["Name", "Sex", "Age", "Pclass", "Fare"]
df[cols].head(3)
df.isna().sum()
Attributes: the facts a table knows about itself#
Problem: answer 'how big, what columns, what types?' instantly.
print(df.shape)
print(df.size)
print(df.ndim)
print(df.columns[:4])
print("Fare" in df.columns)
print(df.index)
df.dtypes.value_counts()
df.drop(columns="Name").head(3).T
arr = df[["Age", "Fare"]].to_numpy()
print(type(arr), arr.shape)
print(arr[:2])
Quick check: did anyone on board have an age above 80?
over_80 = df[df["Age"] > 80]
print(over_80.empty)
print(df["Age"].max())
Common mistake: dot access with the wrong capital letter
df.fare.head()
df["Fare"].head()
Create new columns with whole-column arithmetic#
Problem: derive new facts, like family size, for all 891 rows at once.
df["family_size"] = df["SibSp"] + df["Parch"] + 1
df[["SibSp", "Parch", "family_size"]].head()
df["fare_pp"] = df["Fare"] / df["family_size"]
df[["Fare", "family_size", "fare_pp"]].head().round(2)
df["is_child"] = df["Age"] < 18
df["is_child"].value_counts()
avg_fare = df["Fare"].mean()
above = df["Fare"] > avg_fare
print(round(avg_fare, 2), round(above.mean(), 3))
df["ship"] = "Titanic"
print(df["ship"].nunique(), df.shape)
Common mistake: a list that doesn't match the number of rows
df["deck"] = ["A", "B", "C"]
df["deck"] = df["Cabin"].str[0]
df["deck"].value_counts().head(4)
months = df.assign(age_months=df["Age"] * 12)
print("age_months" in months.columns)
print("age_months" in df.columns)
Your turn#
How many passengers travelled with no family at all? Use family_size.
alone = df["family_size"] == 1
print(alone.sum())
print(round(alone.mean() * 100, 1))
Drop and rename columns safely#
Problem: trim and tidy a table without destroying the original.
slim = df.drop(columns=["Ticket", "Cabin", "ship"])
print(df.shape, slim.shape)
Common mistake: dropping a column without saying it's a column
df.drop("Cabin")
print(df.drop(columns="Cabin").shape)
print(df.drop(index=[0, 1]).shape)
names = {"Pclass": "pclass", "SibSp": "sib_sp"}
slim = slim.rename(columns=names)
print(slim.columns.tolist())
Quick check: after all that dropping, does df still have Cabin?
print("Cabin" in df.columns)
print("Cabin" in slim.columns)
Your turn#
Make a table people with Name, Sex and Age, all names in lower case.
people = df[["Name", "Sex", "Age"]]
people = people.rename(columns=str.lower)
people.head(3)
order = ["Name", "Sex", "Age", "pclass", "fare_pp"]
slim[order].head(3).round(2)
Summarise down columns or across rows#
Problem: one number per column, or one number per passenger.
fare = df["Fare"]
print(round(fare.sum(), 2), round(fare.mean(), 2))
print(fare.median(), fare.max())
Common mistake: asking for the mean of a whole mixed table
df.mean()
df[["Age", "Fare", "family_size"]].mean().round(2)
df[["Age", "Fare"]].agg(["min", "max"])
rel = df[["SibSp", "Parch"]].sum(axis=1)
print(rel.head().tolist())
print((rel + 1).equals(df["family_size"]))
top = df["Fare"].idxmax()
print(top)
df.loc[top, ["Name", "Pclass", "Fare", "family_size"]]
Quick check: which class carried the most passengers?
counts = df["Pclass"].value_counts().sort_index()
print(counts)
print(df["Pclass"].nunique())
Your turn#
What share of passengers survived, and what was the median age?
print(round(df["Survived"].mean(), 3))
print(df["Age"].median())
Change types with astype#
Problem: the right type saves memory and prevents silly mistakes.
print(df["Survived"].dtype)
survived = df["Survived"].astype(bool)
print(survived.head(4).tolist())
before = df["Embarked"].memory_usage(deep=True)
df["Embarked"] = df["Embarked"].astype("category")
after = df["Embarked"].memory_usage(deep=True)
print(before, after)
Common mistake: converting a column with gaps to int
df["Age"].astype(int)
age = df["Age"].round().astype("Int64")
print(age.dtype, age.isna().sum())
print(age.head().tolist())
first = df[df["Pclass"] == 1].copy()
first["Fare"] = first["Fare"].round()
print(first["Fare"].iloc[0], df.loc[1, "Fare"])
Mini-project: did paying more per person help?#
Plan: split at the median fare per person, compare, then check within class.
cut = df["fare_pp"].median()
df["high_pp"] = df["fare_pp"] > cut
print(cut, df["high_pp"].sum())
high = df.loc[df["high_pp"], "Survived"].mean()
low = df.loc[~df["high_pp"], "Survived"].mean()
print(round(high, 3), round(low, 3))
third = df[df["Pclass"] == 3].copy()
third_cut = third["fare_pp"].median()
hi3 = third["fare_pp"] > third_cut
rate_hi = third.loc[hi3, "Survived"].mean()
rate_lo = third.loc[~hi3, "Survived"].mean()
print(round(rate_hi, 3), round(rate_lo, 3))
male_hi = (third.loc[hi3, "Sex"] == "male").mean()
male_lo = (third.loc[~hi3, "Sex"] == "male").mean()
print(round(male_hi, 2), round(male_lo, 2))
import matplotlib.pyplot as plt
rates = pd.Series([high, low, rate_hi, rate_lo],
index=["All: top half", "All: bottom half",
"3rd: top half", "3rd: bottom half"])
ax = rates.plot(kind="bar", rot=0, figsize=(8, 4))
ax.set_title("Survival by fare per person: everyone vs 3rd class")
ax.set_xlabel("Group")
ax.set_ylabel("Share who survived")
plt.show()
gap_all = (high - low) * 100
gap_3rd = (rate_hi - rate_lo) * 100
print(f"All: {gap_all:+.0f} pts, third class: {gap_3rd:+.0f} pts")
Recap#
- Attributes (shape, size, columns, dtypes, T) need no brackets
- Whole-column arithmetic builds new columns without loops
- drop and rename return new tables; name
columns=explicitly - axis=0 summarises down columns, axis=1 across rows
- astype changes types; use Int64 when values are missing
Challenges#
- Add a column
fare_x2and check its mean is double the fare mean. - Rename
Survivedtosurvived, then dropPassengerId. - Repeat the mini-project inside first class. Does the gap hold?
What's next#
Lesson 06: Hands-on - Creating and Exploring DataFrames Use everything so far to explore a brand new dataset from scratch.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.
Related lessons
17 minHow to Rename Columns and Indexes in pandas DataFrames for Clear Data Analysis
26 minData Mining with Python: pandas & scikit-learn Crash Course
28 minIntroduction to Pandas DataFrame: Essential Operations and Attributes Explained
12 min