Mathew K AnalyticsLesson 3 · Pandas Full Course
Pandas Series vs DataFrame Explained: The 2 Objects Behind Every Table | Pandas Course #3
In this lesson you'll be able to: Build a Series and read its values, index and type Pull Series out of a real DataFrame and summarise them Explain how a…
- CoursePandas Full Course
- Lesson3 of 65
- Video25 min
- FormatJupyter notebook · 52 code cells
- Data1 dataset
What you'll learn
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
- titanic.csv preview58.9 KB
📓 Full notebook
Download .ipynbPandas 03: Understanding Series and DataFrames#
In this lesson you'll be able to:
- Build a Series and read its values, index and type
- Pull Series out of a real DataFrame and summarise them
- Explain how a DataFrame is a set of Series sharing one index
- Use index alignment and column maths to create new features Dataset: the Titanic passenger list, from start to finish
Warm-up: load it the lesson 02 way#
A pathlib path, checked before loading.
from pathlib import Path
import pandas as pd
path = Path("data") / "pandas" / "titanic.csv"
print(path.exists())
df = pd.read_csv(path)
print(df.shape)
A Series: one column of values with labels#
Values, plus an index that names each one.
four = df.head(4)
ages = pd.Series(four["Age"].to_list())
ages
print(ages.values)
print(ages.index)
print(ages.dtype, len(ages))
names = four["Name"].str.split(",").str[0].to_list()
ages = pd.Series(four["Age"].to_list(), index=names, name="Age")
ages
print(ages["Cumings"])
print(ages.mean())
print(ages > 30)
print(ages.idxmax(), ages.max())
ages.sort_values(ascending=False)
ages.describe()
counts = df["Embarked"].value_counts().to_dict()
print(counts)
ports = pd.Series(counts)
print(ports.index.tolist())
print(ports["C"] + ports["Q"])
Common mistake: asking for position 0 with square brackets
ages[0]
print(ages.iloc[0])
print(ages.loc["Braund"])
Pull a Series out of a real table#
Every column of a DataFrame is a Series with the same index.
age = df["Age"]
print(type(age))
print(age.name, len(age), age.count())
print(age.mean().round(1), age.median(), age.max())
print((age > 60).sum(), "passengers over 60")
filled = age.fillna(age.median())
print(filled.count(), age.count())
print(filled.mean().round(2))
youngest = age.nsmallest(3)
df.loc[youngest.index, ["Name", "Age"]]
df["Sex"].value_counts()
full = {"S": "Southampton", "C": "Cherbourg", "Q": "Queenstown"}
port = df["Embarked"].map(full)
port.value_counts(dropna=False)
Quick check: what's the difference between
df["Age"]anddf[["Age"]]?
print(type(df["Age"]), df["Age"].shape)
print(type(df[["Age"]]), df[["Age"]].shape)
first = df.loc[0]
print(type(first))
first[["Name", "Age", "Fare"]]
Your turn: how many passengers paid a fare below 10?
fare = df["Fare"]
print((fare < 10).sum(), "fares below 10")
print((fare < 10).mean().round(2), "as a share")
A DataFrame: Series that share one index#
Build one from a dictionary: keys become columns.
people = pd.DataFrame({
"Age": four["Age"].to_list(),
"Class": four["Pclass"].to_list(),
"Survived": four["Survived"].to_list(),
}, index=names)
people
print(people.columns.tolist())
print(people.index.tolist())
print(people.shape)
print(people.dtypes)
col = people["Class"]
print(type(col))
print(col.index.equals(people.index))
people["Fare"] = four["Fare"].to_list()
people.loc["Heikkinen"]
people.mean().round(2)
people.T
records = df[["Sex", "Age"]].head(2).to_dict("records")
print(records)
pd.DataFrame(records)
df.dtypes.value_counts()
Common mistake: columns of different lengths
ages3 = df["Age"].head(3).to_list()
classes2 = df["Pclass"].head(2).to_list()
pd.DataFrame({"Age": ages3, "Class": classes2})
classes3 = df["Pclass"].head(3).to_list()
pd.DataFrame({"Age": ages3, "Class": classes3})
Your turn: build a DataFrame from two Series.
two = pd.DataFrame({"Name": df["Name"], "Fare": df["Fare"]})
print(two.shape)
two.head(3)
Labels line up: index alignment#
Maths between Series matches labels, not positions.
a = df.loc[df["Survived"] == 1, "Embarked"].value_counts()
b = df.loc[df["Survived"] == 0, "Embarked"].value_counts().sort_index()
print(a.index.tolist(), b.index.tolist())
a + b
c = b.head(2)
print(a + c)
print(a.add(c, fill_value=0))
alive = df.loc[df["Survived"] == 1, "Pclass"].value_counts()
total = df["Pclass"].value_counts().sort_index()
print(alive.index.tolist(), total.index.tolist())
total.reindex([1, 2, 3, 4])
Quick check: the orders differ. Is
alive / totalstill right?
(alive / total).round(2)
Quick check: what if we divide the raw
.valuesinstead?
raw = alive.values / total.values
print(raw.round(2))
Column maths: build new columns from old ones#
Maths on a whole column at once, no loops.
df["FamilySize"] = df["SibSp"] + df["Parch"] + 1
df[["Name", "SibSp", "Parch", "FamilySize"]].head(3)
df["FamilySize"].value_counts().sort_index()
df["IsAlone"] = (df["FamilySize"] == 1).astype(int)
print(df["IsAlone"].sum(), "travelled alone")
Your turn: add a 0/1 column
IsChildfor ages under 16.
df["IsChild"] = (df["Age"] < 16).astype(int)
print(df["IsChild"].sum(), "children")
young_adult = df["Age"].between(18, 30)
print(young_adult.sum(), "passengers aged 18 to 30")
fare_each = df["Fare"] / df["FamilySize"]
print(type(fare_each))
print(fare_each.round(2).head(3).tolist())
Common mistake: turning a column with gaps into whole numbers
df["Age"].astype(int)
whole = df["Age"].round().astype("Int64")
print(whole.head(6).tolist())
print(whole.dtype, whole.isna().sum())
summary = df["Fare"].describe()
summary.to_frame().T.round(2)
Mini-project: did travelling alone help?#
Use the IsAlone column we built, and compare fairly.
rates = df.groupby("IsAlone")["Survived"].agg(["count", "mean"])
rates.round(2)
by_size = df.groupby("FamilySize")["Survived"].mean()
by_size.round(2)
pd.crosstab(df["Sex"], df["IsAlone"], normalize="columns").round(2)
fair = df.pivot_table(values="Survived", index="Sex",
columns="IsAlone", aggfunc="mean")
fair.round(2)
by_class = df.pivot_table(values="Survived", index="Pclass",
columns="IsAlone", aggfunc="mean")
by_class.round(2)
import matplotlib.pyplot as plt
ax = fair.plot(kind="bar", rot=0)
ax.set_title("Survival by sex: with family vs alone")
ax.set_xlabel("Sex")
ax.set_ylabel("Share who survived")
ax.legend(["With family", "Alone"])
plt.show()
Finding: alone looks deadly overall (0.30 vs 0.51), but most loners were men. Within each sex the gap shrinks; alone women even did better. Within each class, though, loners still fared worse.
Recap#
- A Series is values plus an index;
.values,.index,.dtype locmeans label,ilocmeans position- A DataFrame is Series sharing one index;
df["col"]is a Series - Maths between Series aligns on labels; missing labels give NaN
- Build columns with whole-column maths, like
SibSp + Parch + 1
Challenges#
- Make a Series of fares indexed by
Name. Who paid the most? - Compare survival for family sizes 1, 2-4 and 5+ with
pd.cut. - Divide two value_counts Series to get survival by port.
What's next#
Lesson 04: Creating and Inspecting DataFrames. Build tables from lists, dicts and files, then inspect them fast.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.
Related lessons
28 minIntroduction to Pandas DataFrame: Essential Operations and Attributes Explained
36 minMastering DataFrame Merges in Pandas: Inner, Outer, Left & Right Joins Explained
24 minComprehensive Guide to Pandas Series and DataFrames for Data Analysis in Python
26 min