Mathew K AnalyticsLesson 8 · Pandas Full Course
Pandas loc vs iloc: Select Exactly the Rows and Columns You Need | Pandas Course #8
In this lesson you'll be able to: Pick one column or several with square brackets Select rows and columns by position with iloc Select rows and columns by…
- CoursePandas Full Course
- Lesson8 of 65
- FormatJupyter notebook · 44 code cells
- Data1 dataset
What you'll learn
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
- titanic.csv preview58.9 KB
📓 Full notebook
Download .ipynbPandas 08: Selecting Rows, Columns, and Subsets of Data#
In this lesson you'll be able to:
- Pick one column or several with square brackets
- Select rows and columns by position with iloc
- Select rows and columns by label with loc
- Read and change single values with at and loc
- Grab handy subsets: samples, top rows and column types Dataset: the Titanic passenger list (data/pandas/titanic.csv)
Warm-up: load and check#
Lesson 07 read files into pandas. Load today's file the same way.
import pandas as pd
df = pd.read_csv("data/pandas/titanic.csv")
print(df.shape)
print(df.columns.tolist())
df.head(3)
Pick columns with square brackets#
Problem: work with just the columns a question needs.
ages = df["Age"]
print(type(ages))
ages.head()
cols = ["Name", "Age", "Fare"]
people = df[cols]
print(type(people))
people.head()
Quick check: df["Age"] and df[["Age"]] look alike. What's the difference?
print(df["Age"].shape)
print(df[["Age"]].shape)
Common mistake: listing two columns without the inner brackets
df["Name", "Age"]
df[["Name", "Age"]].head(3)
numbers = df.select_dtypes(include="number")
print(numbers.columns.tolist())
print(df.filter(like="Sib").columns.tolist())
print(df.filter(regex="^S").columns.tolist())
Select by position with iloc#
Problem: grab rows and columns by where they sit, like row 0.
df.iloc[0]
df.iloc[-2:]
df.iloc[0:3, [3, 4, 5]]
Quick check: how many rows does df.iloc[10:20] return?
chunk = df.iloc[10:20]
print(len(chunk))
print(chunk.index[0], chunk.index[-1])
Common mistake: asking for a position past the end
df.iloc[891]
print(len(df))
df.iloc[len(df) - 1, [0, 3]]
Your turn#
Use iloc to show rows 100 to 104 and the last three columns.
df.iloc[100:105, -3:]
Select by label with loc#
Problem: ask for rows and columns by their names, not positions.
df.loc[2:5, ["Name", "Pclass"]]
Quick check: iloc[2:5] gave 3 rows. Why did loc[2:5] give 4?
print(len(df.iloc[2:5]), len(df.loc[2:5]))
df.loc[0, "Name":"Age"]
pax = df.set_index("PassengerId")
pax.loc[[1, 2, 3], ["Name", "Age", "Fare"]]
Common mistake: using a position where loc expects a label
pax.loc[0, "Name"]
print(pax.iloc[0]["Name"])
print(pax.loc[1, "Name"])
by_name = df.set_index("Name")
by_name.loc["Andersson, Mr. Anders Johan", ["Age", "Pclass"]]
Your turn#
Using pax, show the Name, Sex and Survived of passengers 10 to 14.
pax.loc[10:14, ["Name", "Sex", "Survived"]]
Select inside a single column#
Problem: pick values out of a Series, not a whole table.
fares = pax["Fare"]
print(fares.loc[1])
print(fares.iloc[-1])
fares.loc[[62, 830]]
Common mistake: plain square brackets with a number on a Series
fares[0]
print(fares.iloc[0])
print(fares.loc[1])
Read and change single values#
Problem: look up one cell quickly, and correct it safely.
print(pax.at[1, "Fare"])
print(pax.iat[0, 8])
missing = [62, 830]
pax.loc[missing, ["Name", "Embarked"]]
pax.loc[missing, "Embarked"] = "S"
pax.loc[missing, ["Name", "Embarked"]]
Quick check: did that change touch the original df too?
print(df["Embarked"].isna().sum())
print(pax["Embarked"].isna().sum())
Your turn#
Passenger 259 paid one of the top fares. Use at to print her name, and whether she survived.
print(pax.at[259, "Name"])
print(pax.at[259, "Survived"])
Grab handy subsets#
Problem: get a quick, fair look at a big table.
df.sample(n=4, random_state=42)[["Name", "Pclass", "Age"]]
tiny = df.sample(frac=0.01, random_state=42)
print(tiny.shape)
df.nlargest(3, "Fare")[["Name", "Pclass", "Fare"]]
df.nsmallest(3, "Age")[["Name", "Age", "Survived"]]
every_100 = df.iloc[::100, [3, 9]]
every_100
Your turn#
Show the Name, Age and Fare of the 5 oldest passengers.
oldest = df.nlargest(5, "Age")
oldest[["Name", "Age", "Fare"]]
Mini-project: richest versus cheapest tickets#
Question: did the 50 priciest tickets survive more than the 50 cheapest?
Plan: select both groups, compare survival and class, then chart.
rich = df.nlargest(50, "Fare")
cheap = df.nsmallest(50, "Fare")
print(rich["Fare"].min(), cheap["Fare"].max())
rich_rate = rich["Survived"].mean()
cheap_rate = cheap["Survived"].mean()
print(rich_rate, cheap_rate)
print(rich["Pclass"].value_counts().to_dict())
print(cheap["Pclass"].value_counts().to_dict())
odd = cheap.loc[cheap["Pclass"] == 1, ["Name", "Fare", "Survived"]]
odd
print(rich["Age"].mean().round(1), cheap["Age"].mean().round(1))
print(rich["Sex"].value_counts().to_dict())
print(cheap["Sex"].value_counts().to_dict())
import matplotlib.pyplot as plt
rates = pd.Series({"Top 50 fares": rich_rate,
"Bottom 50 fares": cheap_rate})
ax = rates.plot(kind="bar", rot=0, figsize=(6, 4))
ax.set_title("Titanic survival: priciest vs cheapest tickets")
ax.set_xlabel("Ticket group")
ax.set_ylabel("Share who survived")
plt.show()
gap = rich_rate / cheap_rate
print(f"Top 50: {rich_rate:.0%}, bottom 50: {cheap_rate:.0%}")
print(f"About {gap:.1f} times the survival rate")
Recap#
df["col"]gives a Series;df[["a", "b"]]gives a DataFrame- iloc uses positions from 0, and its slices exclude the end
- loc uses labels, and its slices include both ends
- On a Series, prefer .loc and .iloc to plain [] with numbers
- at and iat read one value;
loc[rows, col] = valuechanges data - sample, nlargest and nsmallest give quick, useful subsets
Challenges#
- Use iloc to show the last 5 rows and the first 4 columns.
- Index by Name and look up the Fare and Cabin of two passengers.
- Compare the mean Age of the 100 priciest and 100 cheapest tickets.
What's next#
Lesson 09: Conditional Selection and Boolean Indexing Keep only the rows that meet a rule, like Age < 18 and Pclass == 1.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.
Related lessons
12 minMastering Pandas DataFrames: Applying Functions Across Rows & Columns for Data Analysis
26 minData Mining with Python: pandas & scikit-learn Crash Course
17 minMaster Filtering Rows and Columns in Pandas Using Query and isin for Data Analysis
17 min