Mathew K AnalyticsLesson 4 · Pandas Full Course
Create and Inspect a Pandas DataFrame Like a Pro: info, describe, dtypes | Pandas Course #4
In this lesson you'll be able to: Build a DataFrame from a dictionary, a list of rows or an array Load a real CSV file and take a first look at it Read a…
- CoursePandas Full Course
- Lesson4 of 65
- Video24 min
- FormatJupyter notebook · 44 code cells
- Data1 dataset
What you'll learn
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
- penguins.csv preview13.2 KB
📓 Full notebook
Download .ipynbPandas 04: Creating and Inspecting DataFrames#
In this lesson you'll be able to:
- Build a DataFrame from a dictionary, a list of rows or an array
- Load a real CSV file and take a first look at it
- Read a table's structure with info, dtypes, columns and index
- Summarise numbers with describe and categories with value_counts
- Audit a dataset for gaps before you trust it Dataset: Palmer penguins (data/pandas/penguins.csv)
Warm-up: a Series is one labelled column#
Lesson 03 met Series and DataFrames. Load the file, build a Series.
import pandas as pd
df = pd.read_csv("data/pandas/penguins.csv")
weights = df["body_mass_g"].head(3).to_list()
mass = pd.Series(weights, name="body_mass_g")
print(mass)
print(mass.mean())
print(mass.index)
print(mass.dtype)
Build a DataFrame from a dictionary#
Problem: turn a few values you already have into a proper table.
pick = df.iloc[[0, 1, 277, 152]]
data = {
"species": pick["species"].to_list(),
"island": pick["island"].to_list(),
"body_mass_g": pick["body_mass_g"].to_list(),
}
small = pd.DataFrame(data)
small
tags = ["p1", "p2", "p3", "p4"]
tagged = pd.DataFrame(data, index=tags)
tagged
Common mistake: lists of different lengths
broken = {"species": df["species"].head(2).to_list(),
"body_mass_g": df["body_mass_g"].head(3).to_list()}
pd.DataFrame(broken)
fixed = {"species": df["species"].head(3).to_list(),
"body_mass_g": df["body_mass_g"].head(3).to_list()}
pd.DataFrame(fixed)
Quick check: how many rows and columns does small have?
print(small.shape)
print(len(small))
Other ways in: rows, arrays and chosen columns#
Problem: your data arrives as rows or as a NumPy array, not as columns.
bits = df[["species", "bill_length_mm"]].iloc[2:4]
rows = bits.to_dict("records")
print(rows)
pd.DataFrame(rows)
cols = ["bill_length_mm", "bill_depth_mm"]
arr = df[cols].head(3).to_numpy()
print(arr)
bills = pd.DataFrame(arr, columns=cols)
bills
pd.DataFrame(data, columns=["body_mass_g", "species"])
Your turn#
Build a two-column table of islands and penguin counts from df["island"].value_counts().
counts = df["island"].value_counts()
islands = pd.DataFrame({
"island": counts.index.to_list(),
"penguins": counts.to_list(),
})
islands
print(islands["penguins"].sum())
Take a first look at a real file#
Problem: open a file of hundreds of rows without drowning in it.
print(type(df))
print(df.shape)
df.head()
df.tail(3)
df.sample(4, random_state=42)
cols = ["species", "island", "body_mass_g", "sex"]
df[cols].head()
Common mistake: the wrong file path
path = "data/penguins.csv"
pd.read_csv(path)
import os
print(sorted(os.listdir("data/pandas"))[:6])
print(os.path.exists("data/pandas/penguins.csv"))
Read the structure: info, dtypes, columns and index#
Problem: know every column's type and how many values are missing.
df.info()
df.dtypes
print(df.columns.tolist())
print(df.index)
Quick check: which column has the most missing values?
df.isna().sum()
missing_pct = df.isna().mean() * 100
missing_pct.round(1)
mem = df.memory_usage(deep=True)
print(mem.sum())
Your turn#
List only the number columns. Hint: select_dtypes.
nums = df.select_dtypes(include="number")
print(nums.columns.tolist())
print(nums.shape)
Common mistake: calling an attribute like a method
df.shape()
print(df.shape)
print(df.ndim, df.size)
Summarise numbers and categories#
Problem: describe a whole table in two or three lines of code.
df.describe().round(1)
df.describe(exclude="number")
df["species"].value_counts()
df["island"].value_counts(normalize=True).round(3)
df["sex"].value_counts(dropna=False)
df.nunique()
Quick check: which three penguins are the heaviest, and what species are they?
heavy = df.nlargest(3, "body_mass_g")
heavy[cols]
Your turn#
Find the shortest and longest flipper, and the average, rounded to 1 place.
flip = df["flipper_length_mm"]
print(flip.min(), flip.max())
print(round(flip.mean(), 1))
import matplotlib.pyplot as plt
ax = flip.plot(kind="hist", bins=20, figsize=(7, 4))
ax.set_title("Flipper length of 342 penguins")
ax.set_xlabel("Flipper length (mm)")
ax.set_ylabel("Number of penguins")
plt.show()
Mini-project: is this dataset ready to analyse?#
Plan: find incomplete rows, see who they are, drop them, compare.
gaps = df[df.isna().any(axis=1)]
print(len(gaps))
gaps[cols]
gaps["species"].value_counts()
complete = df.dropna()
print(df.shape, complete.shape)
print(round(len(complete) / len(df) * 100, 1))
print(complete.isna().sum().sum())
print(complete.index[:12].tolist())
audit = pd.DataFrame({
"before": df["species"].value_counts(),
"after": complete["species"].value_counts(),
})
audit["kept_pct"] = (audit["after"] / audit["before"] * 100)
audit.round(1)
ax = audit[["before", "after"]].plot(kind="bar", rot=0,
figsize=(7, 4))
ax.set_title("Penguins per species, before and after dropna")
ax.set_xlabel("Species")
ax.set_ylabel("Number of penguins")
plt.show()
lost = len(df) - len(complete)
worst = audit["kept_pct"].idxmin()
print(f"Dropped {lost} rows; {worst} lost the most.")
Recap#
pd.DataFramebuilds tables from dicts, lists of rows or arrays- Every column list must have the same length
- head, tail and sample peek; info and dtypes show structure
isna().sum()counts gaps; shape and size need no brackets- describe and value_counts summarise numbers and categories
Challenges#
- Build a DataFrame of 3 penguins from a list of dicts, one with a missing sex.
- Use
value_countsto count penguins per island and sex together. - Load
data/pandas/tips.csvand audit it the same way.
What's next#
Lesson 05: Basic DataFrame Operations and Attributes Add, remove and calculate columns, and use every key attribute.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.
Related lessons
26 minData Mining with Python: pandas & scikit-learn Crash Course
21 minHow to Create an Interactive KPI Dashboard Using Python and Pandas
28 minIntroduction to Pandas DataFrame: Essential Operations and Attributes Explained
17 min