Mathew K AnalyticsLesson 2 · Pandas Full Course
Install Pandas the Right Way and Fix Setup Errors for Good | Pandas Course #2
In this lesson you'll be able to: Check which Python and pandas versions you are running Install, upgrade and verify pandas and its helper packages Point…
- CoursePandas Full Course
- Lesson2 of 65
- Video23 min
- FormatJupyter notebook · 42 code cells
- Data3 datasets
What you'll learn
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
- titanic.csv preview58.9 KB
- penguins.csv preview13.2 KB
- online_retail.csv.gz preview7.1 MB
📓 Full notebook
Download .ipynbPandas 02: Installing Pandas and Setting Up the Environment#
In this lesson you'll be able to:
- Check which Python and pandas versions you are running
- Install, upgrade and verify pandas and its helper packages
- Point pandas at your data files without path errors
- Tune display options and make results repeatable Data: the Titanic list, then every file in data/pandas
Warm-up: is everything still working?#
Lesson 01 loaded the Titanic list and grouped it by class.
import pandas as pd
df = pd.read_csv("data/pandas/titanic.csv")
print(df.shape)
df.groupby("Pclass")["Survived"].mean().round(2)
Check what you have: Python and pandas versions#
Most setup problems are version problems. Check before you install.
import platform
print("Python", platform.python_version())
print("pandas", pd.__version__)
ver = tuple(int(p) for p in pd.__version__.split(".")[:2])
print("pandas 2.2 or newer:", ver >= (2, 2))
from importlib.metadata import version
core = ["numpy", "matplotlib", "seaborn"]
for name in core:
print(name, version(name))
Quick check: is this pandas 2 or newer?
major = int(pd.__version__.split(".")[0])
print("major version at least 2:", major >= 2)
Why not just compare the text, like
"10.0" > "2.0"?
print("10.0" > "2.0")
print(10 > 2)
Why the version matters#
pandas 2 and 3 changed defaults: text type, copy rules, old APIs. Code from old tutorials may warn or fail. Match your version.
print(df["Name"].dtype)
print(df["Name"].str.len().max())
Install, upgrade and verify#
Run these in a terminal, not inside the notebook:
python -m pip install --upgrade pandas or conda install pandas
Optional extras#
Excel needs openpyxl, fast file formats need pyarrow.
from importlib.util import find_spec
extras = ["openpyxl", "pyarrow", "sklearn", "plotly"]
for name in extras:
found = find_spec(name) is not None
print(f"{name:<10} {found}")
Common mistake: importing
pandainstead ofpandas
import panda as pd
import pandas as pd
print("pandas", pd.__version__, "is ready")
Your turn: is seaborn importable, and which version is it?
print(find_spec("seaborn") is not None)
import seaborn as sns
print(sns.__version__)
Tip: if
pandasitself is missing, the error is the same. Install it into the Python your notebook uses:%pip install pandas
Where to write your code#
Jupyter: run jupyter lab in a terminal, then open a notebook.
Each cell runs on its own and keeps its results for the next.
Find your files: the working folder and relative paths#
Relative paths start from the folder your code runs in.
from pathlib import Path
data_dir = Path("data") / "pandas"
print(data_dir, data_dir.exists())
files = sorted(p.name for p in data_dir.glob("*.csv*"))
print("titanic.csv" in files, "penguins.csv" in files)
titanic_path = data_dir / "titanic.csv"
print(titanic_path.name)
print(titanic_path.stem, titanic_path.suffix)
print(titanic_path.parent.name)
Common mistake: a path that skips a folder
path = "pandas/titanic.csv"
pd.read_csv(path)
path = data_dir / "titanic.csv"
print(path.exists())
pd.read_csv(path).shape
retail_path = data_dir / "online_retail.csv.gz"
peek = pd.read_csv(retail_path, nrows=3)
peek[["InvoiceNo", "Description", "Quantity", "Country"]]
Quick check: can we catch a typo before read_csv crashes?
guess = data_dir / "titanc.csv"
if guess.exists():
print(pd.read_csv(guess).shape)
else:
print("Not found:", guess.name)
cols = ["Name", "Sex", "Age", "Survived"]
slim = pd.read_csv(titanic_path, usecols=cols)
print(slim.shape)
print(slim.memory_usage(deep=True).sum() < df.memory_usage(deep=True).sum())
Your turn: does
data/pandas/penguins.csvexist? How many rows?
peng_path = data_dir / "penguins.csv"
print(peng_path.exists())
print(len(pd.read_csv(peng_path)), "penguins")
Make output readable: display options#
Options set how many rows, columns and decimals you see.
for opt in ["max_rows", "max_columns", "precision"]:
print(opt, pd.get_option("display." + opt))
pd.set_option("display.precision", 2)
df[["Age", "Fare"]].describe()
with pd.option_context("display.max_rows", 6):
print(df["Name"].head(10))
print(pd.get_option("display.max_rows"))
with pd.option_context("display.max_colwidth", 20):
print(df["Name"].head(3))
totals = df.groupby("Pclass")["Fare"].sum()
with pd.option_context("display.float_format", "{:,.2f}".format):
print(totals)
Common mistake: a misspelt option name
pd.set_option("display.precison", 2)
pd.set_option("display.precision", 3)
print(pd.get_option("display.precision"))
Your turn: show
describe()of Fare with 1 decimal place, temporarily.
with pd.option_context("display.precision", 1):
print(df["Fare"].describe())
pd.reset_option("display.precision")
print(pd.get_option("display.precision"))
Make your work repeatable#
Fix random seeds, save outputs to one folder, record versions.
a = df.sample(3, random_state=1)
b = df.sample(3, random_state=2)
print(a["PassengerId"].tolist(), b["PassengerId"].tolist())
first = df.sample(3, random_state=42)
print(first[["PassengerId", "Name", "Pclass"]])
Quick check: run it again with
random_state=42. Same passengers?
b = df.sample(3, random_state=42)
print(b["PassengerId"].tolist(), first.equals(b))
import os
os.makedirs("output/pandas", exist_ok=True)
libs = ["pandas", "numpy", "matplotlib", "seaborn"]
rows = [{"package": n, "version": version(n)} for n in libs]
versions = pd.DataFrame(rows)
versions
Common mistake: asking for
sklearnby its import name
version("sklearn")
print(version("scikit-learn"))
ver_path = "output/pandas/l02_versions.csv"
versions.to_csv(ver_path, index=False)
pd.read_csv(ver_path)
Mini-project: health check every dataset#
Can this setup load all the files the series needs?
report = []
for f in sorted(data_dir.glob("*.csv*")):
d = pd.read_csv(f, low_memory=False)
report.append({"file": f.name, "rows": len(d),
"cols": d.shape[1],
"missing": int(d.isna().sum().sum())})
health = pd.DataFrame(report)
print(len(health), "files loaded")
health.sort_values("rows", ascending=False).head(5)
import matplotlib.pyplot as plt
sizes = health.set_index("file")["rows"].sort_values()
ax = sizes.plot(kind="barh", logx=True, color="steelblue")
ax.set_title("Rows in each dataset")
ax.set_xlabel("Rows (log scale)")
ax.set_ylabel("File")
plt.show()
complete = health[health["missing"] == 0]
print(complete["file"].tolist())
cells = health["rows"] * health["cols"]
health["pct_missing"] = (health["missing"] / cells * 100).round(1)
health.sort_values("pct_missing", ascending=False).head(4)
health.loc[health["file"] == "titanic.csv"]
health_path = "output/pandas/l02_data_health.csv"
health.to_csv(health_path, index=False)
pd.read_csv(health_path).equals(health)
Finding: every file loads, from flights (144 rows) to online retail (541,909). Diamonds and iris are complete; others have gaps to clean later.
Recap#
pd.__version__andimportlib.metadatareport versions- Install or upgrade in a terminal with pip or conda
ModuleNotFoundErrormeans a misspelt or missing package- Build paths with
pathliband check them with.exists() - Use
option_context,random_stateand an output folder
Challenges#
- Add a
mbcolumn to the health report usingmemory_usage. - Set
display.max_columnsto 5 and show the superstore head. - Add scikit-learn and pyarrow to the saved versions file.
What's next#
Lesson 03: Understanding Series and DataFrames. What a Series is, how DataFrames hold them, and why labels matter.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.
Related lessons
26 minData Mining with Python: pandas & scikit-learn Crash Course
23 minClean Missing & Dirty Operations Data in pandas
36 minComprehensive Pandas Tutorial in Python: Data Analysis from Beginner to Advanced
16 min