Lesson 39 · Pandas Full Course
Handle Time Zones in Pandas: tz_localize, tz_convert and Daylight Saving | Pandas Course #39
In this lesson you'll be able to: Tell naive timestamps from time-zone-aware ones Attach a zone with tzlocalize and switch with tzconvert Survive daylight…
- CoursePandas Full Course
- Lesson39 of 65
- FormatJupyter notebook · 43 code cells
- Data1 dataset
What you'll learn
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
📓 Full notebook
Download .ipynbPandas 39: Handling Time Zones and Offsets#
In this lesson you'll be able to:
- Tell naive timestamps from time-zone-aware ones
- Attach a zone with tz_localize and switch with tz_convert
- Survive daylight saving: missing and repeated hours
- Shift dates with Timedelta, DateOffset and business days
- Line up data from several zones on one clock Data: a month of New York taxi trips, March 2019
Warm-up: trips by hour#
Lesson 38 grouped data by hours and days. Same taxis, same idea.
import pandas as pd
taxis = pd.read_csv("data/pandas/taxis.csv",
parse_dates=["pickup", "dropoff"])
by_hour = taxis.groupby(taxis["pickup"].dt.hour).size()
print(by_hour.idxmax(), by_hour.max())
daily = taxis.groupby(taxis["pickup"].dt.date).size()
print(daily.idxmax(), daily.max())
Naive and aware timestamps#
Problem: a time with no zone could be anywhere on Earth.
first = taxis["pickup"].iloc[0]
print(first)
print(first.tz)
Quick check: does the whole column have a time zone?
print(taxis["pickup"].dt.tz)
ny = taxis["pickup"].dt.tz_localize("America/New_York")
ny.head(3)
print(ny.dt.tz)
print(ny.iloc[0].strftime("%Z %z"))
print(ny.iloc[1].strftime("%Z %z"))
Common mistake: comparing aware times with a naive one
ny >= pd.Timestamp("2019-03-25")
cut = pd.Timestamp("2019-03-25", tz="America/New_York")
late = ny >= cut
print(late.sum())
Convert between zones with tz_convert#
Problem: show the same moment on another city's clock.
taxis["pickup_ny"] = ny
taxis["pickup_utc"] = ny.dt.tz_convert("UTC")
taxis[["pickup_ny", "pickup_utc"]].head(3)
london = ny.dt.tz_convert("Europe/London")
tokyo = ny.dt.tz_convert("Asia/Tokyo")
print(london.iloc[0])
print(tokyo.iloc[0])
rush = taxis["pickup"].dt.hour == 18
tokyo[rush].dt.hour.value_counts()
Quick check: are the New York and UTC columns equal?
same = taxis["pickup_ny"] == taxis["pickup_utc"]
print(same.all())
Common mistake: localising a column that already has a zone
taxis["pickup_utc"].dt.tz_localize("Europe/London")
taxis["pickup_utc"].dt.tz_convert("Europe/London").head(3)
wall = taxis["pickup_utc"].dt.tz_localize(None)
print(wall.dt.tz)
print(wall.iloc[0])
Your turn#
Make the drop-off times aware in New York, then show them in Tokyo.
drop = taxis["dropoff"].dt.tz_localize("America/New_York")
drop.dt.tz_convert("Asia/Tokyo").head(3)
Daylight saving: missing and repeated hours#
Problem: clocks jump, so some local times never happen.
Quick check: any pickups between 2 and 3 a.m. on 10 March?
night = taxis["pickup"].between("2019-03-10 01:00",
"2019-03-10 04:00")
taxis.loc[night, "pickup"].sort_values()
offset = taxis["pickup_ny"].dt.strftime("%z")
offset.value_counts()
Common mistake: localising a time that never existed
pd.Timestamp("2019-03-10 02:30").tz_localize("America/New_York")
gap = pd.Timestamp("2019-03-10 02:30")
print(gap.tz_localize("America/New_York",
nonexistent="shift_forward"))
Quick check: in November the clocks go back. Which 01:30 is it?
autumn = pd.Timestamp("2019-11-03 01:30")
print(autumn.tz_localize("America/New_York", ambiguous=True))
print(autumn.tz_localize("America/New_York", ambiguous=False))
days = pd.date_range("2019-03-09", periods=3, freq="D",
tz="America/New_York")
print(days.to_series().diff())
Your turn#
How many hours ahead of New York was London for each trip?
ldn = ny.dt.tz_convert("Europe/London").dt.tz_localize(None)
ahead = ldn - ny.dt.tz_localize(None)
ahead.value_counts()
Shift dates with offsets#
Problem: 'one day later' can mean 24 hours or the same time tomorrow.
Quick check: noon on 9 March, plus one day. Noon on the 10th?
sat = pd.Timestamp("2019-03-09 12:00", tz="America/New_York")
print(sat + pd.Timedelta(days=1))
print(sat + pd.DateOffset(days=1))
fri = taxis.loc[taxis["pickup"].dt.dayofweek == 4, "pickup"]
ride = fri.iloc[0]
print(ride.day_name(), ride)
print(ride + pd.offsets.BDay(1))
end = taxis["pickup_ny"] + pd.offsets.MonthEnd(0)
nxt = taxis["pickup_ny"] + pd.offsets.MonthBegin(1)
print(end.iloc[0], nxt.iloc[0], sep="\n")
monday = taxis["pickup"] + pd.offsets.Week(weekday=0)
print(monday.head(3))
print(monday.dt.day_name().value_counts())
start = taxis["pickup_ny"].dt.floor("D")
(taxis["pickup_ny"] - start).head(3)
Your turn#
Invoices are due 5 business days after a trip. When is the first one due?
due = taxis["pickup"] + pd.offsets.BDay(5)
pd.DataFrame({"trip": taxis["pickup"], "due": due}).head(3)
Line up data from several zones#
Problem: records from different cities must share one clock.
dep = pd.Timestamp("2019-03-25 20:00", tz="America/New_York")
arr = pd.Timestamp("2019-03-26 07:05", tz="Europe/London")
print(arr - dep)
both = pd.Series([dep, arr])
print(both.dtype)
print(pd.to_datetime(both, utc=True))
Common mistake: parsing text with mixed UTC offsets
stamps = ["2019-03-25 20:00-04:00", "2019-03-26 07:05+00:00"]
pd.to_datetime(stamps)
utc = pd.to_datetime(stamps, utc=True)
print(utc)
print(utc.tz_convert("Europe/London"))
by_utc = taxis.groupby(taxis["pickup_utc"].dt.hour).size()
print(by_utc.idxmax(), by_utc.max())
Your turn#
The flight lands at 07:05 in London. What time is it in New York?
print(arr.tz_convert("America/New_York"))
Mini-project: would a UTC report change our story?#
Plan: count trips per day on both clocks, measure the shift, chart it.
local = taxis.groupby(taxis["pickup_ny"].dt.date).size()
world = taxis.groupby(taxis["pickup_utc"].dt.date).size()
print(local.idxmax(), local.max())
print(world.idxmax(), world.max())
ny_day = taxis["pickup_ny"].dt.date
moved = ny_day != taxis["pickup_utc"].dt.date
print(moved.sum(), round(moved.mean(), 3))
hours = taxis.loc[moved, "pickup_ny"].dt.hour
print(hours.min(), hours.max())
hourly = pd.DataFrame({"New York": by_hour, "UTC": by_utc})
hourly.loc[17:23]
both_days = pd.DataFrame({"New York": local, "UTC": world})
both_days.tail(3)
change = (world - local).dropna()
print(change.idxmin(), change.min())
print(change.idxmax(), change.max())
import matplotlib.pyplot as plt
ax = both_days.plot(figsize=(9, 4), marker="o", markersize=3)
ax.set_title("Taxi trips per day: New York vs UTC calendar")
ax.set_xlabel("Date")
ax.set_ylabel("Trips")
plt.show()
share = moved.mean()
print(f"{share:.0%} of trips change date in UTC; "
f"rush hour moves from {by_hour.idxmax()}:00 "
f"to {by_utc.idxmax()}:00.")
Recap#
- Naive times have no zone; aware times know theirs
- tz_localize attaches a zone once; tz_convert switches clocks
- Daylight saving skips and repeats hours: use nonexistent and ambiguous
- Timedelta adds exact time; DateOffset and BDay follow the calendar
- Mixed zones? Parse with utc=True and convert at the end
Challenges#
- Convert the drop-offs to UTC and find the busiest UTC day.
- Count taxi trips per hour on a Tokyo clock.
- Add 3 business days to every Friday pickup. Which weekday is it?
What's next#
Lesson 40: Quick Visualizations using Pandas plot() Turn any Series or DataFrame into a chart in one line.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.



