Mathew K Analytics

Lesson 40 · Pandas Full Course

Make Charts Straight from Pandas with .plot() (Line, Bar, Hist, Scatter) | Pandas Course #40

In this lesson you'll be able to: Turn a Series or DataFrame into a chart in one line Choose the chart with kind=: line, bar, hist, box, scatter, pie Add…

What you'll learn

Datasets used in this lesson

Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.

📓 Full notebook

Download .ipynb

Pandas 40: Quick Visualizations using Pandas plot()#

In this lesson you'll be able to:

  • Turn a Series or DataFrame into a chart in one line
  • Choose the chart with kind=: line, bar, hist, box, scatter, pie
  • Add titles, labels, size and colour
  • Plot several columns, side by side or in panels
  • Combine charts in one figure and save them as files Data: New York taxi trips and restaurant tips

Warm-up: trips per hour, on the right clock#

Lesson 39 attached the New York time zone to every pickup.

import pandas as pd
taxis = pd.read_csv("data/pandas/taxis.csv",
                    parse_dates=["pickup", "dropoff"])
ny = taxis["pickup"].dt.tz_localize("America/New_York")
by_hour = taxis.groupby(ny.dt.hour).size()
by_hour.head(4)
pickup
0    205
1    110
2    101
3     67
dtype: int64

Your first chart in one line#

Problem: 24 numbers are hard to read. A picture isn't.

import matplotlib.pyplot as plt
by_hour.plot()
plt.show()
No description has been provided for this image

Quick check: which hour is quietest? Read it off the chart.

print(by_hour.idxmin(), by_hour.min())
print(by_hour.idxmax(), by_hour.max())
5 51
18 417
by_hour.plot(title="Taxi pickups by hour, March 2019",
             xlabel="Hour of day (New York)",
             ylabel="Trips", figsize=(8, 4))
plt.show()
No description has been provided for this image

Common mistake: plotting a text column

taxis["payment"].plot()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[6], line 1
----> 1 taxis["payment"].plot()
      2 plt.show()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\plotting\_core.py:1185, in PlotAccessor.__call__(self, *args, **kwargs)
   1182             label_name = label_kw or data.columns
   1183             data.columns = label_name
-> 1185 return plot_backend.plot(data, kind=kind, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\plotting\_matplotlib\__init__.py:71, in plot(data, kind, **kwargs)
     69         kwargs["ax"] = getattr(ax, "left_ax", ax)
     70 plot_obj = PLOT_CLASSES[kind](data, **kwargs)
---> 71 plot_obj.generate()
     72 plt.draw_if_interactive()
     73 return plot_obj.result

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\plotting\_matplotlib\core.py:518, in MPLPlot.generate(self)
    516 @final
    517 def generate(self) -> None:
--> 518     self._compute_plot_data()
    519     fig = self.fig
    520     self._make_plot(fig)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\plotting\_matplotlib\core.py:718, in MPLPlot._compute_plot_data(self)
    716 # no non-numeric frames or series allowed
    717 if is_empty:
--> 718     raise TypeError("no numeric data to plot")
    720 self.data = numeric_data.apply(type(self)._convert_to_ndarray)

TypeError: no numeric data to plot
pay = taxis["payment"].value_counts()
print(pay)
pay.plot(kind="bar", title="How taxi riders pay", rot=0)
plt.show()
payment
credit card    4577
cash           1812
Name: count, dtype: int64
No description has been provided for this image
by_hour.plot(marker="o", grid=True, xticks=range(0, 24, 3),
             title="Taxi pickups by hour", xlabel="Hour",
             ylabel="Trips", figsize=(8, 4))
plt.show()
print(by_hour.loc[[7, 8, 9]])
No description has been provided for this image
pickup
7    221
8    313
9    319
dtype: int64

Your turn#

Plot taxi trips per weekday as a bar chart, Monday first.

week = ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday",
        "Saturday", "Sunday"]
per_day = ny.dt.day_name().value_counts().reindex(week)
per_day.plot(kind="bar", rot=45, title="Taxi trips by weekday",
             ylabel="Trips")
plt.show()
print(per_day.idxmax(), per_day.max())
No description has been provided for this image
Friday 1115

Pick the chart with kind=#

Problem: different questions need different chart shapes.

  • line: trends | bar, barh: compare groups
  • hist, box: spread of values | scatter: two numbers together
  • pie, area, hexbin: shares, totals, dense scatters
tips = pd.read_csv("data/pandas/tips.csv")
print(tips.shape)
tips.head(3)
(244, 7)
total_bill tip sex smoker day time size
0 16.99 1.01 Female No Sun Dinner 2
1 10.34 1.66 Male No Sun Dinner 3
2 21.01 3.50 Male No Sun Dinner 3
tips["total_bill"].plot(kind="hist", bins=20,
                        title="Restaurant bills",
                        xlabel="Total bill ($)")
plt.show()
print(tips["total_bill"].median())
No description has been provided for this image
17.795
order = ["Thur", "Fri", "Sat", "Sun"]
avg_tip = tips.groupby("day")["tip"].mean().reindex(order)
print(avg_tip.round(2))
avg_tip.plot(kind="bar", rot=0, title="Average tip by day",
             ylabel="Tip ($)")
plt.show()
day
Thur    2.77
Fri     2.73
Sat     2.99
Sun     3.26
Name: tip, dtype: float64
No description has been provided for this image

Common mistake: a scatter plot with no columns named

tips.plot(kind="scatter")
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[13], line 1
----> 1 tips.plot(kind="scatter")
      2 plt.show()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\plotting\_core.py:1129, in PlotAccessor.__call__(self, *args, **kwargs)
   1127 if kind in self._dataframe_kinds:
   1128     if isinstance(data, ABCDataFrame):
-> 1129         return plot_backend.plot(data, x=x, y=y, kind=kind, **kwargs)
   1130     else:
   1131         raise ValueError(f"plot kind {kind} can only be used for data frames")

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\plotting\_matplotlib\__init__.py:70, in plot(data, kind, **kwargs)
     68             ax = plt.gca()
     69         kwargs["ax"] = getattr(ax, "left_ax", ax)
---> 70 plot_obj = PLOT_CLASSES[kind](data, **kwargs)
     71 plot_obj.generate()
     72 plt.draw_if_interactive()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\plotting\_matplotlib\core.py:1340, in ScatterPlot.__init__(self, data, x, y, s, c, colorbar, norm, **kwargs)
   1337 self.colorbar = colorbar
   1338 self.norm = norm
-> 1340 super().__init__(data, x, y, **kwargs)
   1341 if is_integer(c) and not holds_integer(self.data.columns):
   1342     c = self.data.columns[c]

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\plotting\_matplotlib\core.py:1268, in PlanePlot.__init__(self, data, x, y, **kwargs)
   1266 MPLPlot.__init__(self, data, **kwargs)
   1267 if x is None or y is None:
-> 1268     raise ValueError(self._kind + " requires an x and y column")
   1269 if is_integer(x) and not holds_integer(self.data.columns):
   1270     x = self.data.columns[x]

ValueError: scatter requires an x and y column
tips.plot(kind="scatter", x="total_bill", y="tip", alpha=0.6,
          title="Bill vs tip", xlabel="Bill ($)", ylabel="Tip ($)")
plt.show()
print(tips["total_bill"].corr(tips["tip"]).round(2))
No description has been provided for this image
0.68
tips[["total_bill", "tip"]].plot(kind="box", title="Bills and tips",
                                 ylabel="Dollars")
plt.show()
print(tips["tip"].quantile([0.25, 0.5, 0.75]))
No description has been provided for this image
0.25    2.0000
0.50    2.9000
0.75    3.5625
Name: tip, dtype: float64
tips["tip"].plot.hist(bins=15, title="Tips", xlabel="Tip ($)")
plt.show()
print(tips["tip"].mode().iloc[0])
No description has been provided for this image
2.0
trips = ny.dt.date.value_counts().sort_index()
running = trips.cumsum()
running.plot(kind="area", alpha=0.5, title="Trips so far",
             xlabel="Date", ylabel="Total trips", figsize=(8, 4))
plt.show()
print(running.iloc[-1])
No description has been provided for this image
6433

Your turn#

Draw a horizontal bar chart of taxi pickups per borough.

boro = taxis["pickup_borough"].value_counts()
boro.plot(kind="barh", title="Pickups by borough",
          xlabel="Trips")
plt.show()
print(boro)
No description has been provided for this image
pickup_borough
Manhattan    5268
Queens        657
Brooklyn      383
Bronx          99
Name: count, dtype: int64

Label and style a chart#

Problem: a chart nobody can read is a chart nobody uses.

ax = tips["tip"].plot(kind="hist", bins=15, color="teal",
                      edgecolor="white", figsize=(7, 4))
ax.set_title("How much do diners tip?")
ax.set_xlabel("Tip ($)")
ax.set_ylabel("Number of bills")
plt.show()
No description has been provided for this image
share = tips["time"].value_counts()
share.plot(kind="pie", autopct="%1.0f%%", ylabel="",
           title="Bills by time of day")
plt.show()
print(share)
No description has been provided for this image
time
Dinner    176
Lunch      68
Name: count, dtype: int64

Common mistake: a pie chart of a whole table

smokers = tips.groupby("smoker")[["tip", "total_bill"]].sum()
smokers.plot(kind="pie")
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[21], line 2
      1 smokers = tips.groupby("smoker")[["tip", "total_bill"]].sum()
----> 2 smokers.plot(kind="pie")
      3 plt.show()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\plotting\_core.py:1135, in PlotAccessor.__call__(self, *args, **kwargs)
   1133 if isinstance(data, ABCDataFrame):
   1134     if y is None and kwargs.get("subplots") is False:
-> 1135         raise ValueError(
   1136             f"{kind} requires either y column or 'subplots=True'"
   1137         )
   1138     if y is not None:
   1139         if is_integer(y) and not holds_integer(data.columns):

ValueError: pie requires either y column or 'subplots=True'
smokers.plot(kind="pie", y="tip", autopct="%1.0f%%", ylabel="",
             legend=False, title="Share of all tip money")
plt.show()
print(smokers)
No description has been provided for this image
           tip  total_bill
smoker                    
No      451.77     2897.43
Yes     279.81     1930.34

Quick check: which is easier to compare, two pie slices or two bars?

ax = share.plot(kind="barh", color=["steelblue", "orange"],
                title="Bills by time of day")
ax.set_xlabel("Bills")
plt.show()
No description has been provided for this image

Your turn#

Plot a labelled histogram of taxi fares, 30 bins, in any colour.

ax = taxis["fare"].plot(kind="hist", bins=30, color="slateblue")
ax.set_title("Taxi fares, March 2019")
ax.set_xlabel("Fare ($)")
ax.set_ylabel("Trips")
plt.show()
print(taxis["fare"].median(), taxis["fare"].max())
No description has been provided for this image
9.5 150.0

Plot several columns at once#

Problem: compare columns without drawing each one by hand.

march = ny.dt.month == 3
daily = taxis[march].groupby(ny[march].dt.date)[["fare", "tip"]].mean()
print(daily.round(2).head(3))
daily.plot(title="Average fare and tip per day", ylabel="$",
           figsize=(9, 4))
plt.show()
             fare   tip
pickup                 
2019-03-01  12.23  1.84
2019-03-02  11.91  1.69
2019-03-03  12.95  1.82
No description has been provided for this image

Quick check: fares sit near 13 and tips near 2. Can you see the tip line moving?

daily.plot(subplots=True, figsize=(9, 5), legend=False,
           title=["Average fare ($)", "Average tip ($)"])
plt.show()
print(daily["tip"].min().round(2), daily["tip"].max().round(2))
No description has been provided for this image
1.48 2.34
ax = daily.plot(secondary_y="tip", figsize=(9, 4),
                title="Fare (left) and tip (right) per day")
ax.set_ylabel("Average fare ($)")
ax.right_ax.set_ylabel("Average tip ($)")
plt.show()
print(daily["fare"].corr(daily["tip"]).round(2))
No description has been provided for this image
0.57
counts = pd.crosstab(tips["day"], tips["time"]).reindex(order)
print(counts)
counts.plot(kind="bar", stacked=True, rot=0, ylabel="Bills",
            title="Bills by day, lunch vs dinner")
plt.show()
time  Dinner  Lunch
day                
Thur       1     61
Fri       12      7
Sat       87      0
Sun       76      0
No description has been provided for this image
taxis.plot(kind="hexbin", x="distance", y="fare", gridsize=25,
           cmap="Blues", title="Distance vs fare",
           xlabel="Miles", ylabel="Fare ($)")
plt.show()
print(taxis["distance"].corr(taxis["fare"]).round(2))
No description has been provided for this image
0.92

Combine charts and save them#

Problem: show charts side by side, and keep them as files.

fig, axes = plt.subplots(1, 2, figsize=(10, 4))
tips["tip"].plot(kind="hist", bins=15, ax=axes[0],
                 title="Tips", xlabel="Tip ($)")
tips.plot(kind="scatter", x="size", y="tip", ax=axes[1],
          title="Party size vs tip")
plt.tight_layout()
plt.show()
No description has been provided for this image
print(tips.groupby("size")["tip"].mean().round(2))
size
1    1.44
2    2.58
3    3.39
4    4.14
5    4.03
6    5.22
Name: tip, dtype: float64

Common mistake: saving from the Axes

import os
os.makedirs("output/pandas", exist_ok=True)
ax = tips["total_bill"].plot(kind="hist", bins=20, title="Bills")
ax.savefig("output/pandas/l40_bills.png")
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[32], line 4
      2 os.makedirs("output/pandas", exist_ok=True)
      3 ax = tips["total_bill"].plot(kind="hist", bins=20, title="Bills")
----> 4 ax.savefig("output/pandas/l40_bills.png")
      5 plt.show()

AttributeError: 'Axes' object has no attribute 'savefig'
No description has been provided for this image
ax = tips["total_bill"].plot(kind="hist", bins=20, title="Bills")
ax.set_xlabel("Total bill ($)")
fig = ax.get_figure()
fig.savefig("output/pandas/l40_bills.png", dpi=100)
plt.close(fig)
img = plt.imread("output/pandas/l40_bills.png")
print(os.path.exists("output/pandas/l40_bills.png"))
print(img.ndim, img.shape[2])
True
3 4

Your turn#

Put the borough bars above the fare histogram, then save the figure.

fig, axes = plt.subplots(2, 1, figsize=(7, 7))
boro.plot(kind="barh", ax=axes[0], title="Pickups by borough")
taxis["fare"].plot(kind="hist", bins=30, ax=axes[1],
                   title="Taxi fares", xlabel="Fare ($)")
fig.tight_layout()
fig.savefig("output/pandas/l40_taxis.png", dpi=100)
print(os.path.exists("output/pandas/l40_taxis.png"))
True
No description has been provided for this image

Mini-project: do diners tip more at dinner?#

Plan: tip as a % of the bill, compare lunch and dinner, chart it.

tips["pct"] = tips["tip"] / tips["total_bill"] * 100
tips.groupby("time")["pct"].describe().round(1)
count mean std min 25% 50% 75% max
time
Dinner 176.0 16.0 6.7 3.6 12.3 15.5 18.8 71.0
Lunch 68.0 16.4 4.0 7.3 13.9 15.4 19.4 26.6
axes = tips.plot.box(column="pct", by="time", figsize=(7, 4),
                     ylabel="Tip (% of bill)")
plt.show()
No description has been provided for this image
high = tips[tips["pct"] > 30]
high[["total_bill", "tip", "size", "time", "pct"]].round(1)
total_bill tip size time pct
67 3.1 1.0 1 Dinner 32.6
172 7.2 5.2 2 Dinner 71.0
178 9.6 4.0 2 Dinner 41.7
by_day = tips.groupby("day")["pct"].median().reindex(order)
by_day.plot(kind="bar", rot=0, ylabel="Median tip (%)",
            title="Median tip % by day")
plt.show()
print(by_day.round(1))
No description has been provided for this image
day
Thur    15.4
Fri     15.6
Sat     15.2
Sun     16.1
Name: pct, dtype: float64
typical = tips[tips["pct"] <= 30]
med = typical.groupby("time")["pct"].median().round(1)
med.plot(kind="bar", rot=0, ylabel="Median tip (%)",
         title="Median tip %: lunch vs dinner")
plt.show()
print(med)
No description has been provided for this image
time
Dinner    15.3
Lunch     15.4
Name: pct, dtype: float64
lunch, dinner = med["Lunch"], med["Dinner"]
print(f"Median tip: lunch {lunch}%, dinner {dinner}%, "
      f"a gap of {abs(dinner - lunch):.1f} points.")
Median tip: lunch 15.4%, dinner 15.3%, a gap of 0.1 points.

Recap#

  • .plot() turns any Series or DataFrame into a chart
  • kind= picks the shape: line, bar, barh, hist, box, scatter, pie
  • Count text columns first; scatter needs x= and y=
  • Keep the Axes to add titles and labels
  • ax= combines charts; save the figure, not the Axes

Challenges#

  1. Plot average taxi tip by payment type as a bar chart.
  2. Draw a pie chart of smokers vs non-smokers in the tips data.
  3. Put a histogram of party size next to a box plot of tip %.

What's next#

Lesson 41: Creating Line, Bar, Histogram, and Box Plots Go deeper on the four workhorse charts and when to use each.

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.