Mathew K Analytics

Lesson 41 · Pandas Full Course

Line, Bar, Histogram and Box Plots in Pandas: Pick the Right Chart | Pandas Course #41

In this lesson you'll be able to: Draw line plots of trends, with several lines and a moving average Compare groups with sorted, grouped and stacked bar…

What you'll learn

Datasets used in this lesson

Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.

📓 Full notebook

Download .ipynb

Pandas 41: Creating Line, Bar, Histogram, and Box Plots#

In this lesson you'll be able to:

  • Draw line plots of trends, with several lines and a moving average
  • Compare groups with sorted, grouped and stacked bar charts
  • Read the shape of data with histograms and choose good bins
  • Compare spreads and spot outliers with box plots
  • Pick the right chart and combine all four in one figure Data: airline passengers 1949-1960 and restaurant tips

Warm-up: one-line charts#

Lesson 40 used .plot(kind=...) on the tips data. Same data, same tool.

import pandas as pd
import matplotlib.pyplot as plt
tips = pd.read_csv("data/pandas/tips.csv")
tips["pct"] = tips["tip"] / tips["total_bill"] * 100
print(tips.groupby("time")["pct"].median().round(1))
time
Dinner    15.5
Lunch     15.4
Name: pct, dtype: float64
tips["pct"].plot(kind="hist", bins=20, title="Tip as % of bill",
                 xlabel="Tip (%)")
plt.show()
No description has been provided for this image

Line plots: change over time#

Problem: see a trend and a seasonal pattern in a time series.

flights = pd.read_csv("data/pandas/flights.csv")
print(flights.shape)
flights.head(3)
(144, 3)
year month passengers
0 1949 January 112
1 1949 February 118
2 1949 March 132

Common mistake: the wrong month code when building dates

text = flights["year"].astype(str) + "-" + flights["month"]
pd.to_datetime(text, format="%Y-%b")
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[5], line 2
      1 text = flights["year"].astype(str) + "-" + flights["month"]
----> 2 pd.to_datetime(text, format="%Y-%b")

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\tools\datetimes.py:1040, in to_datetime(arg, errors, dayfirst, yearfirst, utc, format, exact, unit, origin, cache)
   1038         result = arg.map(cache_array)
   1039     else:
-> 1040         values = convert_listlike(arg._values, format)
   1041         result = arg._constructor(values, index=arg.index, name=arg.name)
   1042 elif isinstance(arg, (ABCDataFrame, abc.MutableMapping)):

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\tools\datetimes.py:435, in _convert_listlike_datetimes(arg, format, name, utc, unit, errors, dayfirst, yearfirst, exact)
    433 # `format` could be inferred, or user didn't ask for mixed-format parsing.
    434 if format is not None and format != "mixed":
--> 435     return _array_strptime_with_fallback(arg, name, utc, format, exact, errors)
    437 result, tz_parsed = objects_to_datetime64(
    438     arg,
    439     dayfirst=dayfirst,
   (...)
    443     allow_object=True,
    444 )
    446 if tz_parsed is not None:
    447     # We can take a shortcut since the datetime64 numpy array
    448     # is in UTC

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\tools\datetimes.py:470, in _array_strptime_with_fallback(arg, name, utc, fmt, exact, errors)
    459 def _array_strptime_with_fallback(
    460     arg,
    461     name,
   (...)
    465     errors: str,
    466 ) -> Index:
    467     """
    468     Call array_strptime, with fallback behavior depending on 'errors'.
    469     """
--> 470     result, tz_out = array_strptime(arg, fmt, exact=exact, errors=errors, utc=utc)
    471     if tz_out is not None:
    472         unit = np.datetime_data(result.dtype)[0]

File pandas/_libs/tslibs/strptime.pyx:564, in pandas._libs.tslibs.strptime.array_strptime()

File pandas/_libs/tslibs/strptime.pyx:512, in pandas._libs.tslibs.strptime.array_strptime()

File pandas/_libs/tslibs/strptime.pyx:632, in pandas._libs.tslibs.strptime._parse_with_format()

ValueError: unconverted data remains when parsing with format "%Y-%b": "uary". You might want to try:
    - passing `format` if your strings have a consistent format;
    - passing `format='ISO8601'` if your strings are all ISO8601 but not necessarily in exactly the same format;
    - passing `format='mixed'`, and the format will be inferred for each element individually. You might want to use `dayfirst` alongside this.
flights["date"] = pd.to_datetime(text, format="%Y-%B")
air = flights.set_index("date")["passengers"]
print(air.head(3))
date
1949-01-01    112
1949-02-01    118
1949-03-01    132
Name: passengers, dtype: int64
ax = air.plot(figsize=(9, 4), title="Monthly airline passengers")
ax.set_xlabel("Month")
ax.set_ylabel("Passengers (thousands)")
plt.show()
print(air.iloc[0], air.iloc[-1])
No description has been provided for this image
112 432

Quick check: which month is the busiest in most years?

peak = flights.loc[flights.groupby("year")["passengers"].idxmax()]
peak["month"].value_counts()
month
July      7
August    5
Name: count, dtype: int64
wide = flights.pivot(index="month", columns="year",
                     values="passengers")
months = flights['month'].unique()
wide = wide.reindex(months)
wide[[1949, 1955, 1960]].head(3)
year 1949 1955 1960
month
January 112 242 417
February 118 233 391
March 132 267 419
ax = wide[[1949, 1955, 1960]].plot(marker="o", figsize=(9, 4),
                                  title="Seasonal pattern")
ax.set_xlabel("Month")
ax.set_ylabel("Passengers (thousands)")
plt.show()
No description has been provided for this image
ax = air.plot(label="Monthly", alpha=0.5, figsize=(9, 4))
trend = air.rolling(12).mean()
trend.plot(ax=ax, label="12-month average", linewidth=2)
ax.set_title("Passengers with a 12-month average")
ax.set_ylabel("Passengers (thousands)")
ax.legend()
plt.show()
print(trend.dropna().iloc[[0, -1]].round(1))
No description has been provided for this image
date
1949-12-01    126.7
1960-12-01    476.2
Name: passengers, dtype: float64
air.plot(logy=True, figsize=(9, 4), title="Passengers, log scale",
         ylabel="Passengers (log scale)", xlabel="Month")
plt.show()
No description has been provided for this image
yoy = air.pct_change(12) * 100
ax = yoy.plot(figsize=(9, 4), title="Growth on the same month a "
              "year before")
ax.set_ylabel("Growth (%)")
plt.show()
print(yoy.loc["1950"].mean().round(1), yoy.loc["1960"].mean().round(1))
No description has been provided for this image
10.0 11.2

Your turn#

Plot only the summer months, June to August, as three lines over the years.

summer = wide.loc[["June", "July", "August"]].T
ax = summer.plot(marker="o", title="Summer months over the years")
ax.set_ylabel("Passengers (thousands)")
plt.show()
print(summer.loc[1960])
No description has been provided for this image
month
June      535
July      622
August    606
Name: 1960, dtype: int64

Bar plots: compare categories#

Problem: compare totals or averages across groups.

order = ["Thur", "Fri", "Sat", "Sun"]
tips["day"] = pd.Categorical(tips["day"], categories=order,
                             ordered=True)
bills = tips["day"].value_counts(sort=False)
print(bills)
day
Thur    62
Fri     19
Sat     87
Sun     76
Name: count, dtype: int64
ax = bills.plot(kind="bar", rot=0, color="steelblue",
                title="Bills per day")
ax.set_xlabel("Day")
ax.set_ylabel("Bills")
plt.show()
No description has been provided for this image

Common mistake: averaging a table that has text columns

tips.groupby("day", observed=True).mean()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[17], line 1
----> 1 tips.groupby("day", observed=True).mean()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\groupby\groupby.py:2285, in GroupBy.mean(self, numeric_only, skipna, engine, engine_kwargs)
   2277     return self._numba_agg_general(
   2278         grouped_mean,
   2279         executor.float_dtype_mapping,
   (...)
   2282         skipna=skipna,
   2283     )
   2284 else:
-> 2285     result = self._cython_agg_general(
   2286         "mean",
   2287         alt=lambda x: Series(x, copy=False).mean(
   2288             numeric_only=numeric_only, skipna=skipna
   2289         ),
   2290         numeric_only=numeric_only,
   2291         skipna=skipna,
   2292     )
   2293     return result.__finalize__(self.obj, method="groupby")

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\groupby\groupby.py:1808, in GroupBy._cython_agg_general(self, how, alt, numeric_only, min_count, **kwargs)
   1805     result = self._agg_py_fallback(how, values, ndim=data.ndim, alt=alt)
   1806     return result
-> 1808 new_mgr = data.grouped_reduce(array_func)
   1809 res = self._wrap_agged_manager(new_mgr)
   1810 if how in ["idxmin", "idxmax"]:
   1811     # mypy expects how to be Literal["idxmin", "idxmax"].

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\managers.py:1667, in BlockManager.grouped_reduce(self, func)
   1665             result_blocks = extend_blocks(applied, result_blocks)
   1666     else:
-> 1667         applied = blk.apply(func)
   1668         result_blocks = extend_blocks(applied, result_blocks)
   1670 if len(result_blocks) == 0:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\blocks.py:347, in Block.apply(self, func, **kwargs)
    341 @final
    342 def apply(self, func, **kwargs) -> list[Block]:
    343     """
    344     apply the function to my values; return a block if we are not
    345     one
    346     """
--> 347     result = func(self.values, **kwargs)
    349     result = maybe_coerce_values(result)
    350     return self._split_op_result(result)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\groupby\groupby.py:1783, in GroupBy._cython_agg_general.<locals>.array_func(values)
   1781 def array_func(values: ArrayLike) -> ArrayLike:
   1782     try:
-> 1783         result = self._grouper._cython_operation(
   1784             "aggregate",
   1785             values,
   1786             how,
   1787             axis=data.ndim - 1,
   1788             min_count=min_count,
   1789             **kwargs,
   1790         )
   1791     except NotImplementedError:
   1792         # generally if we have numeric_only=False
   1793         # and non-applicable functions
   1794         # try to python agg
   1795         # TODO: shouldn't min_count matter?
   1796         # TODO: avoid special casing SparseArray here
   1797         if how in ["any", "all"] and isinstance(values, SparseArray):

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\groupby\ops.py:963, in BaseGrouper._cython_operation(self, kind, values, how, axis, min_count, **kwargs)
    959 assert kind in ["transform", "aggregate"]
    961 cy_op = WrappedCythonOp(kind=kind, how=how, has_dropped_na=self.has_dropped_na)
--> 963 return cy_op.cython_operation(
    964     values=values,
    965     axis=axis,
    966     min_count=min_count,
    967     comp_ids=self.ids,
    968     ngroups=self.ngroups,
    969     **kwargs,
    970 )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\groupby\ops.py:553, in WrappedCythonOp.cython_operation(self, values, axis, min_count, comp_ids, ngroups, **kwargs)
    549 self._validate_axis(axis, values)
    551 if not isinstance(values, np.ndarray):
    552     # i.e. ExtensionArray
--> 553     return values._groupby_op(
    554         how=self.how,
    555         has_dropped_na=self.has_dropped_na,
    556         min_count=min_count,
    557         ngroups=ngroups,
    558         ids=comp_ids,
    559         **kwargs,
    560     )
    562 return self._cython_op_ndim_compat(
    563     values,
    564     min_count=min_count,
   (...)
    568     **kwargs,
    569 )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\arrow\array.py:2748, in ArrowExtensionArray._groupby_op(self, how, has_dropped_na, min_count, ngroups, ids, **kwargs)
   2736 if isinstance(self.dtype, StringDtype):
   2737     if how in [
   2738         "prod",
   2739         "mean",
   (...)
   2746         "skew",
   2747     ]:
-> 2748         raise TypeError(
   2749             f"dtype '{self.dtype}' does not support operation '{how}'"
   2750         )
   2751     return super()._groupby_op(
   2752         how=how,
   2753         has_dropped_na=has_dropped_na,
   (...)
   2757         **kwargs,
   2758     )
   2760 # maybe convert to a compatible dtype optimized for groupby

TypeError: dtype 'str' does not support operation 'mean'
cols = ["total_bill", "tip"]
avg = tips.groupby("day", observed=True)[cols].mean()
ax = avg.plot(kind="bar", rot=0, title="Average bill and tip")
ax.set_ylabel("Dollars")
plt.show()
print(avg.round(2))
No description has been provided for this image
      total_bill   tip
day                   
Thur       17.68  2.77
Fri        17.15  2.73
Sat        20.44  2.99
Sun        21.41  3.26
yearly = flights.groupby("year")["passengers"].sum()
ax = yearly.sort_values().plot(kind="barh", color="darkorange",
                               title="Passengers per year")
ax.set_xlabel("Passengers (thousands)")
plt.show()
print(yearly.iloc[[0, -1]])
No description has been provided for this image
year
1949    1520
1960    5714
Name: passengers, dtype: int64
ax = bills.plot(kind="bar", rot=0, color="slategray",
                title="Bills per day", ylabel="Bills")
ax.bar_label(ax.containers[0])
plt.show()
print(bills.sum())
No description has been provided for this image
244
mix = pd.crosstab(tips["day"], tips["smoker"], normalize="index")
ax = mix.plot(kind="bar", stacked=True, rot=0,
              title="Smokers share by day")
ax.set_ylabel("Share of bills")
plt.show()
print(mix.round(2))
No description has been provided for this image
smoker    No   Yes
day               
Thur    0.73  0.27
Fri     0.21  0.79
Sat     0.52  0.48
Sun     0.75  0.25

Your turn#

Bar chart: average total bill for each party size.

by_size = tips.groupby("size")["total_bill"].mean()
ax = by_size.plot(kind="bar", rot=0, title="Average bill by party")
ax.set_xlabel("Party size")
ax.set_ylabel("Average bill ($)")
plt.show()
print(by_size.round(2))
No description has been provided for this image
size
1     7.24
2    16.45
3    23.28
4    28.61
5    30.07
6    34.83
Name: total_bill, dtype: float64

Histograms: the shape of a distribution#

Problem: see where values cluster, and how far they stretch.

fig, axes = plt.subplots(1, 2, figsize=(10, 4))
tips["total_bill"].plot(kind="hist", bins=5, ax=axes[0],
                        title="5 bins", xlabel="Bill ($)")
tips["total_bill"].plot(kind="hist", bins=40, ax=axes[1],
                        title="40 bins", xlabel="Bill ($)")
plt.tight_layout()
plt.show()
No description has been provided for this image

Quick check: is the bill histogram symmetric, or does it lean?

print(tips["total_bill"].mean().round(2))
print(tips["total_bill"].median())
print(tips["total_bill"].skew().round(2))
19.79
17.795
1.13
ax = tips.loc[tips["time"] == "Lunch", "total_bill"].plot(
    kind="hist", bins=20, alpha=0.5, density=True, label="Lunch")
tips.loc[tips["time"] == "Dinner", "total_bill"].plot(
    kind="hist", bins=20, alpha=0.5, density=True, label="Dinner",
    ax=ax)
ax.set_title("Bills: lunch vs dinner")
ax.set_xlabel("Bill ($)")
ax.legend()
plt.show()
No description has been provided for this image
print(tips.groupby("time")["total_bill"].median())
time
Dinner    18.390
Lunch     15.965
Name: total_bill, dtype: float64
ax = tips["total_bill"].plot(kind="hist", bins=50, cumulative=True,
                             density=True, histtype="step",
                             title="Share of bills up to each value")
ax.set_xlabel("Bill ($)")
ax.set_ylabel("Share of bills")
plt.show()
print((tips["total_bill"] <= 20).mean().round(2))
No description has been provided for this image
0.6

Your turn#

Histogram of tip %, with a dashed line at the median.

med = tips["pct"].median()
ax = tips["pct"].plot(kind="hist", bins=30, title="Tip %")
ax.axvline(med, color="red", linestyle="--", label="Median")
ax.set_xlabel("Tip (% of bill)")
ax.legend()
plt.show()
print(round(med, 1))
No description has been provided for this image
15.5

Box plots: spreads and outliers#

Problem: compare whole distributions across groups at once.

axes = tips.plot.box(column="total_bill", by="day",
                     figsize=(8, 4), ylabel="Bill ($)")
plt.show()
print(tips.groupby("day", observed=True)["total_bill"].median())
No description has been provided for this image
day
Thur    16.20
Fri     15.38
Sat     18.24
Sun     19.63
Name: total_bill, dtype: float64
q1, q3 = tips["total_bill"].quantile([0.25, 0.75])
limit = q3 + 1.5 * (q3 - q1)
outliers = tips[tips["total_bill"] > limit]
print(round(limit, 2), len(outliers))
40.3 9

Quick check: which month has the widest spread across the years?

ax = wide.T.plot(kind="box", figsize=(10, 4), rot=45,
                 title="Passengers by month, 1949-1960")
ax.set_ylabel("Passengers (thousands)")
plt.show()
spread = wide.max(axis=1) - wide.min(axis=1)
print(spread.idxmax(), spread.max())
No description has been provided for this image
July 474

Common mistake: a box plot of a text column

tips["time"].plot(kind="box")
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[32], line 1
----> 1 tips["time"].plot(kind="box")
      2 plt.show()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\plotting\_core.py:1185, in PlotAccessor.__call__(self, *args, **kwargs)
   1182             label_name = label_kw or data.columns
   1183             data.columns = label_name
-> 1185 return plot_backend.plot(data, kind=kind, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\plotting\_matplotlib\__init__.py:71, in plot(data, kind, **kwargs)
     69         kwargs["ax"] = getattr(ax, "left_ax", ax)
     70 plot_obj = PLOT_CLASSES[kind](data, **kwargs)
---> 71 plot_obj.generate()
     72 plt.draw_if_interactive()
     73 return plot_obj.result

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\plotting\_matplotlib\core.py:518, in MPLPlot.generate(self)
    516 @final
    517 def generate(self) -> None:
--> 518     self._compute_plot_data()
    519     fig = self.fig
    520     self._make_plot(fig)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\plotting\_matplotlib\core.py:718, in MPLPlot._compute_plot_data(self)
    716 # no non-numeric frames or series allowed
    717 if is_empty:
--> 718     raise TypeError("no numeric data to plot")
    720 self.data = numeric_data.apply(type(self)._convert_to_ndarray)

TypeError: no numeric data to plot
axes = tips.plot.box(column="tip", by="time", figsize=(7, 4),
                     ylabel="Tip ($)")
plt.show()
print(tips.groupby("time")["tip"].median())
No description has been provided for this image
time
Dinner    3.00
Lunch     2.25
Name: tip, dtype: float64

Your turn#

Box plot of tip % by party size. Which size tips the highest share?

axes = tips.plot.box(column="pct", by="size", figsize=(8, 4),
                     ylabel="Tip (% of bill)")
plt.show()
print(tips.groupby("size")["pct"].median().round(1))
No description has been provided for this image
size
1    20.3
2    15.6
3    15.9
4    14.7
5    12.1
6    16.3
Name: pct, dtype: float64

Choose and combine the four#

Problem: match each question to a chart, then build one figure.

  • Change over time? line | Compare groups? bar
  • Shape of one column? histogram | Spread across groups? box

Common mistake: indexing a 2 by 2 grid with one number

fig, axs = plt.subplots(2, 2, figsize=(10, 7))
tips["total_bill"].plot(kind="hist", ax=axs[2])
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
Cell In[35], line 2
      1 fig, axs = plt.subplots(2, 2, figsize=(10, 7))
----> 2 tips["total_bill"].plot(kind="hist", ax=axs[2])
      3 plt.show()

IndexError: index 2 is out of bounds for axis 0 with size 2
No description has been provided for this image
fig, axs = plt.subplots(2, 2, figsize=(10, 7))
air.plot(ax=axs[0, 0], title="Line: passengers over time")
yearly.plot(kind="bar", ax=axs[0, 1], title="Bar: per year")
tips["total_bill"].plot(kind="hist", bins=20, ax=axs[1, 0],
                        title="Histogram: bills")
tips.plot.box(column="tip", by="time", ax=axs[1, 1])
axs[1, 1].set_title("Box: tip by time")
fig.tight_layout()
plt.show()
No description has been provided for this image
import os
os.makedirs("output/pandas", exist_ok=True)
fig.savefig("output/pandas/l41_four_charts.png", dpi=100)
img = plt.imread("output/pandas/l41_four_charts.png")
print(img.ndim, img.shape[2])
3 4

Mini-project: how did air travel grow, 1949-1960?#

Plan: growth with bars, the summer peak with lines, the seasonal swing with boxes.

growth = yearly.iloc[-1] / yearly.iloc[0]
print(round(growth, 2))
change = yearly.pct_change() * 100
print(change.round(1).describe()[['min', 'mean', 'max']])
3.76
min      3.400000
mean    12.909091
max     21.800000
Name: passengers, dtype: float64
ax = change.plot(kind="bar", rot=45, color="seagreen",
                 title="Growth on the year before (%)")
ax.set_ylabel("Growth (%)")
plt.show()
print(change.idxmin(), change.min().round(1))
No description has been provided for this image
1958 3.4
stats = flights.groupby("year")["passengers"].agg(["min", "max"])
stats["swing"] = stats["max"] - stats["min"]
stats["ratio"] = (stats["max"] / stats["min"]).round(2)
stats.iloc[[0, 5, -1]]
min max swing ratio
year
1949 104 148 44 1.42
1954 188 302 114 1.61
1960 390 622 232 1.59
ax = stats[["swing"]].plot(marker="o", legend=False,
                           title="Gap between peak and low month")
ax.set_ylabel("Passengers (thousands)")
plt.show()
No description has been provided for this image
first, last = stats["ratio"].iloc[0], stats["ratio"].iloc[-1]
july = (peak["month"] == "July").sum()
print(f"Traffic grew {growth:.1f}x; July peaked in "
      f"{july} of 12 years; "
      f"peak/low ratio {first} -> {last}.")
Traffic grew 3.8x; July peaked in 7 of 12 years; peak/low ratio 1.42 -> 1.59.

Recap#

  • Line: trends over time; add a rolling mean or log scale
  • Bar: compare groups; sort, group or stack the bars
  • Histogram: the shape of one column; bins change the story
  • Box: spreads and outliers across groups, with column= and by=
  • Grids use axs[row, col]; save the figure with savefig

Challenges#

  1. Line plot of December passengers across the years.
  2. Grouped bars of average tip by day, split by smoker.
  3. Box plot of tip % by day. Which day has the widest spread?

What's next#

Lesson 42: Integrating Pandas with Matplotlib and Seaborn Take full control of figures, and make statistical charts with seaborn.

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.