Mathew K Analytics

Lesson 42 · Pandas Full Course

Pandas + Matplotlib + Seaborn: Make Professional Charts in Python | Pandas Course #42

In this lesson you'll be able to: Take control of any pandas chart through its Matplotlib axes Put several pandas charts side by side in one figure Hand a…

What you'll learn

Datasets used in this lesson

Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.

📓 Full notebook

Download .ipynb

Pandas 42: Integrating Pandas with Matplotlib and Seaborn#

In this lesson you'll be able to:

  • Take control of any pandas chart through its Matplotlib axes
  • Put several pandas charts side by side in one figure
  • Hand a DataFrame straight to Seaborn and colour by a column
  • Mix Seaborn and Matplotlib: reference lines, notes, themes
  • Save a finished figure to a file and check it Dataset: restaurant tips (data/pandas/tips.csv)

Warm-up: a quick pandas chart#

Lesson 41 drew line, bar, histogram and box plots with pandas.

import pandas as pd
import matplotlib.pyplot as plt
df = pd.read_csv("data/pandas/tips.csv")
print(df.shape)
df.head()
(244, 7)
total_bill tip sex smoker day time size
0 16.99 1.01 Female No Sun Dinner 2
1 10.34 1.66 Male No Sun Dinner 3
2 21.01 3.50 Male No Sun Dinner 3
3 23.68 3.31 Male No Sun Dinner 2
4 24.59 3.61 Female No Sun Dinner 4
df["total_bill"].plot(kind="hist", bins=20, edgecolor="black")
plt.title("Distribution of total bills")
plt.xlabel("Total bill ($)")
plt.ylabel("Number of bills")
plt.show()
No description has been provided for this image

Every pandas chart is a Matplotlib chart#

Problem: style and adjust a pandas chart beyond its own options.

ax = df["tip"].plot(kind="hist", bins=20)
print(type(ax))
<class 'matplotlib.axes._axes.Axes'>
No description has been provided for this image
ax = df["tip"].plot(kind="hist", bins=20, color="teal")
ax.set_title("Most tips are between 2 and 4 dollars")
ax.set_xlabel("Tip ($)")
ax.set_ylabel("Number of bills")
ax.axvline(df["tip"].median(), color="black", linestyle="--")
plt.show()
No description has been provided for this image
print(df["tip"].median())
print((df["tip"].between(2, 4)).mean().round(2))
2.9
0.65

Common mistake: a scatter plot with only one column

df.plot(kind="scatter", x="total_bill")
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[7], line 1
----> 1 df.plot(kind="scatter", x="total_bill")
      2 plt.show()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\plotting\_core.py:1129, in PlotAccessor.__call__(self, *args, **kwargs)
   1127 if kind in self._dataframe_kinds:
   1128     if isinstance(data, ABCDataFrame):
-> 1129         return plot_backend.plot(data, x=x, y=y, kind=kind, **kwargs)
   1130     else:
   1131         raise ValueError(f"plot kind {kind} can only be used for data frames")

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\plotting\_matplotlib\__init__.py:70, in plot(data, kind, **kwargs)
     68             ax = plt.gca()
     69         kwargs["ax"] = getattr(ax, "left_ax", ax)
---> 70 plot_obj = PLOT_CLASSES[kind](data, **kwargs)
     71 plot_obj.generate()
     72 plt.draw_if_interactive()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\plotting\_matplotlib\core.py:1340, in ScatterPlot.__init__(self, data, x, y, s, c, colorbar, norm, **kwargs)
   1337 self.colorbar = colorbar
   1338 self.norm = norm
-> 1340 super().__init__(data, x, y, **kwargs)
   1341 if is_integer(c) and not holds_integer(self.data.columns):
   1342     c = self.data.columns[c]

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\plotting\_matplotlib\core.py:1268, in PlanePlot.__init__(self, data, x, y, **kwargs)
   1266 MPLPlot.__init__(self, data, **kwargs)
   1267 if x is None or y is None:
-> 1268     raise ValueError(self._kind + " requires an x and y column")
   1269 if is_integer(x) and not holds_integer(self.data.columns):
   1270     x = self.data.columns[x]

ValueError: scatter requires an x and y column
ax = df.plot(kind="scatter", x="total_bill", y="tip", alpha=0.6)
ax.set_title("Bigger bills, bigger tips")
ax.set_xlabel("Total bill ($)")
ax.set_ylabel("Tip ($)")
plt.show()
No description has been provided for this image

Quick check: how strongly do bill and tip move together?

print(round(df["total_bill"].corr(df["tip"]), 2))
0.68

Several charts in one figure#

Problem: compare two views side by side instead of scrolling.

  • A figure is the whole canvas
  • Each axes is one chart drawn on it
fig, (left, right) = plt.subplots(1, 2, figsize=(10, 4))
df["total_bill"].plot(kind="hist", bins=20, ax=left)
df["tip"].plot(kind="hist", bins=20, ax=right, color="teal")
left.set_title("Total bill ($)")
right.set_title("Tip ($)")
fig.tight_layout()
plt.show()
No description has been provided for this image
day_order = ["Thur", "Fri", "Sat", "Sun"]
counts = df["day"].value_counts().reindex(day_order)
avg_tip = df.groupby("day")["tip"].mean().reindex(day_order)
print(counts.to_dict())
print(avg_tip.round(2).to_dict())
{'Thur': 62, 'Fri': 19, 'Sat': 87, 'Sun': 76}
{'Thur': 2.77, 'Fri': 2.73, 'Sat': 2.99, 'Sun': 3.26}
fig, axes = plt.subplots(1, 2, figsize=(10, 4), sharex=True)
counts.plot(kind="bar", ax=axes[0], rot=0, color="grey")
avg_tip.plot(kind="bar", ax=axes[1], rot=0, color="teal")
axes[0].set(title="Bills per day", xlabel="Day", ylabel="Bills")
axes[1].set(title="Average tip", xlabel="Day", ylabel="Tip ($)")
fig.tight_layout()
plt.show()
No description has been provided for this image

Quick check: Friday has the fewest bills. Is its average tip the lowest?

print(avg_tip.idxmin(), round(avg_tip.min(), 2))
Fri 2.73

Hand the DataFrame to Seaborn#

Problem: colour and split a chart by a category without loops.

import seaborn as sns
ax = sns.histplot(data=df, x="total_bill", bins=20, kde=True)
ax.set(title="Total bills with a smooth density curve",
       xlabel="Total bill ($)", ylabel="Number of bills")
plt.show()
No description has been provided for this image
ax = sns.scatterplot(data=df, x="total_bill", y="tip",
                     hue="time", alpha=0.7)
ax.set(title="Tips against bills, lunch and dinner",
       xlabel="Total bill ($)", ylabel="Tip ($)")
plt.show()
No description has been provided for this image
df.groupby("time")["total_bill"].agg(["size", "mean"]).round(2)
size mean
time
Dinner 176 20.80
Lunch 68 17.17
ax = sns.regplot(data=df, x="total_bill", y="tip",
                 scatter_kws={"alpha": 0.5})
ax.set(title="Each extra dollar of bill adds about 10 cents",
       xlabel="Total bill ($)", ylabel="Tip ($)")
plt.show()
No description has been provided for this image
import numpy as np
slope, intercept = np.polyfit(df["total_bill"], df["tip"], 1)
print(round(slope, 3), round(intercept, 2))
0.105 0.92

Common mistake: a column name Seaborn cannot find

sns.boxplot(data=df, x="day", y="full_bill")
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[19], line 1
----> 1 sns.boxplot(data=df, x="day", y="full_bill")
      2 plt.show()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\seaborn\categorical.py:1597, in boxplot(data, x, y, hue, order, hue_order, orient, color, palette, saturation, fill, dodge, width, gap, whis, linecolor, linewidth, fliersize, hue_norm, native_scale, log_scale, formatter, legend, ax, **kwargs)
   1589 def boxplot(
   1590     data=None, *, x=None, y=None, hue=None, order=None, hue_order=None,
   1591     orient=None, color=None, palette=None, saturation=.75, fill=True,
   (...)
   1594     legend="auto", ax=None, **kwargs
   1595 ):
-> 1597     p = _CategoricalPlotter(
   1598         data=data,
   1599         variables=dict(x=x, y=y, hue=hue),
   1600         order=order,
   1601         orient=orient,
   1602         color=color,
   1603         legend=legend,
   1604     )
   1606     if ax is None:
   1607         ax = plt.gca()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\seaborn\categorical.py:67, in _CategoricalPlotter.__init__(self, data, variables, order, orient, require_numeric, color, legend)
     56 def __init__(
     57     self,
     58     data=None,
   (...)
     64     legend="auto",
     65 ):
---> 67     super().__init__(data=data, variables=variables)
     69     # This method takes care of some bookkeeping that is necessary because the
     70     # original categorical plots (prior to the 2021 refactor) had some rules that
     71     # don't fit exactly into VectorPlotter logic. It may be wise to have a second
   (...)
     76     # default VectorPlotter rules. If we do decide to make orient part of the
     77     # _base variable assignment, we'll want to figure out how to express that.
     78     if self.input_format == "wide" and orient in ["h", "y"]:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\seaborn\_base.py:634, in VectorPlotter.__init__(self, data, variables)
    629 # var_ordered is relevant only for categorical axis variables, and may
    630 # be better handled by an internal axis information object that tracks
    631 # such information and is set up by the scale_* methods. The analogous
    632 # information for numeric axes would be information about log scales.
    633 self._var_ordered = {"x": False, "y": False}  # alt., used DefaultDict
--> 634 self.assign_variables(data, variables)
    636 # TODO Lots of tests assume that these are called to initialize the
    637 # mappings to default values on class initialization. I'd prefer to
    638 # move away from that and only have a mapping when explicitly called.
    639 for var in ["hue", "size", "style"]:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\seaborn\_base.py:679, in VectorPlotter.assign_variables(self, data, variables)
    674 else:
    675     # When dealing with long-form input, use the newer PlotData
    676     # object (internal but introduced for the objects interface)
    677     # to centralize / standardize data consumption logic.
    678     self.input_format = "long"
--> 679     plot_data = PlotData(data, variables)
    680     frame = plot_data.frame
    681     names = plot_data.names

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\seaborn\_core\data.py:58, in PlotData.__init__(self, data, variables)
     51 def __init__(
     52     self,
     53     data: DataSource,
     54     variables: dict[str, VariableSpec],
     55 ):
     57     data = handle_data_source(data)
---> 58     frame, names, ids = self._assign_variables(data, variables)
     60     self.frame = frame
     61     self.names = names

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\seaborn\_core\data.py:232, in PlotData._assign_variables(self, data, variables)
    230     else:
    231         err += "An entry with this name does not appear in `data`."
--> 232     raise ValueError(err)
    234 else:
    235 
    236     # Otherwise, assume the value somehow represents data
    237 
    238     # Ignore empty data structures
    239     if isinstance(val, Sized) and len(val) == 0:

ValueError: Could not interpret value `full_bill` for `y`. An entry with this name does not appear in `data`.
ax = sns.boxplot(data=df, x="day", y="total_bill", order=day_order)
ax.set(title="Weekend bills run higher",
       xlabel="Day", ylabel="Total bill ($)")
plt.show()
No description has been provided for this image

Compare groups: pandas numbers, Seaborn pictures#

Problem: show a grouped average, and how much it varies.

by_group = df.groupby(["sex", "smoker"])["tip"].mean()
by_group.round(2)
sex     smoker
Female  No        2.77
        Yes       2.93
Male    No        3.11
        Yes       3.05
Name: tip, dtype: float64
ax = sns.barplot(data=df, x="sex", y="tip", hue="smoker",
                 errorbar=None)
ax.set(title="Average tip by sex and smoker",
       xlabel="Sex", ylabel="Average tip ($)")
plt.show()
No description has been provided for this image
ax = sns.barplot(data=df, x="day", y="tip", order=day_order,
                 errorbar=("ci", 95), color="teal")
ax.set(title="Average tip by day, with 95% intervals",
       xlabel="Day", ylabel="Average tip ($)")
plt.show()
No description has been provided for this image

Your turn#

Draw a violin plot of tip by time, split by smoker.

ax = sns.violinplot(data=df, x="time", y="tip", hue="smoker",
                    split=True)
ax.set(title="Tip by meal time and smoker",
       xlabel="Meal", ylabel="Tip ($)")
plt.show()
No description has been provided for this image
df.groupby(["time", "smoker"])["tip"].median()
time    smoker
Dinner  No        3.00
        Yes       3.00
Lunch   No        2.18
        Yes       2.50
Name: tip, dtype: float64

Mix both libraries: themes, notes and reference lines#

Problem: make a chart presentation-ready, not just correct.

sns.set_theme(style="whitegrid")
df["tip_pct"] = 100 * df["tip"] / df["total_bill"]
print(df["tip_pct"].describe().round(1))
count    244.0
mean      16.1
std        6.1
min        3.6
25%       12.9
50%       15.5
75%       19.1
max       71.0
Name: tip_pct, dtype: float64
ax = sns.boxplot(data=df, x="day", y="tip_pct", order=day_order)
ax.axhline(15, color="red", linestyle="--")
ax.annotate("71% tip on a $7.25 bill", xy=(3, 71), xytext=(1.2, 60),
            arrowprops={"arrowstyle": "->", "color": "black"})
ax.set(title="Tip percentage by day",
       xlabel="Day", ylabel="Tip (% of bill)")
plt.show()
No description has been provided for this image
top = df.nlargest(1, "tip_pct")
top[["total_bill", "tip", "day", "tip_pct"]].round(2)
total_bill tip day tip_pct
172 7.25 5.15 Sun 71.03
with sns.axes_style("darkgrid"):
    ax = sns.countplot(data=df, x="day", hue="smoker",
                       order=day_order)
    ax.set(title="Bills by day and smoker",
           xlabel="Day", ylabel="Bills")
    plt.show()
No description has been provided for this image
pd.crosstab(df["day"], df["smoker"]).reindex(day_order)
smoker No Yes
day
Thur 45 17
Fri 4 15
Sat 45 42
Sun 57 19

Quick check: on which day do smokers outnumber non-smokers?

table = pd.crosstab(df["day"], df["smoker"])
table[table["Yes"] > table["No"]]
smoker No Yes
day
Fri 4 15

Save a figure and check it#

Problem: get a chart into a report, a slide or an email.

import os
os.makedirs("output/pandas", exist_ok=True)
fig, ax = plt.subplots(figsize=(6, 4))
sns.histplot(data=df, x="tip", bins=15, ax=ax, color="purple")
ax.set(title="Tips", xlabel="Tip ($)", ylabel="Bills")
fig.tight_layout()
fig.savefig("output/pandas/l42_tips_hist.png", dpi=100)
plt.close(fig)
img = plt.imread("output/pandas/l42_tips_hist.png")
print(img.shape)
(400, 600, 4)

Common mistake: saving to a folder that does not exist

fig.savefig("reports/l42_tips.png")
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[34], line 1
----> 1 fig.savefig("reports/l42_tips.png")

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\matplotlib\figure.py:3515, in Figure.savefig(self, fname, transparent, **kwargs)
   3513     for ax in self.axes:
   3514         _recursively_make_axes_transparent(stack, ax)
-> 3515 self.canvas.print_figure(fname, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\matplotlib\backend_bases.py:2289, in FigureCanvasBase.print_figure(self, filename, dpi, facecolor, edgecolor, orientation, format, bbox_inches, pad_inches, bbox_extra_artists, backend, **kwargs)
   2285 try:
   2286     # _get_renderer may change the figure dpi (as vector formats
   2287     # force the figure dpi to 72), so we need to set it again here.
   2288     with cbook._setattr_cm(self.figure, dpi=dpi):
-> 2289         result = print_method(
   2290             filename,
   2291             facecolor=facecolor,
   2292             edgecolor=edgecolor,
   2293             orientation=orientation,
   2294             bbox_inches_restore=_bbox_inches_restore,
   2295             **kwargs)
   2296 finally:
   2297     if bbox_inches and restore_bbox:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\matplotlib\backend_bases.py:2146, in FigureCanvasBase._switch_canvas_and_return_print_method.<locals>.<lambda>(*args, **kwargs)
   2142     optional_kws = {  # Passed by print_figure for other renderers.
   2143         "dpi", "facecolor", "edgecolor", "orientation",
   2144         "bbox_inches_restore"}
   2145     skip = optional_kws - {*inspect.signature(meth).parameters}
-> 2146     print_method = functools.wraps(meth)(lambda *args, **kwargs: meth(
   2147         *args, **{k: v for k, v in kwargs.items() if k not in skip}))
   2148 else:  # Let third-parties do as they see fit.
   2149     print_method = meth

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\matplotlib\backends\backend_agg.py:537, in FigureCanvasAgg.print_png(self, filename_or_obj, metadata, pil_kwargs)
    490 def print_png(self, filename_or_obj, *, metadata=None, pil_kwargs=None):
    491     """
    492     Write the figure to a PNG file.
    493 
   (...)
    535         *metadata*, including the default 'Software' key.
    536     """
--> 537     self._print_pil(filename_or_obj, "png", pil_kwargs, metadata)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\matplotlib\backends\backend_agg.py:486, in FigureCanvasAgg._print_pil(self, filename_or_obj, fmt, pil_kwargs, metadata)
    481 """
    482 Draw the canvas, then save it using `.image.imsave` (to which
    483 *pil_kwargs* and *metadata* are forwarded).
    484 """
    485 FigureCanvasAgg.draw(self)
--> 486 mpl.image.imsave(
    487     filename_or_obj, self.buffer_rgba(), format=fmt, origin="upper",
    488     dpi=self.figure.dpi, metadata=metadata, pil_kwargs=pil_kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\matplotlib\image.py:1729, in imsave(fname, arr, vmin, vmax, cmap, format, origin, dpi, metadata, pil_kwargs)
   1727 pil_kwargs.setdefault("format", format)
   1728 pil_kwargs.setdefault("dpi", (dpi, dpi))
-> 1729 image.save(fname, **pil_kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\PIL\Image.py:2706, in Image.save(self, fp, format, **params)
   2704         fp = builtins.open(filename, "r+b")
   2705     else:
-> 2706         fp = builtins.open(filename, "w+b")
   2707 else:
   2708     fp = cast(IO[bytes], fp)

FileNotFoundError: [Errno 2] No such file or directory: 'reports/l42_tips.png'
os.makedirs("output/pandas/l42_reports", exist_ok=True)
fig.savefig("output/pandas/l42_reports/l42_tips.png", dpi=100)
print(os.path.exists("output/pandas/l42_reports/l42_tips.png"))
True

Mini-project: do weekend diners tip better?#

Plan: label the days, compare dollars, then compare percentages.

weekend = df["day"].isin(["Sat", "Sun"])
df["day_type"] = weekend.map({True: "Weekend", False: "Weekday"})
cols = ["tip", "total_bill", "tip_pct"]
summary = df.groupby("day_type")[cols].mean().round(2)
summary
tip total_bill tip_pct
day_type
Weekday 2.76 17.56 16.33
Weekend 3.12 20.89 15.96
fig, (left, right) = plt.subplots(1, 2, figsize=(10, 4))
summary["tip"].plot(kind="bar", ax=left, rot=0, color="teal")
sns.boxplot(data=df, x="day_type", y="tip_pct", ax=right,
            order=["Weekday", "Weekend"])
left.set(title="Average tip ($)", xlabel="", ylabel="Tip ($)")
right.set(title="Tip as % of bill", xlabel="", ylabel="%")
right.axhline(15, color="red", linestyle="--")
fig.tight_layout()
plt.show()
No description has been provided for this image
df.groupby("day_type")["tip_pct"].median().round(2)
day_type
Weekday    15.38
Weekend    15.56
Name: tip_pct, dtype: float64
gap = summary.loc["Weekend"] - summary.loc["Weekday"]
print(f"Extra tip: ${gap['tip']:.2f}")
print(f"Extra bill: ${gap['total_bill']:.2f}")
print(f"Change in tip share: {gap['tip_pct']:.2f} points")
Extra tip: $0.36
Extra bill: $3.33
Change in tip share: -0.37 points

Recap#

  • pandas .plot() returns a Matplotlib axes: keep it as ax
  • plt.subplots() plus ax= puts several charts in one figure
  • Seaborn takes data=df and column names; hue splits by category
  • Add axhline, annotate and themes with Matplotlib and Seaborn
  • fig.savefig() writes a file, into a folder that must exist

Challenges#

  1. Two side-by-side charts: bills per party size and avg tip_pct.
  2. Seaborn scatter of tip_pct against total_bill, coloured by day.
  3. Save your favourite chart to output/pandas and read it back.

What's next#

Lesson 43: Data Correlation and Heatmaps Measure how columns move together and map it all in one picture.

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.