Mathew K Analytics

Lesson 44 · Pandas Full Course

Interactive Charts with Plotly and Pandas (Hover, Animate, Share) | Pandas Course #44

In this lesson you'll be able to: Turn a DataFrame into an interactive chart with plotly.express Shape data with pandas so Plotly can read it Pack four…

What you'll learn

Datasets used in this lesson

Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.

📓 Full notebook

Download .ipynb

Pandas 44: Building Interactive Visuals with Plotly and Pandas#

In this lesson you'll be able to:

  • Turn a DataFrame into an interactive chart with plotly.express
  • Shape data with pandas so Plotly can read it
  • Pack four columns into one bubble chart
  • Split charts into panels and animate them over time
  • Save charts as HTML and PNG files to share Dataset: Gapminder (data/pandas/gapminder.csv)

Setup: pip install plotly kaleido In this video each chart appears as an image; on your machine you can hover and zoom.

Warm-up: wealth and health#

Lesson 43 measured correlation. One number before we plot.

import pandas as pd
df = pd.read_csv("data/pandas/gapminder.csv")
print(df.shape)
df.head(3)
(1704, 6)
country year pop continent lifeExp gdpPercap
0 Afghanistan 1952 8425333.0 Asia 28.801 779.445314
1 Afghanistan 1957 9240934.0 Asia 30.332 820.853030
2 Afghanistan 1962 10267083.0 Asia 31.997 853.100710
latest = df[df["year"] == 2007]
pair = latest[["gdpPercap", "lifeExp"]]
print(round(pair.corr(method="spearman").iloc[0, 1], 2))
0.86

Your first Plotly chart from a DataFrame#

Problem: a chart readers can explore, not just look at.

import plotly.express as px
japan = df[df["country"] == "Japan"]
fig = px.line(japan, x="year", y="lifeExp", markers=True,
              title="Life expectancy in Japan, 1952-2007")
fig.update_layout(xaxis_title="Year",
                  yaxis_title="Life expectancy (years)")
fig.show()
No description has been provided for this image
first, last = japan["lifeExp"].iloc[[0, -1]]
print(round(first, 1), round(last, 1), round(last - first, 1))
63.0 82.6 19.6
picks = ["Japan", "China", "India", "Rwanda", "South Africa"]
five = df[df["country"].isin(picks)]
fig = px.line(five, x="year", y="lifeExp", color="country",
              labels={"lifeExp": "Life expectancy", "year": "Year"},
              title="Five countries, five different paths")
fig.show()
No description has been provided for this image

Quick check: in which year did Rwanda hit its lowest point?

rwanda = df[df["country"] == "Rwanda"].set_index("year")["lifeExp"]
print(rwanda.idxmin(), round(rwanda.min(), 1))
1992 23.6
fig = px.line(five, x="year", y="lifeExp", color="country",
              hover_data={"pop": ":,", "gdpPercap": ":,.0f"},
              title="Hover any point for population and income")
fig.update_layout(xaxis_title="Year",
                  yaxis_title="Life expectancy (years)")
fig.show()
No description has been provided for this image
print(fig.data[0].name, fig.data[0].customdata.shape)
China (12, 2)

Shape the data with pandas first#

Problem: Plotly needs columns; groupby results hide keys in the index.

by_cont = latest.groupby("continent")["lifeExp"].mean().round(1)
by_cont
continent
Africa      54.8
Americas    73.6
Asia        70.7
Europe      77.6
Oceania     80.7
Name: lifeExp, dtype: float64

Common mistake: plotting a groupby result without reset_index

px.bar(by_cont, x="continent", y="lifeExp")
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[11], line 1
----> 1 px.bar(by_cont, x="continent", y="lifeExp")

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\plotly\express\_chart_types.py:390, in bar(data_frame, x, y, color, pattern_shape, facet_row, facet_col, facet_col_wrap, facet_row_spacing, facet_col_spacing, hover_name, hover_data, custom_data, text, base, error_x, error_x_minus, error_y, error_y_minus, animation_frame, animation_group, category_orders, labels, color_discrete_sequence, color_discrete_map, color_continuous_scale, pattern_shape_sequence, pattern_shape_map, range_color, color_continuous_midpoint, opacity, orientation, barmode, log_x, log_y, range_x, range_y, text_auto, title, subtitle, template, width, height)
    341 def bar(
    342     data_frame=None,
    343     x=None,
   (...)
    384     height=None,
    385 ) -> go.Figure:
    386     """
    387     In a bar plot, each row of `data_frame` is represented as a rectangular
    388     mark.
    389     """
--> 390     return make_figure(
    391         args=locals(),
    392         constructor=go.Bar,
    393         trace_patch=dict(textposition="auto"),
    394         layout_patch=dict(barmode=barmode),
    395     )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\plotly\express\_core.py:2533, in make_figure(args, constructor, trace_patch, layout_patch)
   2530 user_provided_colorscale = args.get("color_continuous_scale") is not None
   2531 apply_default_cascade(args, constructor=constructor)
-> 2533 args = build_dataframe(args, constructor)
   2534 if constructor in [go.Treemap, go.Sunburst, go.Icicle] and args["path"] is not None:
   2535     args = process_dataframe_hierarchy(args)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\plotly\express\_core.py:1779, in build_dataframe(args, constructor)
   1776     args["color"] = None
   1777 # now that things have been prepped, we do the systematic rewriting of `args`
-> 1779 df_output, wide_id_vars = process_args_into_dataframe(
   1780     args,
   1781     wide_mode,
   1782     var_name,
   1783     value_name,
   1784     is_pd_like,
   1785     native_namespace,
   1786 )
   1787 df_output: nw.DataFrame
   1788 # now that `df_output` exists and `args` contains only references, we complete
   1789 # the special-case and wide-mode handling by further rewriting args and/or mutating
   1790 # df_output

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\plotly\express\_core.py:1380, in process_args_into_dataframe(args, wide_mode, var_name, value_name, is_pd_like, native_namespace)
   1378         if argument == "index":
   1379             err_msg += "\n To use the index, pass it in directly as `df.index`."
-> 1380         raise ValueError(err_msg)
   1381 elif length and (actual_len := len(df_input)) != length:
   1382     raise ValueError(
   1383         "All arguments should have the same length. "
   1384         "The length of column argument `df[%s]` is %d, whereas the "
   (...)
   1391         )
   1392     )

ValueError: Value of 'x' is not the name of a column in 'data_frame'. Expected one of ['lifeExp'] but received: continent
table = by_cont.reset_index()
fig = px.bar(table, x="continent", y="lifeExp", text_auto=True,
             color="continent",
             title="Average life expectancy by continent, 2007")
fig.update_layout(xaxis_title="Continent",
                  yaxis_title="Life expectancy (years)",
                  showlegend=False)
fig.show()
No description has been provided for this image
counts = latest["continent"].value_counts().reset_index()
print(counts.columns.tolist())
fig = px.pie(counts, names="continent", values="count",
             title="Countries per continent")
fig.show()
['continent', 'count']
No description has been provided for this image

Your turn#

Bar chart of the median GDP per person by continent in 2007.

gdp = latest.groupby("continent")["gdpPercap"].median()
gdp = gdp.round().reset_index()
fig = px.bar(gdp, x="continent", y="gdpPercap", text_auto=True,
             title="Median GDP per person by continent, 2007")
fig.update_layout(xaxis_title="Continent",
                  yaxis_title="GDP per person (US$)")
fig.show()
No description has been provided for this image

Bubble charts: four columns in one picture#

Problem: compare wealth, health, size and region at once.

fig = px.scatter(latest, x="gdpPercap", y="lifeExp",
                 size="pop", color="continent",
                 hover_name="country", log_x=True, size_max=50,
                 title="Wealth and health, 2007")
fig.update_layout(xaxis_title="GDP per person (log scale)",
                  yaxis_title="Life expectancy (years)")
fig.show()
No description has been provided for this image

Quick check: which three countries are furthest right?

cols = ["country", "continent", "gdpPercap", "lifeExp"]
latest.nlargest(3, "gdpPercap")[cols].round(1)
country continent gdpPercap lifeExp
1151 Norway Europe 49357.2 80.2
863 Kuwait Asia 47307.0 77.6
1367 Singapore Asia 47143.2 80.0
latest.nsmallest(3, "lifeExp")[cols].round(1)
country continent gdpPercap lifeExp
1463 Swaziland Africa 4513.5 39.6
1043 Mozambique Africa 823.7 42.1
1691 Zambia Africa 1271.2 42.4

Your turn#

Draw the same bubble chart for 1952.

y1952 = df[df["year"] == 1952]
fig = px.scatter(y1952, x="gdpPercap", y="lifeExp", size="pop",
                 color="continent", hover_name="country",
                 log_x=True, size_max=50,
                 title="Wealth and health, 1952")
fig.update_layout(xaxis_title="GDP per person (log scale)",
                  yaxis_title="Life expectancy (years)")
fig.show()
No description has been provided for this image
print(round(y1952["lifeExp"].median(), 1))
print(round(latest["lifeExp"].median(), 1))
45.1
71.9

Many lines from long data#

Problem: one line per group, built from a tidy table.

keys = ["continent", "year"]
trend = df.groupby(keys, as_index=False)["lifeExp"].mean()
print(trend.shape)
trend.head(3).round(1)
(60, 3)
continent year lifeExp
0 Africa 1952 39.1
1 Africa 1957 41.3
2 Africa 1962 43.3

Common mistake: a column name with the wrong capitals

px.line(trend, x="year", y="lifeexp", color="continent")
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[21], line 1
----> 1 px.line(trend, x="year", y="lifeexp", color="continent")

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\plotly\express\_chart_types.py:279, in line(data_frame, x, y, line_group, color, line_dash, symbol, hover_name, hover_data, custom_data, text, facet_row, facet_col, facet_col_wrap, facet_row_spacing, facet_col_spacing, error_x, error_x_minus, error_y, error_y_minus, animation_frame, animation_group, category_orders, labels, orientation, color_discrete_sequence, color_discrete_map, line_dash_sequence, line_dash_map, symbol_sequence, symbol_map, markers, log_x, log_y, range_x, range_y, line_shape, render_mode, title, subtitle, template, width, height)
    230 def line(
    231     data_frame=None,
    232     x=None,
   (...)
    273     height=None,
    274 ) -> go.Figure:
    275     """
    276     In a 2D line plot, each row of `data_frame` is represented as a vertex of
    277     a polyline mark in 2D space.
    278     """
--> 279     return make_figure(args=locals(), constructor=go.Scatter)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\plotly\express\_core.py:2533, in make_figure(args, constructor, trace_patch, layout_patch)
   2530 user_provided_colorscale = args.get("color_continuous_scale") is not None
   2531 apply_default_cascade(args, constructor=constructor)
-> 2533 args = build_dataframe(args, constructor)
   2534 if constructor in [go.Treemap, go.Sunburst, go.Icicle] and args["path"] is not None:
   2535     args = process_dataframe_hierarchy(args)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\plotly\express\_core.py:1779, in build_dataframe(args, constructor)
   1776     args["color"] = None
   1777 # now that things have been prepped, we do the systematic rewriting of `args`
-> 1779 df_output, wide_id_vars = process_args_into_dataframe(
   1780     args,
   1781     wide_mode,
   1782     var_name,
   1783     value_name,
   1784     is_pd_like,
   1785     native_namespace,
   1786 )
   1787 df_output: nw.DataFrame
   1788 # now that `df_output` exists and `args` contains only references, we complete
   1789 # the special-case and wide-mode handling by further rewriting args and/or mutating
   1790 # df_output

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\plotly\express\_core.py:1380, in process_args_into_dataframe(args, wide_mode, var_name, value_name, is_pd_like, native_namespace)
   1378         if argument == "index":
   1379             err_msg += "\n To use the index, pass it in directly as `df.index`."
-> 1380         raise ValueError(err_msg)
   1381 elif length and (actual_len := len(df_input)) != length:
   1382     raise ValueError(
   1383         "All arguments should have the same length. "
   1384         "The length of column argument `df[%s]` is %d, whereas the "
   (...)
   1391         )
   1392     )

ValueError: Value of 'y' is not the name of a column in 'data_frame'. Expected one of ['continent', 'year', 'lifeExp'] but received: lifeexp
fig = px.line(trend, x="year", y="lifeExp", color="continent",
              markers=True,
              title="Average life expectancy by continent")
fig.update_layout(xaxis_title="Year",
                  yaxis_title="Life expectancy (years)")
fig.show()
No description has been provided for this image
wide = trend.pivot(index="year", columns="continent",
                   values="lifeExp")
fig = px.line(wide, title="Same chart from a wide table")
fig.update_layout(xaxis_title="Year", yaxis_title="Years")
fig.show()
No description has been provided for this image
fig = px.line(trend, x="year", y="lifeExp", color="continent",
              template="simple_white",
              title="Life expectancy by continent, restyled")
fig.update_traces(line_width=3)
fig.update_layout(hovermode="x unified", xaxis_title="Year",
                  yaxis_title="Life expectancy (years)")
fig.show()
No description has been provided for this image

Panels and animation#

Problem: compare groups or years without cramming one chart.

fig = px.histogram(latest, x="lifeExp", facet_col="continent",
                   nbins=10, title="Life expectancy, 2007")
fig.update_layout(yaxis_title="Countries")
fig.update_xaxes(title_text="Years")
fig.for_each_annotation(lambda a: a.update(text=a.text.split("=")[1]))
fig.show()
No description has been provided for this image
latest.groupby("continent")["lifeExp"].agg(["min", "max"]).round(1)
min max
continent
Africa 39.6 76.4
Americas 60.9 80.7
Asia 43.8 82.6
Europe 71.8 81.8
Oceania 80.2 81.2
fig = px.scatter(df, x="gdpPercap", y="lifeExp", size="pop",
                 color="continent", hover_name="country",
                 animation_frame="year", log_x=True, size_max=50,
                 range_x=[200, 120000], range_y=[20, 90],
                 labels={"gdpPercap": "GDP per person",
                         "lifeExp": "Life expectancy"},
                 title="Wealth and health, 1952 to 2007")
fig.show()
No description has been provided for this image
print(len(fig.frames))
print(fig.frames[0].name, fig.frames[-1].name)
12
1952 2007

Your turn#

Draw one line chart per continent with facet_col, using trend.

fig = px.line(trend, x="year", y="lifeExp", facet_col="continent",
              title="Each continent in its own panel")
fig.update_layout(yaxis_title="Life expectancy (years)")
fig.update_xaxes(title_text="Year")
fig.show()
No description has been provided for this image

Save and share: HTML and PNG#

Problem: send an interactive chart to someone without Python.

import os
os.makedirs("output/pandas", exist_ok=True)
html_path = "output/pandas/l44_life_by_continent.html"
fig.write_html(html_path)
print(os.path.exists(html_path))
True
with open(html_path, encoding="utf-8") as f:
    page = f.read()
print(len(page) > 1_000_000)
print("Each continent in its own panel" in page)
True
True
png_path = "output/pandas/l44_life_by_continent.png"
fig.write_image(png_path, width=900, height=400)
import matplotlib.pyplot as plt
print(plt.imread(png_path).shape)
(400, 900, 4)

Quick check: which file is bigger, the HTML page or the PNG?

html_size = os.path.getsize(html_path)
png_size = os.path.getsize(png_path)
print(html_size > 10 * png_size)
True

Common mistake: writing into a folder that does not exist

fig.write_html("charts/l44_life.html")
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[34], line 1
----> 1 fig.write_html("charts/l44_life.html")

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\plotly\basedatatypes.py:3709, in BaseFigure.write_html(self, *args, **kwargs)
   3606 """
   3607 Write a figure to an HTML file representation
   3608 
   (...)
   3705 None
   3706 """
   3707 import plotly.io as pio
-> 3709 return pio.write_html(self, *args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\plotly\io\_html.py:497, in write_html(fig, file, config, auto_play, include_plotlyjs, include_mathjax, post_script, full_html, animation_opts, validate, default_width, default_height, auto_open, div_id)
    494 # Write HTML string
    495 if path is not None:
    496     # To use a different file encoding, pass a file descriptor
--> 497     path.write_text(html_str, "utf-8")
    498 else:
    499     file.write(html_str)

File ~\AppData\Local\Programs\Python\Python312\Lib\pathlib.py:1047, in Path.write_text(self, data, encoding, errors, newline)
   1044     raise TypeError('data must be str, not %s' %
   1045                     data.__class__.__name__)
   1046 encoding = io.text_encoding(encoding)
-> 1047 with self.open(mode='w', encoding=encoding, errors=errors, newline=newline) as f:
   1048     return f.write(data)

File ~\AppData\Local\Programs\Python\Python312\Lib\pathlib.py:1013, in Path.open(self, mode, buffering, encoding, errors, newline)
   1011 if "b" not in mode:
   1012     encoding = io.text_encoding(encoding)
-> 1013 return io.open(self, mode, buffering, encoding, errors, newline)

FileNotFoundError: [Errno 2] No such file or directory: 'charts\\l44_life.html'
os.makedirs("output/pandas/l44_charts", exist_ok=True)
fig.write_html("output/pandas/l44_charts/l44_life.html")
print(os.path.exists("output/pandas/l44_charts/l44_life.html"))
True

Mini-project: who caught up?#

Measure each continent's gap to Europe in 1952 and in 2007.

gap = wide.sub(wide["Europe"], axis=0)
gap = gap.drop(columns=["Europe", "Oceania"])
gap.loc[[1952, 2007]].round(1)
continent Africa Americas Asia
year
1952 -25.3 -11.1 -18.1
2007 -22.8 -4.0 -6.9
closed = (gap.loc[2007] - gap.loc[1952]).round(1)
closed.sort_values(ascending=False)
continent
Asia        11.2
Americas     7.1
Africa       2.4
dtype: float64
bars = closed.rename("years").reset_index()
fig = px.bar(bars, x="continent", y="years", text_auto=True,
             title="Years of the gap closed, 1952 to 2007")
fig.update_layout(xaxis_title="Continent",
                  yaxis_title="Years closed")
fig.show()
No description has been provided for this image
long_gap = gap.reset_index().melt(id_vars="year",
                                  var_name="continent",
                                  value_name="gap")
fig = px.line(long_gap, x="year", y="gap", color="continent",
              markers=True, title="Life expectancy gap to Europe")
fig.add_hline(y=0, line_dash="dash")
fig.update_layout(xaxis_title="Year",
                  yaxis_title="Years behind Europe")
fig.show()
No description has been provided for this image
best = closed.idxmax()
worst = closed.idxmin()
print(f"Most catch-up: {best}, {closed[best]} years")
print(f"Least catch-up: {worst}, {closed[worst]} years")
print(f"{worst} gap in 2007: {gap.loc[2007, worst]:.1f} years")
Most catch-up: Asia, 11.2 years
Least catch-up: Africa, 2.4 years
Africa gap in 2007: -22.8 years

Recap#

  • px.line, px.bar and px.scatter take a DataFrame and column names
  • Use reset_index() or as_index=False so keys become columns
  • color, size and hover_name pack extra columns into one chart
  • facet_col splits panels; animation_frame adds a play button
  • write_html shares interactivity; write_image needs Kaleido

Challenges#

  1. Animated bar chart of average GDP per person by continent.
  2. Bubble chart of Asia only, 2007, coloured by country.
  3. Save your favourite chart as both HTML and PNG.

What's next#

Lesson 45: Categorical Data and Memory Optimization Make big tables smaller and faster with the right data types.

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.