Lesson 9 · Real-World Data Analytics
Web Scraping in Python with Requests & BeautifulSoup: Price Tracker | Data Analytics #9
Learn web scraping safely: we build a product listing page from real retail prices, serve it locally, then scrape it with requests and BeautifulSoup the way…
- CourseReal-World Data Analytics
- Lesson9 of 100
- FormatJupyter notebook · 30 code cells
- Data1 dataset
What you'll learn
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
- online_retail_clean.csv preview36.2 MB
📓 Full notebook
Download .ipynbData Analytics 100, Video 9: Scraping Real Product Prices for Competitive Analysis#
- Video nine of the hundred-video real-world data analytics series.
- Actually scraping a real live web page with requests and BeautifulSoup, then comparing real prices across real time periods.
- Let's get into it.
This lesson is for analysts who want to learn the mechanics of web scraping for price monitoring: fetching a page, parsing it, following pagination and turning the results into a table you can compare against your own data. The business question is a practical one: how have prices on a product listing moved compared with an earlier benchmark, and which moves deserve a closer look?
To keep things safe and repeatable, you scrape a real page served locally inside the notebook rather than a third-party site. The listing is built from online_retail_clean.csv, the cleaned online retail data from earlier lessons, using StockCode, Description, UnitPrice, Quantity and InvoiceDate. You control the page, so you can verify every scraped value.
In this lesson you will:
- split the data into an earlier and a later period and build a paginated HTML listing
- check
robots.txt, then fetch pages withrequestsand parse them with BeautifulSoup - loop through every page and build a clean pandas DataFrame from the results
- compare scraped prices with the earlier period, flag big moves and estimate revenue impact
You should be comfortable with pandas groupby and basic Python loops and functions. No HTML knowledge is assumed.
Part 1: Why This Runs Against a Real Local Page#
Scraping a live competitor site from a tutorial would be fragile (the site can change) and could break its terms of use. Instead, this notebook serves its own page locally, so the scraping code is identical to the real thing but fully under your control. This step imports requests for fetching pages and BeautifulSoup from bs4 for parsing HTML, then loads the cleaned retail data. The output (391150, 9) confirms 391,150 rows and 9 columns are available to build the listing from.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import requests
from bs4 import BeautifulSoup
clean = pd.read_csv('online_retail_clean.csv', parse_dates=['InvoiceDate'])
clean.shape
Part 2: Real Two Time Periods to Compare#
Price monitoring always compares a current price with a benchmark. Here the benchmark is the shop's own earlier trading. The code splits the data at 1 June 2011 with boolean comparisons on InvoiceDate. The output shows 143,150 rows before the split and 248,000 after it. The later period is larger partly because it covers the busy autumn and pre-Christmas season, so keep in mind that seasonal factors may affect prices too.
split_date = pd.Timestamp('2011-06-01')
period_earlier = clean[clean['InvoiceDate'] < split_date]
period_later = clean[clean['InvoiceDate'] >= split_date]
period_earlier.shape[0], period_later.shape[0]
Part 3: Real Average Price per Product, Each Period#
To compare prices you need one price per product per period. The code groups each period by StockCode and takes the mean UnitPrice, builds a description lookup from each product's most common description, and uses .index.intersection(...) to keep only products sold in both periods. The output shows 2,700 products in common. Note that a simple mean of row prices is not weighted by quantity, so if bulk orders at a lower price tier become more or less common, the average can move without the list price changing.
price_earlier = period_earlier.groupby('StockCode')['UnitPrice'].mean()
price_later = period_later.groupby('StockCode')['UnitPrice'].mean()
desc_lookup = clean.groupby('StockCode')['Description'].agg(lambda s: s.mode().iloc[0])
common_products = price_earlier.index.intersection(price_later.index)
len(common_products)
Part 4: Picking the Real Top Products to List#
A real listing page shows a limited set of products, so the code picks the 30 best sellers from the later period. It sums later-period Quantity per product, uses .reindex(common_products) to restrict to products present in both periods, sorts in descending order and keeps the top 30 stock codes. The output confirms 30 products. Choosing high-volume items also makes sense for monitoring, since price changes on best sellers have the biggest effect on revenue.
qty_later = period_later.groupby('StockCode')['Quantity'].sum()
listing_products = qty_later.reindex(common_products).sort_values(ascending=False).head(30).index
len(listing_products)
Part 5: Building the Real HTML Listing Page#
This function writes the HTML for one page of the listing. Each product becomes a div with class product, a data-sku attribute holding the stock code, and two span tags for the name and the later-period price formatted as GBP 2.05. The expression -(-n // size) is a trick for rounding a division up, giving the total number of pages, and a Next link is added on every page except the last. There is no output yet. The class names chosen here are exactly what the scraper will search for later.
def render_listing_page(page_num, page_size=10):
start = (page_num - 1) * page_size
items = list(listing_products[start:start + page_size])
total_pages = -(-len(listing_products) // page_size)
cards = ''
for sc in items:
cards += f'<div class="product" data-sku="{sc}"><span class="name">{desc_lookup[sc]}</span><span class="price">GBP {price_later[sc]:.2f}</span></div>'
nav = ''
if page_num < total_pages:
nav += f'<a class="next" href="/listing?page={page_num + 1}">Next</a>'
return f'<html><body><h1>Product Listing</h1><div class="catalog">{cards}</div><div class="pagination">{nav}</div></body></html>'
Part 6: Real HTTP Server Setup#
To scrape over HTTP you need a web server, and Python's standard library includes one. This step imports threading so the server can run in the background, the http.server classes for handling requests, and urlparse and parse_qs for reading the page number from a URL. It also defines a robots.txt text. robots.txt is a file websites use to tell automated visitors which paths they may access; here it allows /listing. Nothing is printed at this stage.
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import urlparse, parse_qs
ROBOTS_TXT = 'User-agent: *' + chr(10) + 'Allow: /listing' + chr(10)
Part 7: Real Request Handler#
The request handler decides what the server sends back for each URL. Its do_GET method parses the path: /listing returns the rendered page for the requested page number (defaulting to 1), /robots.txt returns the rules, and anything else gets a 404 Not Found page. The helper _send_html sets the status code and a Content-Type header before writing the body, and log_message is overridden to keep the notebook output quiet. There is no output, as this only defines the class.
class ListingHandler(BaseHTTPRequestHandler):
def _send_html(self, status, html):
body = html.encode('utf-8')
self.send_response(status)
self.send_header('Content-Type', 'text/html; charset=utf-8')
self.end_headers()
self.wfile.write(body)
def log_message(self, format, *args):
pass
def do_GET(self):
parsed = urlparse(self.path)
query = parse_qs(parsed.query)
if parsed.path == '/listing':
page = int(query.get('page', ['1'])[0])
self._send_html(200, render_listing_page(page))
elif parsed.path == '/robots.txt':
self._send_html(200, ROBOTS_TXT)
else:
self._send_html(404, '<html><body><h1>Not Found</h1></body></html>')
Part 8: Starting the Real Server#
Now the server starts. Passing port 0 asks the operating system for any free port, which avoids clashes if something else is already running. serve_forever runs in a separate thread with daemon=True, so the notebook stays usable and the thread will not block Python from exiting. The output shows the page live at http://127.0.0.1:51619. Your port number will almost certainly differ each time you run it, which is why the code builds BASE_URL from server.server_port instead of hardcoding it.
server = ThreadingHTTPServer(('127.0.0.1', 0), ListingHandler)
server_thread = threading.Thread(target=server.serve_forever, daemon=True)
server_thread.start()
BASE_URL = f'http://127.0.0.1:{server.server_port}'
print(f'Real listing page live at {BASE_URL}')
Part 9: Real Politeness Check, robots.txt#
Checking robots.txt before scraping is a basic professional courtesy and, on many sites, part of the terms of use. The code fetches it with requests.get and shows the text. The output reads User-agent: * followed by Allow: /listing, meaning every crawler may access the listing path. On a real site you would also look for Disallow lines and any crawl-delay instruction, and respect them.
robots_response = requests.get(f'{BASE_URL}/robots.txt')
robots_response.text
Part 10: Real First Page Request#
The first request confirms the page is reachable before you write any parsing code. requests.get fetches the listing, and the code checks the status code and the length of the returned HTML. Only the last expression is displayed, so the output shows just the length: 1,457 characters. Add print(response.status_code) if you want to see the status too, which should be 200 for success. Checking the status before parsing avoids confusing errors when a site returns an error page instead of data.
response = requests.get(f'{BASE_URL}/listing')
response.status_code
len(response.text)
Part 11: Real Parsing with BeautifulSoup#
Raw HTML is just text, so you need a parser to navigate it. BeautifulSoup(response.text, 'lxml') builds a searchable tree using the fast lxml parser. soup.h1.text reads the page heading (not displayed, as it is not the last line), and find_all('div', class_='product') returns every product card. Note the underscore in class_, needed because class is a reserved word in Python. The output shows 10 products on the first page, matching the page size.
soup = BeautifulSoup(response.text, 'lxml')
soup.h1.text
len(soup.find_all('div', class_='product'))
Part 12: Real Extraction Loop Across All Pages#
Most listings are split across pages, so a scraper has to keep going until there is nothing left. The while True loop requests each page using params={'page': page}, which requests turns into ?page=1 and so on, parses it, and stops with break when a page has no product cards. For each card it reads the data-sku attribute, the name text and the price text, strips GBP and converts the price with float. The output shows 30 rows scraped, the full listing. On a real site, add a short time.sleep between requests to avoid overloading the server.
scraped_rows = []
page = 1
while True:
resp = requests.get(f'{BASE_URL}/listing', params={'page': page})
page_soup = BeautifulSoup(resp.text, 'lxml')
cards = page_soup.find_all('div', class_='product')
if not cards:
break
for card in cards:
sku = card['data-sku']
name = card.find('span', class_='name').text
price_text = card.find('span', class_='price').text.replace('GBP', '').strip()
scraped_rows.append({'StockCode': sku, 'ScrapedName': name, 'ScrapedPrice': float(price_text)})
page += 1
len(scraped_rows)
Part 13: Building the Real Scraped DataFrame#
A list of dictionaries converts neatly into a DataFrame, one row per dictionary. The code builds the DataFrame and sets StockCode as the index so it can be aligned with the period price tables. The output shows the first five rows, including POPCORN HOLDER at 0.84 GBP and JUMBO BAG RED RETROSPOT at 2.05. These are now ordinary pandas data, ready for comparison.
scraped_df = pd.DataFrame(scraped_rows).set_index('StockCode')
scraped_df.head(5)
Part 14: Real Sanity Check on the Scrape#
Scrapers fail quietly: a changed class name or a stray character can produce wrong numbers without any error. Because you built the page, you can check every scraped price against the source. The code aligns price_later to the scraped rows with .reindex, rounds both to two decimals and compares them. .all() returns True only if every comparison is True. The output True confirms all 30 prices were scraped correctly. On a real site, spot-check a few products by hand instead.
check = scraped_df['ScrapedPrice'].round(2) == price_later.reindex(scraped_df.index).round(2)
check.all()
Part 15: Comparing Against the Real Earlier Period#
With current prices in hand, the next step is to compare them with the earlier benchmark. The code adds each product's earlier average price with .reindex and computes the percentage change as new minus old, divided by old, times 100. The output shows the first five: JUMBO BAG RED RETROSPOT went from 1.95 to 2.05 GBP, up 5.12%, while RABBIT NIGHT LIGHT fell 1.48%. The tiny -0.01% for the bird ornament comes from unrounded averages, not a real price cut.
scraped_df['EarlierPrice'] = price_earlier.reindex(scraped_df.index)
scraped_df['PctChange'] = (scraped_df['ScrapedPrice'] - scraped_df['EarlierPrice']) / scraped_df['EarlierPrice'] * 100
scraped_df[['EarlierPrice', 'ScrapedPrice', 'PctChange']].round(2).head(5)
Part 16: Real Biggest Price Increases#
Sorting by percentage change brings the biggest rises to the top, which is usually where a pricing team looks first. The code sorts with ascending=False and shows the top five. The output is led by BROCADE RING PURSE, up 37.46%, VICTORIAN GLASS HANGING T-LIGHT, up 31.67%, and PACK OF 12 LONDON TISSUES, up 25.42%. After those, the changes drop to single figures. Large jumps in an average price can reflect a change in the mix of order sizes as well as a list price change, so check before acting.
scraped_df.sort_values('PctChange', ascending=False)[['ScrapedName', 'PctChange']].head(5).round(2)
Part 17: Real Biggest Price Decreases#
Price falls matter as much as rises, since they can signal clearance or a competitor response. The code sorts in ascending order and shows the five lowest changes. The output shows two Christmas items, WOODEN STAR CHRISTMAS SCANDINAVIAN down 57.65% and WOODEN HEART CHRISTMAS SCANDINAVIAN down 55.29%, followed by three products with tiny decreases of 1.48% or less. Big falls on seasonal decorations are consistent with end-of-season or bulk pricing, a useful reminder to think about timing when reading price changes.
scraped_df.sort_values('PctChange')[['ScrapedName', 'PctChange']].head(5).round(2)
Part 18: Visualizing Real Price Changes#
A bar chart of every product's change gives a quick overview that a table cannot. The code sorts by change, colours bars red for decreases and green for increases with a list comprehension, and shortens names to 25 characters with .str[:25] so labels fit. Open scraped_price_changes.png and look for the two long red bars at the bottom, the few long green bars at the top, and the many short green bars in between showing small rises across most of the listing.
plt.figure(figsize=(10, 7))
sorted_changes = scraped_df.sort_values('PctChange')
colors = ['crimson' if v < 0 else 'seagreen' for v in sorted_changes['PctChange']]
plt.barh(sorted_changes['ScrapedName'].str[:25], sorted_changes['PctChange'], color=colors)
plt.xlabel('Real Percentage Price Change')
plt.title('Real Price Change by Product, Scraped Listing vs Earlier Period')
plt.tight_layout()
plt.savefig('scraped_price_changes.png', dpi=120)
plt.close()
Part 19: Real Average Price Movement Overall#
A single average gives a headline for the whole listing. The code takes the mean of PctChange and prints it. The output says that across 30 scraped products, the average price moved 1.36% compared with the earlier period. Be careful with this figure: it gives every product equal weight and mixes a few very large changes with many small ones. The median, or a quantity-weighted average, would tell you more about the typical shopper's experience.
avg_pct_change = scraped_df['PctChange'].mean()
print(f'Across {len(scraped_df)} real scraped products, average price moved {round(avg_pct_change, 2)}% versus the earlier real period.')
Part 20: Flagging Real Products for Review#
Not every price move needs attention, so a simple threshold helps a team focus. The code keeps products whose absolute change, from .abs(), is greater than 5%, catching both rises and falls. The output shows 8 products flagged for review. Looking at the earlier tables, these include the three large rises, the two Christmas decorations with large falls, and smaller rises just over 5% such as JUMBO BAG RED RETROSPOT. Adjust the threshold to match how many items a team can realistically review.
flagged = scraped_df[scraped_df['PctChange'].abs() > 5]
flagged.shape[0]
Part 24: Real 404 Check#
Good scrapers handle missing pages gracefully, so it helps to see what a failed request looks like. The code requests a path the server does not know and checks the status code. The output is 404, the standard HTTP code for Not Found. requests does not raise an error for a 404 by default, so always check status_code, or call response.raise_for_status(), before parsing, otherwise you may end up parsing an error page as if it held data.
missing_response = requests.get(f'{BASE_URL}/does-not-exist')
missing_response.status_code
Part 21: Real Total Pages Scraped#
Reporting how much was scraped makes it easy to spot an incomplete run. The page variable is left one past the last page with products, because the loop only stops after fetching an empty page, so the code prints page - 1. The output reports 3 pages and 30 products, which matches 10 products per page. If a future run shows fewer pages than expected, check whether the site layout or pagination has changed.
print(f'Scraped {page - 1} real pages, {len(scraped_rows)} real products total, from the real live local listing.')
Part 22: Saving the Real Comparison Table#
Saving the comparison lets the pricing team review it in a spreadsheet. The code rounds to two decimals and writes scraped_price_comparison.csv, keeping StockCode as the index column. Reloading and comparing row counts gives True, so all 30 products were saved. In a real monitoring setup you would add a date column and append each run, building a price history over time.
scraped_df.round(2).to_csv('scraped_price_comparison.csv')
reloaded = pd.read_csv('scraped_price_comparison.csv')
reloaded.shape[0] == scraped_df.shape[0]
Part 23: Shutting Down the Real Server#
Any server you start should be stopped when you are done, otherwise it keeps a port open until the notebook kernel stops. server.shutdown() stops the serve_forever loop and server.server_close() releases the network socket. The output confirms the server shut down cleanly. Doing this explicitly is good practice whenever a notebook starts background processes.
server.shutdown()
server.server_close()
print('Real local listing server shut down cleanly.')
Part 25: Real Response Headers#
Response headers carry useful information about what the server sent back. The code reads the Content-Type header from the first response, which is still stored in the response object even though the server is now shut down. The output is text/html; charset=utf-8, confirming an HTML page in UTF-8 encoding. On real sites, checking this header helps you notice when a server returns JSON, a PDF or an error page instead of the HTML you expected.
response.headers['Content-Type']
Part 26: Real Count of Increases vs Decreases#
Counting rises and falls shows the overall direction of pricing. The code sums two boolean conditions, > 0 and < 0, to count products whose price went up or down. The output shows 25 increases and 5 decreases. So although the average change was a modest 1.36%, prices rose for the large majority of listed products, and the average was held down by the two large Christmas decoration falls.
n_increased = (scraped_df['PctChange'] > 0).sum()
n_decreased = (scraped_df['PctChange'] < 0).sum()
n_increased, n_decreased
Part 27: Visualizing the Real Distribution of Price Changes#
A histogram shows the shape of the price changes rather than individual products. The code plots PctChange in 15 bins and adds a dashed red line at zero with plt.axvline. Open price_change_distribution.png and look for a cluster of products just to the right of zero, a few outliers well to the right, and the two isolated bars far to the left. With only 30 products, each bar represents just a handful of items, so avoid reading too much into its exact shape.
plt.figure(figsize=(8, 5))
plt.hist(scraped_df['PctChange'], bins=15, color='steelblue', edgecolor='white')
plt.axvline(0, color='red', linestyle='--', label='Real no change')
plt.xlabel('Real Percentage Price Change')
plt.ylabel('Real Number of Products')
plt.title('Real Distribution of Scraped Price Changes')
plt.legend()
plt.tight_layout()
plt.savefig('price_change_distribution.png', dpi=120)
plt.close()
Part 28: Real Revenue Impact of the Price Changes#
Percentages do not show money, so this step estimates the revenue effect. For each product the code multiplies the price difference by the quantity sold in the later period, then sums. The output is about 7,958 GBP in total. This is a rough estimate: it assumes the same quantity would have sold at the earlier price, which ignores price elasticity, the way demand responds to price, covered in the price and discount lesson earlier in the series.
scraped_df['LaterQty'] = qty_later.reindex(scraped_df.index)
scraped_df['RevenueImpact'] = (scraped_df['ScrapedPrice'] - scraped_df['EarlierPrice']) * scraped_df['LaterQty']
round(scraped_df['RevenueImpact'].sum(), 2)
Part 29: Real Products with the Largest Real Revenue Impact#
Ranking by absolute revenue impact shows which price changes mattered most in money, whether up or down. The code sorts the absolute impact with .abs() and uses the resulting index order with .reindex. The output is led by VICTORIAN GLASS HANGING T-LIGHT at about +4,743 GBP, then the two Christmas decorations at about -4,654 and -4,428, and JUMBO BAG RED RETROSPOT at about +2,776. A modest 5.12% rise on a best seller can matter more than a 37% rise on a slower product.
scraped_df.reindex(scraped_df['RevenueImpact'].abs().sort_values(ascending=False).index)[['ScrapedName', 'PctChange', 'RevenueImpact']].head(5).round(2)
Part 30: Real Recap Print#
The recap gives a single sentence to share with a pricing or commercial team. The code prints the number of products compared and the total estimated revenue impact, using the format :,.0f for thousands separators. The output reads 30 products compared, with an estimated total revenue impact of about 7,958 GBP. Quote it alongside the assumption behind it, that quantities stay the same, so nobody mistakes it for a precise forecast.
print(f"Scraped and compared {len(scraped_df)} real products; estimated total revenue impact of price changes: {round(scraped_df['RevenueImpact'].sum(), 0):,.0f} GBP.")
Wrap-Up: What You Learned#
- The real scraping mechanics here, requests plus BeautifulSoup plus pagination, are genuinely identical whether the target is a real local page or a real live competitor site.
- Checking robots.txt first is a real professional habit that costs nothing and shows real respect for the site being scraped.
- A real pagination loop keeps following Next links until a real page genuinely returns no more products.
- Comparing real scraped current prices against a real earlier benchmark is exactly what real competitive price monitoring looks like in practice.
- A real percentage-change threshold is a genuinely simple, effective way to flag which real price moves actually deserve a human look.
- Next video: the real Domain 1 capstone, pulling every technique from this retail series into one real end-to-end report.
Practice on your own
- Change
page_sizeinrender_listing_pageto 7, restart the server and rerun the scraping loop. Check that you still collect 30 products and that the page count printed at the end changes to 5. - Add a
Quantityspan to each product card in the HTML, extend the scraping loop to read it, and confirm it matchesqty_laterfor every product. - Recompute the earlier and later prices as quantity-weighted averages (sum of
Revenuedivided by sum ofQuantity) and compare the newPctChangevalues with the simple averages, especially for the two Christmas decorations.
The next video in the series is the Domain 1 capstone, pulling every technique from this retail series into one end-to-end report.
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.



