Lesson 7 · Python for Data Analysts

Web APIs & Requests In Depth

Everything you need to talk to real web APIs from Python: GET, POST, JSON, auth, pagination, retries, and error handling. No prior networking experience…

⬇ Download notebook Open in Colab ↗

CoursePython for Data Analysts
Lesson7 of 12
Video27 min
FormatJupyter notebook · 20 code cells

JSON Requests

Data

No separate download needed — the notebook creates or downloads everything it uses.

📓 Full notebook

Download .ipynb

Web APIs & Requests In Depth#

Everything you need to talk to real web APIs from Python: GET, POST, JSON, auth, pagination, retries, and error handling.
No prior networking experience needed. Let's get straight into it.

Before You Start#

Open a new Jupyter Notebook in VS Code and select your Python interpreter as the kernel.
If requests isn't installed yet, open a terminal in VS Code and run: pip install requests

Part 1: HTTP Basics#

What Is HTTP?#

HTTP is the protocol web clients and servers use to exchange requests and responses.
Every request has a method: GET reads data, POST creates it, PUT/PATCH update it, DELETE removes it.
Every response has a status code: 2xx means success, 4xx means the client made a mistake, 5xx means the server failed.

Setting Up a Local Practice API#

import json
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import urlparse, parse_qs

USERS = [
    {'id': 1, 'name': 'Amir Hassan', 'role': 'engineer'},
    {'id': 2, 'name': 'Bianca Silva', 'role': 'designer'},
    {'id': 3, 'name': 'Carlos Mendes', 'role': 'engineer'},
    {'id': 4, 'name': 'Deepa Nair', 'role': 'manager'},
    {'id': 5, 'name': 'Elin Berg', 'role': 'designer'},
]
next_id = [6]
flaky_attempts = {'count': 0}

class APIHandler(BaseHTTPRequestHandler):
    def _send_json(self, status, payload):
        body = json.dumps(payload).encode('utf-8')
        self.send_response(status)
        self.send_header('Content-Type', 'application/json')
        self.send_header('Content-Length', str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, format, *args):
        pass

    def do_GET(self):
        parsed = urlparse(self.path)
        query = parse_qs(parsed.query)

        if parsed.path == '/users':
            page = int(query.get('page', ['1'])[0])
            limit = int(query.get('limit', ['2'])[0])
            start = (page - 1) * limit
            end = start + limit
            self._send_json(200, {'page': page, 'limit': limit, 'total': len(USERS), 'results': USERS[start:end]})
        elif parsed.path.startswith('/users/'):
            user_id = int(parsed.path.rsplit('/', 1)[-1])
            match = next((u for u in USERS if u['id'] == user_id), None)
            if match:
                self._send_json(200, match)
            else:
                self._send_json(404, {'error': 'user not found'})
        elif parsed.path == '/secure':
            if self.headers.get('X-API-Key') == 'secret123':
                self._send_json(200, {'message': 'welcome, you are authenticated'})
            else:
                self._send_json(401, {'error': 'missing or invalid API key'})
        elif parsed.path == '/flaky':
            flaky_attempts['count'] += 1
            if flaky_attempts['count'] < 3:
                self._send_json(500, {'error': 'temporary server error, try again'})
            else:
                self._send_json(200, {'message': 'success after retrying'})
        else:
            self._send_json(404, {'error': 'not found'})

    def do_POST(self):
        parsed = urlparse(self.path)
        if parsed.path == '/users':
            length = int(self.headers.get('Content-Length', 0))
            body = json.loads(self.rfile.read(length) or b'{}')
            new_user = {'id': next_id[0], 'name': body.get('name', 'Unknown'), 'role': body.get('role', 'Unknown')}
            next_id[0] += 1
            USERS.append(new_user)
            self._send_json(201, new_user)
        else:
            self._send_json(404, {'error': 'not found'})

server = ThreadingHTTPServer(('127.0.0.1', 0), APIHandler)
server_thread = threading.Thread(target=server.serve_forever, daemon=True)
server_thread.start()
BASE_URL = f'http://127.0.0.1:{server.server_port}'
print(f'Local practice API running at {BASE_URL}')

Local practice API running at http://127.0.0.1:49703

import requests

Part 2: Making GET Requests#

response = requests.get(f'{BASE_URL}/users')
print(response.status_code)
print(response.text)

200
{"page": 1, "limit": 2, "total": 5, "results": [{"id": 1, "name": "Amir Hassan", "role": "engineer"}, {"id": 2, "name": "Bianca Silva", "role": "designer"}]}

response = requests.get(f'{BASE_URL}/users', params={'page': 2, 'limit': 2})
print(response.url)
print(response.status_code)

http://127.0.0.1:49703/users?page=2&limit=2
200

response = requests.get(f'{BASE_URL}/users/2')
print(response.status_code)
print(response.headers['Content-Type'])

200
application/json

response = requests.get(f'{BASE_URL}/users/999')
print(response.status_code)
print(response.ok)

404
False

Part 3: Working With JSON#

response = requests.get(f'{BASE_URL}/users/1')
user = response.json()
print(type(user))
print(user['name'], '-', user['role'])

<class 'dict'>
Amir Hassan - engineer

response = requests.get(f'{BASE_URL}/users', params={'limit': 5})
data = response.json()
print(data['total'])
names = [u['name'] for u in data['results']]
print(names)

5
['Amir Hassan', 'Bianca Silva', 'Carlos Mendes', 'Deepa Nair', 'Elin Berg']

Part 4: POST Requests#

new_user = {'name': 'Farid Khan', 'role': 'analyst'}
response = requests.post(f'{BASE_URL}/users', json=new_user)
print(response.status_code)
print(response.json())

201
{'id': 6, 'name': 'Farid Khan', 'role': 'analyst'}

response = requests.get(f'{BASE_URL}/users', params={'limit': 10})
print(response.json()['total'])

Part 5: Error Handling#

response = requests.get(f'{BASE_URL}/users/999')
try:
    response.raise_for_status()
    print('Success')
except requests.exceptions.HTTPError as e:
    print(f'Request failed: {e}')

Request failed: 404 Client Error: Not Found for url: http://127.0.0.1:49703/users/999

try:
    response = requests.get(f'{BASE_URL}/does-not-exist', timeout=5)
    response.raise_for_status()
except requests.exceptions.Timeout:
    print('The request timed out')
except requests.exceptions.HTTPError as e:
    print(f'HTTP error: {e}')
except requests.exceptions.RequestException as e:
    print(f'Something else went wrong: {e}')

HTTP error: 404 Client Error: Not Found for url: http://127.0.0.1:49703/does-not-exist

response = requests.get(f'{BASE_URL}/secure')
print(response.status_code, response.json())

response = requests.get(f'{BASE_URL}/secure', headers={'X-API-Key': 'secret123'})
print(response.status_code, response.json())

401 {'error': 'missing or invalid API key'}
200 {'message': 'welcome, you are authenticated'}

all_names = []
page = 1
while True:
    response = requests.get(f'{BASE_URL}/users', params={'page': page, 'limit': 2})
    data = response.json()
    if not data['results']:
        break
    all_names.extend(u['name'] for u in data['results'])
    page += 1

print(f'Collected {len(all_names)} users across {page - 1} pages')
print(all_names)

Collected 6 users across 3 pages
['Amir Hassan', 'Bianca Silva', 'Carlos Mendes', 'Deepa Nair', 'Elin Berg', 'Farid Khan']

Part 7: Sessions and Retries#

session = requests.Session()
session.headers.update({'X-API-Key': 'secret123'})

response = session.get(f'{BASE_URL}/secure')
print(response.status_code)
response = session.get(f'{BASE_URL}/users/1')
print(response.status_code)

200
200

import time

max_attempts = 5
for attempt in range(1, max_attempts + 1):
    response = requests.get(f'{BASE_URL}/flaky')
    if response.ok:
        print(f'Succeeded on attempt {attempt}: {response.json()}')
        break
    print(f'Attempt {attempt} failed with status {response.status_code}, retrying...')
    time.sleep(0.1)
else:
    print('Giving up after max attempts')

Attempt 1 failed with status 500, retrying...
Attempt 2 failed with status 500, retrying...
Succeeded on attempt 3: {'message': 'success after retrying'}

Capstone Project: A Reusable API Client#

class APIClient:
    def __init__(self, base_url, api_key=None):
        self.base_url = base_url
        self.session = requests.Session()
        if api_key:
            self.session.headers.update({'X-API-Key': api_key})

    def get_users(self, page=1, limit=10):
        response = self.session.get(f'{self.base_url}/users', params={'page': page, 'limit': limit})
        response.raise_for_status()
        return response.json()

    def get_user(self, user_id):
        response = self.session.get(f'{self.base_url}/users/{user_id}')
        response.raise_for_status()
        return response.json()

    def create_user(self, name, role):
        response = self.session.post(f'{self.base_url}/users', json={'name': name, 'role': role})
        response.raise_for_status()
        return response.json()

    def get_secure_data(self):
        response = self.session.get(f'{self.base_url}/secure')
        response.raise_for_status()
        return response.json()

client = APIClient(BASE_URL, api_key='secret123')

print(client.get_users(limit=3))
print(client.get_user(2))
print(client.create_user('Grace Oduya', 'engineer'))
print(client.get_secure_data())

{'page': 1, 'limit': 3, 'total': 6, 'results': [{'id': 1, 'name': 'Amir Hassan', 'role': 'engineer'}, {'id': 2, 'name': 'Bianca Silva', 'role': 'designer'}, {'id': 3, 'name': 'Carlos Mendes', 'role': 'engineer'}]}
{'id': 2, 'name': 'Bianca Silva', 'role': 'designer'}
{'id': 7, 'name': 'Grace Oduya', 'role': 'engineer'}
{'message': 'welcome, you are authenticated'}

Wrap-Up: What You Learned#

HTTP fundamentals: methods, status codes, and how requests and responses actually work.
Making GET requests, with query parameters, headers, and full JSON responses.
Making POST requests to create resources, sending JSON bodies.
Layered error handling with raise_for_status, timeouts, and specific exception types.
Authentication headers and walking a fully paginated API.
Sessions for shared config, and a manual retry loop for flaky endpoints.
A capstone reusable APIClient class tying all of it together.
You went from a raw GET request to a production-shaped API client in one sitting. If you want the next build to land in your feed automatically, subscribing is the move see you in the next one.

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.

☕ Buy me a coffee Other ways to help