Mathew K Analytics

Lesson 11 · Python for Data Analysts

SQL for Python Users with sqlite3

Everything you need to create, query, and join real databases straight from Python: tables, joins, parameterized queries, and pandas integration. No prior…

What you'll learn

Datasets used in this lesson

Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.

📓 Full notebook

Download .ipynb

SQL for Python Users with sqlite3#

  • Everything you need to create, query, and join real databases straight from Python: tables, joins, parameterized queries, and pandas integration.
  • No prior SQL experience needed. Let's get straight into it.

Before You Start#

  • Open a new Jupyter Notebook in VS Code and select your Python interpreter as the kernel.
  • sqlite3 ships with Python itself, no install needed; if pandas isn't installed for the pandas-integration section, open a terminal and run: pip install pandas

Part 1: Why Databases?#

Beyond CSV Files#

  • CSV files work fine for one flat table, but real data usually has relationships: customers who place orders, orders that contain products.
  • Databases store related tables separately, then join them together on demand, without duplicating data.
  • SQL, Structured Query Language, is how you ask a database questions: which rows, which columns, in what order.

Part 2: Connecting and Creating Tables#

import sqlite3

conn = sqlite3.connect('shop.db')
cursor = conn.cursor()
print(type(conn))
print(type(cursor))
<class 'sqlite3.Connection'>
<class 'sqlite3.Cursor'>
cursor.execute('''
    CREATE TABLE IF NOT EXISTS customers (
        id INTEGER PRIMARY KEY,
        name TEXT NOT NULL,
        city TEXT,
        signup_date TEXT
    )
''')
conn.commit()
print('customers table ready')
customers table ready
cursor.execute('''
    CREATE TABLE IF NOT EXISTS orders (
        id INTEGER PRIMARY KEY,
        customer_id INTEGER,
        product TEXT NOT NULL,
        amount REAL,
        order_date TEXT,
        FOREIGN KEY (customer_id) REFERENCES customers (id)
    )
''')
conn.commit()
print('orders table ready')
orders table ready

Part 3: Inserting Data#

cursor.execute(
    'INSERT INTO customers (name, city, signup_date) VALUES (?, ?, ?)',
    ('Amir Hassan', 'Austin', '2023-01-15')
)
conn.commit()
print('One customer inserted')
One customer inserted
import random

random.seed(5)
cities = ['Austin', 'Denver', 'Seattle', 'Miami', 'Boston']
names = ['Bianca Silva', 'Carlos Mendes', 'Deepa Nair', 'Elin Berg', 'Farid Khan', 'Grace Oduya', 'Hana Kim', 'Ivan Petrov', 'Julia Novak', 'Kenji Sato', 'Lena Fischer', 'Marco Rossi', 'Nadia Ali', 'Omar Idris', 'Priya Rao']

customer_rows = []
for i, name in enumerate(names):
    city = random.choice(cities)
    month = random.randint(1, 12)
    day = random.randint(1, 28)
    customer_rows.append((name, city, f'2023-{month:02d}-{day:02d}'))

cursor.executemany(
    'INSERT INTO customers (name, city, signup_date) VALUES (?, ?, ?)',
    customer_rows
)
conn.commit()
print(f'{len(customer_rows)} more customers inserted')
15 more customers inserted
random.seed(9)
products = ['Wireless Mouse', 'Mechanical Keyboard', 'Desk Lamp', 'Webcam', 'Monitor Stand']
order_rows = []
for _ in range(60):
    customer_id = random.randint(1, 16)
    product = random.choice(products)
    amount = round(random.uniform(15, 250), 2)
    month = random.randint(1, 12)
    day = random.randint(1, 28)
    order_rows.append((customer_id, product, amount, f'2023-{month:02d}-{day:02d}'))

cursor.executemany(
    'INSERT INTO orders (customer_id, product, amount, order_date) VALUES (?, ?, ?, ?)',
    order_rows
)
conn.commit()
print(f'{len(order_rows)} orders inserted')
60 orders inserted

Part 4: Querying Data#

cursor.execute('SELECT * FROM customers LIMIT 5')
rows = cursor.fetchall()
for row in rows:
    print(row)
(1, 'Amir Hassan', 'Austin', '2023-01-15')
(2, 'Bianca Silva', 'Boston', '2023-05-24')
(3, 'Carlos Mendes', 'Seattle', '2023-12-27')
(4, 'Deepa Nair', 'Boston', '2023-01-27')
(5, 'Elin Berg', 'Miami', '2023-04-21')
cursor.execute("SELECT name, city FROM customers WHERE city = 'Austin'")
austin_customers = cursor.fetchall()
print(austin_customers)
[('Amir Hassan', 'Austin'), ('Farid Khan', 'Austin'), ('Ivan Petrov', 'Austin'), ('Julia Novak', 'Austin'), ('Marco Rossi', 'Austin'), ('Priya Rao', 'Austin')]
cursor.execute('SELECT name, amount FROM orders JOIN customers ON orders.customer_id = customers.id ORDER BY amount DESC LIMIT 1')
top_order = cursor.fetchone()
print(top_order)
('Elin Berg', 246.78)

Part 5: Joins and Grouping#

cursor.execute('''
    SELECT customers.name, orders.product, orders.amount
    FROM orders
    JOIN customers ON orders.customer_id = customers.id
    LIMIT 5
''')
joined_rows = cursor.fetchall()
for row in joined_rows:
    print(row)
('Omar Idris', 'Monitor Stand', 102.73)
('Amir Hassan', 'Desk Lamp', 133.15)
('Kenji Sato', 'Monitor Stand', 235.05)
('Marco Rossi', 'Mechanical Keyboard', 180.32)
('Nadia Ali', 'Mechanical Keyboard', 54.58)
cursor.execute('''
    SELECT customers.name, COUNT(orders.id) AS order_count, SUM(orders.amount) AS total_spent
    FROM customers
    JOIN orders ON customers.id = orders.customer_id
    GROUP BY customers.name
    ORDER BY total_spent DESC
    LIMIT 5
''')
top_spenders = cursor.fetchall()
for row in top_spenders:
    print(row)
('Marco Rossi', 7, 1047.08)
('Elin Berg', 6, 883.53)
('Carlos Mendes', 8, 752.74)
('Amir Hassan', 6, 649.53)
('Bianca Silva', 6, 606.03)

Part 6: pandas Integration#

import pandas as pd

customers_df = pd.read_sql('SELECT * FROM customers', conn)
print(customers_df.head())
print(customers_df.shape)
   id           name     city signup_date
0   1    Amir Hassan   Austin  2023-01-15
1   2   Bianca Silva   Boston  2023-05-24
2   3  Carlos Mendes  Seattle  2023-12-27
3   4     Deepa Nair   Boston  2023-01-27
4   5      Elin Berg    Miami  2023-04-21
(16, 4)
spending_query = '''
    SELECT customers.city, SUM(orders.amount) AS total_spent
    FROM customers
    JOIN orders ON customers.id = orders.customer_id
    GROUP BY customers.city
    ORDER BY total_spent DESC
'''
city_spending = pd.read_sql(spending_query, conn)
print(city_spending)
      city  total_spent
0   Austin      2203.85
1    Miami      1326.86
2   Boston      1244.10
3  Seattle      1179.23
4   Denver       855.16
new_customers = pd.DataFrame({
    'name': ['Sofia Reyes', 'Toms Silva'],
    'city': ['Chicago', 'Portland'],
    'signup_date': ['2024-02-01', '2024-02-03'],
})
new_customers.to_sql('customers', conn, if_exists='append', index=False)
print('2 new customers written via pandas')
2 new customers written via pandas

Part 7: Parameterized Queries#

unsafe_city = "Austin' OR '1'='1"
unsafe_query = f"SELECT * FROM customers WHERE city = '{unsafe_city}'"
print(unsafe_query)
SELECT * FROM customers WHERE city = 'Austin' OR '1'='1'
cursor.execute('SELECT * FROM customers WHERE city = ?', (unsafe_city,))
safe_results = cursor.fetchall()
print(len(safe_results))
0

Part 8: Context Managers#

with sqlite3.connect('shop.db') as auto_conn:
    auto_cursor = auto_conn.cursor()
    auto_cursor.execute("INSERT INTO customers (name, city, signup_date) VALUES ('Uma Patel', 'Chicago', '2024-03-01')")
print('Insert committed automatically on exit')
Insert committed automatically on exit
cursor.execute("SELECT COUNT(*) FROM customers WHERE name = 'Uma Patel'")
print(cursor.fetchone())
(1,)

Capstone Project: A Sales Reporting Function#

def build_sales_report(connection, min_orders=1):
    top_customers = pd.read_sql('''
        SELECT customers.name, customers.city, COUNT(orders.id) AS order_count, SUM(orders.amount) AS total_spent
        FROM customers
        JOIN orders ON customers.id = orders.customer_id
        GROUP BY customers.id
        HAVING COUNT(orders.id) >= ?
        ORDER BY total_spent DESC
    ''', connection, params=(min_orders,))

    product_popularity = pd.read_sql('''
        SELECT product, COUNT(*) AS times_ordered, AVG(amount) AS avg_amount
        FROM orders
        GROUP BY product
        ORDER BY times_ordered DESC
    ''', connection)

    city_summary = pd.read_sql('''
        SELECT customers.city, COUNT(DISTINCT customers.id) AS customer_count, SUM(orders.amount) AS total_revenue
        FROM customers
        JOIN orders ON customers.id = orders.customer_id
        GROUP BY customers.city
        ORDER BY total_revenue DESC
    ''', connection)

    return {
        'top_customers': top_customers,
        'product_popularity': product_popularity,
        'city_summary': city_summary,
    }
reports = build_sales_report(conn, min_orders=2)
print(reports['top_customers'].head())
            name     city  order_count  total_spent
0    Marco Rossi   Austin            7      1047.08
1      Elin Berg    Miami            6       883.53
2  Carlos Mendes  Seattle            8       752.74
3    Amir Hassan   Austin            6       649.53
4   Bianca Silva   Boston            6       606.03
print(reports['product_popularity'])
print()
print(reports['city_summary'])
               product  times_ordered  avg_amount
0       Wireless Mouse             14   97.627857
1        Monitor Stand             13  114.471538
2            Desk Lamp             13  121.667692
3               Webcam             10  149.240000
4  Mechanical Keyboard             10   88.020000

      city  customer_count  total_revenue
0   Austin               5        2203.85
1    Miami               3        1326.86
2   Boston               3        1244.10
3  Seattle               2        1179.23
4   Denver               2         855.16
conn.close()
print('Connection closed')
Connection closed

Wrap-Up: What You Learned#

  • Why relational databases matter, and connecting to and creating tables with sqlite3.
  • Inserting data with execute and executemany, and committing changes.
  • Querying with SELECT, WHERE, ORDER BY, and fetchone/fetchall/fetchmany.
  • Joining tables together, and aggregating with GROUP BY, COUNT, SUM, and HAVING.
  • Reading and writing straight to pandas with read_sql and to_sql.
  • Parameterized queries, and why they're essential for safety, not just convenience.
  • Using with blocks for automatic commits.
  • A capstone reporting function combining joins, grouping, and pandas into reusable business reports.
  • You went from an empty database file to a full, joined, multi-report analysis in one sitting. If you want the next build to land in your feed automatically, subscribing is the move see you in the next one.

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.