Mathew K Analytics

Lesson 38 · Python for Data Science

Mastering DataFrame Merging in Python: Essential Techniques for Data Preparation

Welcome! Today, we will learn how to combine DataFrames using Python and pandas. Learning to merge DataFrames is important for working with real-world data.…

⬇ Download notebookOpen in Colab ↗

What you'll learn

Data

No separate download needed — the notebook creates or downloads everything it uses.

📓 Full notebook

Download .ipynb
 

Merging DataFrames in Python#

Welcome! Today, we will learn how to combine DataFrames using Python and pandas.

Learning to merge DataFrames is important for working with real-world data.

Let us get started!

# Let us import pandas to start working with DataFrames
import pandas as pd
# Let us create our first DataFrame with students and their classes
students = pd.DataFrame({
    'student_id': [1, 2, 3],
    'name': ['Sam', 'Alex', 'Jordan'],
    'class': ['Math', 'English', 'Art']
})

students
student_id name class
0 1 Sam Math
1 2 Alex English
2 3 Jordan Art
# Here is another DataFrame showing student test scores
scores = pd.DataFrame({
    'student_id': [1, 2, 4],
    'test_score': [88, 92, 77]
})

scores
student_id test_score
0 1 88
1 2 92
2 4 77

Why merge DataFrames?#

  • Often, information is spread across different tables.
  • To answer questions, you need to bring the data together.

Merging lets you see everything in one view.

# Let us merge the students and scores DataFrames on student_id
merged = pd.merge(students, scores, on='student_id')

merged
student_id name class test_score
0 1 Sam Math 88
1 2 Alex English 92
# Notice that student_id 4 did not appear in the merged result.
# That is because it only combines rows where the IDs are present in both tables.
# Use how='outer' to keep all rows from both DataFrames
merged_outer = pd.merge(students, scores, on='student_id', how='outer')

merged_outer
student_id name class test_score
0 1 Sam Math 88.0
1 2 Alex English 92.0
2 3 Jordan Art NaN
3 4 NaN NaN 77.0
# how='left' keeps all rows from the left table (students)
merged_left = pd.merge(students, scores, on='student_id', how='left')

merged_left
student_id name class test_score
0 1 Sam Math 88.0
1 2 Alex English 92.0
2 3 Jordan Art NaN
# how='right' keeps all rows from the right table (scores)
merged_right = pd.merge(students, scores, on='student_id', how='right')

merged_right
student_id name class test_score
0 1 Sam Math 88
1 2 Alex English 92
2 4 NaN NaN 77
# What if the columns have different names? Let us see an example.
info = pd.DataFrame({
    'id': [1, 2, 3],
    'hobby': ['Reading', 'Hiking', 'Drawing']
})

pd.merge(students, info, left_on='student_id', right_on='id')
student_id name class id hobby
0 1 Sam Math 1 Reading
1 2 Alex English 2 Hiking
2 3 Jordan Art 3 Drawing

How to handle duplicate column names?#

When merging, columns with the same name (not the one you join on) get suffixes like '_x' and '_y'.

This helps keep data clear.

# Here is how suffixes work if both DataFrames have a 'class' column
classes = pd.DataFrame({
    'student_id': [2, 3],
    'class': ['Science', 'Drama']
})

pd.merge(students, classes, on='student_id', how='inner')
student_id name class_x class_y
0 2 Alex English Science
1 3 Jordan Art Drama
# Use the 'suffixes' argument to customize the suffixes
pd.merge(students, classes, on='student_id', how='inner', suffixes=('_orig', '_new'))
student_id name class_orig class_new
0 2 Alex English Science
1 3 Jordan Art Drama
# time for you to try: type two column names to merge on, separated by a comma
columns = input('Type two columns to merge on, separated by a comma: ').split(',')
columns = [col.strip() for col in columns]
result = pd.merge(students, scores, left_on=columns[0], right_on=columns[0])
result
student_id name class test_score
0 1 Sam Math 88
1 2 Alex English 92
 

Real-World Example: Merging Survey Results#

Imagine you have survey responses in one file and user info in another.

Merging lets you answer questions about who said what.

# Mini-project: Combine purchase and customer info to see who bought what
purchases = pd.DataFrame({
    'purchase_id': [101, 102, 103],
    'customer_id': [1, 1, 2],
    'item': ['Book', 'Pen', 'Notebook']
})
customers = pd.DataFrame({
    'customer_id': [1, 2, 3],
    'customer_name': ['Lee', 'Dana', 'Morgan']
})

purchase_info = pd.merge(purchases, customers, on='customer_id')
purchase_info
purchase_id customer_id item customer_name
0 101 1 Book Lee
1 102 1 Pen Lee
2 103 2 Notebook Dana
# Part 2: Let us see what happens with an outer join
all_purchases = pd.merge(purchases, customers, on='customer_id', how='outer')
all_purchases
purchase_id customer_id item customer_name
0 101.0 1 Book Lee
1 102.0 1 Pen Lee
2 103.0 2 Notebook Dana
3 NaN 3 NaN Morgan
# What if you get an error? For example, columns are missing or mis-typed.
try:
    pd.merge(purchases, customers, on='wrong_column')
except Exception as e:
    print('Error:', e)
    
Error: 'wrong_column'
# Best practice: Always look at your data before merging
print(students.head())
print(scores.head())
   student_id    name    class
0           1     Sam     Math
1           2    Alex  English
2           3  Jordan      Art
   student_id  test_score
0           1          88
1           2          92
2           4          77
# Common mistake: Forgetting to reset the index after merging
merged_with_index = pd.merge(students, scores, on='student_id')
merged_reset = merged_with_index.reset_index(drop=True)
merged_reset
student_id name class test_score
0 1 Sam Math 88
1 2 Alex English 92
# Extra tip: You can merge on multiple columns too
# Let us add a new DataFrame for this example
grades = pd.DataFrame({
    'student_id': [1, 2, 2],
    'class': ['Math', 'English', 'Math'],
    'grade': ['A', 'B', 'A']
})

multi_merge = pd.merge(students, grades, on=['student_id', 'class'], how='inner')
multi_merge
student_id name class grade
0 1 Sam Math A
1 2 Alex English B

Challenge: Merge your own DataFrames!#

  • Make two tables: one with IDs and first names, one with IDs and favorite colors.
  • Merge them on the ID column and see the result.

Did you get it right? Awesome! Keep practicing.

Recap#

  • Merging joins tables together, like matching puzzle pieces.
  • Use pd.merge and try different options: on, left_on, right_on, how.
  • Try merges with more than one column or with custom suffixes.

Thanks for learning! Practice is the key.

Thanks for watching! If you learned something, please:#

  • Like this video
  • Comment your questions below
  • Subscribe for more Python lessons
  • Share it with a friend

Happy coding!

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.