Mathew K Analytics

Lesson 26 · Real-World Data Analytics

Hashtag Analysis in Python: Trends, Co-occurrence & Topic Clusters | Data Analytics #26

What are people actually talking about? We analyse hashtags across 944 real social posts in Python: extract them with regex, find the top hashtags, build a…

What you'll learn

Datasets used in this lesson

Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.

📓 Full notebook

Download .ipynb

Data Analytics 100, Video 26: Real Hashtag and Topic Trend Analysis#

  • Video twenty-six of the hundred-video real-world data analytics series.
  • Returning to the real nine hundred forty-four social posts, this time hunting for real hashtag patterns and clusters.
  • Let's get into it.

This lesson is for learners who can already load a CSV with pandas and want to start pulling structure out of messy social media text. The business question is simple: which topics do people actually tag their posts with, which tags travel together, and does tagging behaviour differ between normal and flagged content? A marketing or trust-and-safety team asks exactly these questions when deciding which conversations to monitor.

The dataset is the same file of 944 real social posts used earlier in the series, social_posts_raw.csv, with three columns: an id, a label (1 for flagged posts, 0 for the rest) and the raw tweet text. There is no timestamp, so post order stands in as a rough proxy for time.

In this lesson you will:

  • extract hashtags and mentions with a regular expression (a compact text-matching pattern)
  • count frequencies and pairs of tags that appear together with Counter and itertools.combinations
  • compare tagging across labels and save tidy frequency tables to CSV

You should be comfortable with pandas apply and groupby; see earlier lessons in the series if not.

Part 1: Real Hashtags, Real Patterns#

Before analysing any text you need to know what you are working with. The code imports the tools used throughout: re for regular expressions, Counter for counting items, combinations for building tag pairs and matplotlib for charts. It then reads social_posts_raw.csv and checks .shape. The output is (944, 3): 944 posts and three columns (id, label, tweet). Checking the shape first is a cheap habit that catches a wrong file or a broken load straight away.

import pandas as pd
import re
from collections import Counter
from itertools import combinations
import matplotlib.pyplot as plt
posts = pd.read_csv('social_posts_raw.csv')
posts.shape
(944, 3)

Part 2: Real Hashtag Extraction#

Hashtags are buried inside free text, so the first job is to pull them out into a structured list. The helper function lowercases each tweet with str(text).lower() so that #Love and #love count as one tag, then re.findall(r'#(\w+)', ...) returns every run of word characters after a #. Applying it with .apply() gives a new hashtags column holding a Python list per post. The output shows the idea: one post yields [run], another [lyft, disapointed, getthanked], and the birthday post yields an empty list. Wrapping in str() protects you from missing values that would otherwise raise an error.

def extract_hashtags(text):
    return re.findall(r'#(\w+)', str(text).lower())
posts['hashtags'] = posts['tweet'].apply(extract_hashtags)
posts[['tweet', 'hashtags']].head(3)
tweet hashtags
0 @user when a father is dysfunctional and is s... [run]
1 @user @user thanks for #lyft credit i can't us... [lyft, disapointed, getthanked]
2 bihday your majesty []

Part 3: Real Hashtag Volume#

Before ranking tags you want a sense of scale. The list comprehension flattens the column of lists into one long list of every hashtag mention, then len() counts mentions and len(set(...)) counts distinct tags. The output is (2220, 1514): 2,220 mentions but 1,514 different hashtags. That ratio is a warning sign worth noticing early. With so many distinct tags per mention, most tags must be used only once or twice, which shapes every analysis that follows.

all_hashtags = [tag for tags in posts['hashtags'] for tag in tags]
total_mentions = len(all_hashtags)
distinct_hashtags = len(set(all_hashtags))
total_mentions, distinct_hashtags
(2220, 1514)

Part 4: Real Top Twenty Hashtags#

Frequency is the first lens on what people talk about. Counter(all_hashtags) builds a dictionary-like tally, and .most_common(20) returns the 20 highest counts as (tag, count) pairs. The output is led by love with 46 mentions, then positive (28), thankful (19) and healthy (17). Further down, blog, silver and gold each have 15 and forex has 13, a hint of a finance-themed group. Even the top tag appears in only a few percent of the 2,220 mentions, so no single topic dominates.

hashtag_counts = Counter(all_hashtags)
top20 = hashtag_counts.most_common(20)
top20
[('love', 46),
 ('positive', 28),
 ('thankful', 19),
 ('healthy', 17),
 ('model', 16),
 ('blog', 15),
 ('silver', 15),
 ('gold', 15),
 ('forex', 13),
 ('altwaystoheal', 12),
 ('orlando', 11),
 ('i_am', 11),
 ('affirmation', 11),
 ('fun', 11),
 ('smile', 11),
 ('fathersday', 10),
 ('weekend', 9),
 ('summer', 9),
 ('happiness', 9),
 ('healing', 9)]

Part 5: Visualizing Real Top Hashtags#

A ranked list is easier to scan as a chart, especially for stakeholders. The code wraps the top 15 tags in a small DataFrame and draws a horizontal bar chart with plt.barh, which keeps long tag names readable. plt.gca().invert_yaxis() flips the axis so the most-used tag sits at the top rather than the bottom. The chart is saved to top_hashtags.png and the figure is closed, so nothing displays here. When you open the image, look for how quickly the bars shrink after love and positive.

top15_df = pd.DataFrame(hashtag_counts.most_common(15), columns=['hashtag', 'count'])
plt.figure(figsize=(9, 6))
plt.barh(top15_df['hashtag'], top15_df['count'], color='mediumvioletred')
plt.xlabel('Real Mention Count')
plt.title('Real Top 15 Hashtags')
plt.gca().invert_yaxis()
plt.tight_layout()
plt.savefig('top_hashtags.png', dpi=120)
plt.close()

Part 6: Real Hashtag Usage by Content Label#

Comparing groups is where descriptive counts start to become insight. The code creates a True/False has_hashtag column by checking whether each list has length above zero. Grouping by label and taking .mean() of a boolean gives the share of True values, and multiplying by 100 turns it into a percentage. The output shows 73.1% of normal posts (label 0) and 78.9% of flagged posts (label 1) use at least one hashtag. Flagged posts tag slightly more often, though the gap is modest and the flagged group is much smaller.

posts['has_hashtag'] = posts['hashtags'].apply(len) > 0
hashtag_rate_by_label = posts.groupby('label')['has_hashtag'].mean().round(3) * 100
hashtag_rate_by_label
label
0    73.1
1    78.9
Name: has_hashtag, dtype: float64

Part 7: Real Hashtags per Post Distribution#

Knowing whether a post has a hashtag is useful, but knowing how many tells you more about posting style. posts['hashtags'].apply(len) stores a count per post, and .describe() summarises it. The output shows an average of 2.35 hashtags per post with a median of 2, while a quarter of posts have none (25th percentile is 0) and the busiest post carries 20. The mean sitting above the median, plus that maximum, signals a right-skewed distribution: a few heavy taggers pull the average up.

posts['hashtag_count'] = posts['hashtags'].apply(len)
posts['hashtag_count'].describe().round(2)
count    944.00
mean       2.35
std        2.46
min        0.00
25%        0.00
50%        2.00
75%        4.00
max       20.00
Name: hashtag_count, dtype: float64

Part 8: Real Hashtag Co-occurrence#

Single tags tell you what is popular. Pairs of tags tell you what goes together, which is how topic clusters emerge. For each post, the code de-duplicates and sorts the tags with sorted(set(tags)), then combinations(unique_tags, 2) produces every unordered pair, and each pair is counted in a Counter. Sorting matters: it makes sure ('gold', 'silver') and ('silver', 'gold') are counted as the same pair. The output shows 3,991 distinct tag pairs found across the posts.

cooccurrence = Counter()
for tags in posts['hashtags']:
    unique_tags = sorted(set(tags))
    for tag_a, tag_b in combinations(unique_tags, 2):
        cooccurrence[(tag_a, tag_b)] += 1
len(cooccurrence)
3991

Part 9: Real Top Co-occurring Pairs#

With the pairs counted, .most_common(10) surfaces the strongest associations. The output reveals two clear groups. One is wellness: positive and thankful appear together 16 times, and affirmation, i_am and positive pair with each other 11 times each. The other is finance: blog, silver, gold and forex pair with each other 13 to 15 times. When the same small set of tags pairs in every combination like this, it usually means a single account or campaign posting near-identical content.

top_pairs = cooccurrence.most_common(10)
top_pairs
[(('positive', 'thankful'), 16),
 (('blog', 'silver'), 15),
 (('blog', 'gold'), 14),
 (('gold', 'silver'), 14),
 (('blog', 'forex'), 13),
 (('forex', 'gold'), 13),
 (('forex', 'silver'), 13),
 (('affirmation', 'i_am'), 11),
 (('affirmation', 'positive'), 11),
 (('i_am', 'positive'), 11)]

Part 10: Real Topic Cluster Identification#

The pairs suggest two topic clusters, so the next step is to define them explicitly and count the posts in each. The code writes each cluster as a Python set of tags, then keeps rows where set(tags) & financial_tags (the set intersection) is non-empty, wrapped in bool() to give True or False. The output is (15, 31): 15 posts touch the financial cluster and 31 the wellness cluster. These are small numbers out of 944, a useful reminder that strong co-occurrence does not always mean large volume.

financial_tags = {'gold', 'silver', 'forex', 'blog'}
wellness_tags = {'positive', 'thankful', 'affirmation', 'i_am'}
financial_posts = posts[posts['hashtags'].apply(lambda tags: bool(set(tags) & financial_tags))]
wellness_posts = posts[posts['hashtags'].apply(lambda tags: bool(set(tags) & wellness_tags))]
len(financial_posts), len(wellness_posts)
(15, 31)

Part 11: Real Cluster Overlap Check#

Before treating the two clusters as separate topics you should confirm they really are separate. The code turns the id column of each subset into a set and intersects them with &, then counts the result. The output is 0, so no post belongs to both clusters. That makes the clusters clean and safe to report side by side. If you found overlap, you would need to decide whether to assign shared posts to one cluster or report them separately.

overlap_ids = set(financial_posts['id']) & set(wellness_posts['id'])
len(overlap_ids)
0

Part 12: Real Single-Use Hashtags#

Social tags usually follow a long tail: a few tags are used a lot and most are used once. The code collects every tag whose count is exactly 1 from hashtag_counts.items(), then divides by the number of distinct hashtags and rounds to one decimal. The output is 83.6, meaning 83.6% of distinct hashtags appear only once. For an analyst this means frequency-based dashboards will ignore most of the vocabulary, and grouping tags into broader topics is often more useful than tracking individual ones.

single_use = [tag for tag, count in hashtag_counts.items() if count == 1]
single_use_pct = round(len(single_use) / distinct_hashtags * 100, 1)
single_use_pct
83.6

Part 13: Real Hashtag Length Analysis#

Tag length is a quick proxy for style: short tags are generic, long ones are often campaign slogans or joined-up phrases. The code builds a pandas Series of character lengths for every mention and calls .describe(). The output shows a mean of 7.37 characters and a median of 7, with half of all mentions between 5 and 9 characters. The longest is 27 characters. Because this counts every mention, popular tags such as love are counted many times, so the figures describe typical usage rather than the vocabulary.

hashtag_lengths = pd.Series([len(tag) for tag in all_hashtags])
hashtag_lengths.describe().round(2)
count    2220.00
mean        7.37
std         3.69
min         1.00
25%         5.00
50%         7.00
75%         9.00
max        27.00
dtype: float64

Part 14: Real Longest Hashtags Used#

Looking at the extremes helps you sanity-check the extraction and spot oddities. The code takes the distinct tags with set(), sorts them by len with reverse=True and keeps the top five. The output includes golfstrengthandconditioning and internationaldayofyoga2016, which are genuine joined-up phrases. One entry ends in a strange character (thinkbigsundaywithmarsha followed by ð), a sign of an emoji stored with broken text encoding. It is worth knowing that \w in Python matches such characters, so encoding problems can leak into your tags.

longest_hashtags = sorted(set(all_hashtags), key=len, reverse=True)[:5]
longest_hashtags
['golfstrengthandconditioning',
 'internationaldayofyoga2016',
 'thinkbigsundaywithmarshað',
 'whenrealtorscompeteyouwin',
 'wheresallthenaturalphotos']

Part 15: Real Word Count vs Hashtag Count Correlation#

A natural question is whether longer posts simply carry more hashtags. The code counts words per tweet with str(t).split() and then uses .corr() to compute the Pearson correlation (a number from -1 to 1 measuring how closely two variables move together in a straight line). The output is -0.078, which is close to zero. In plain terms, word count and hashtag count are almost unrelated here. Note that hashtags are themselves counted as words, so this is a rough measure rather than a clean comparison.

posts['word_count'] = posts['tweet'].apply(lambda t: len(str(t).split()))
wordcount_hashtag_corr = posts['word_count'].corr(posts['hashtag_count'])
round(wordcount_hashtag_corr, 3)
np.float64(-0.078)

Part 16: Real Zero-Hashtag Posts#

Posts with no hashtags are invisible to any tag-based monitoring, so it is worth knowing how many there are. The code filters rows where hashtag_count == 0, divides by the total number of posts and rounds. The output shows 26.5% of posts carry no hashtag at all. This is consistent with the earlier label rates of roughly 73% to 79% using hashtags. For a real monitoring plan, you would need text-based methods to cover this quarter of the conversation.

zero_hashtag_posts = posts[posts['hashtag_count'] == 0]
zero_hashtag_pct = round(len(zero_hashtag_posts) / len(posts) * 100, 1)
zero_hashtag_pct
26.5

Part 17: Real Heavy Hashtag Users#

Posts stuffed with tags often behave differently, for example promotional or spammy content. The code keeps posts with hashtag_count >= 5. It then calls len(heavy_hashtag_posts), but because that line is not the last expression, Jupyter does not display its value. The output instead shows the first three heavy posts: one with 7 hashtags and two with 5, including a sports celebration and an economic data update. Wrap the length in print() if you want both results shown from one step.

heavy_hashtag_posts = posts[posts['hashtag_count'] >= 5]
len(heavy_hashtag_posts)
heavy_hashtag_posts[['tweet', 'hashtag_count']].head(3)
tweet hashtag_count
7 the next school year is the year for exams.ðŸ˜... 7
8 we won!!! love the land!!! #allin #cavs #champ... 5
10 ↝ #ireland consumer price index (mom) climb... 5

Part 18: Real Hashtag Diversity by Label#

Comparing how many different tags each group uses tells you how varied their topics are. The code builds two flat lists of tags using .loc with a boolean condition on label, then counts distinct tags in each with set(). The output is (124, 1406): flagged posts use 124 distinct hashtags and normal posts 1,406. Be careful interpreting this. The flagged group contains far fewer posts, so a smaller vocabulary is expected and does not by itself mean flagged content is less diverse.

flagged_tags = [tag for tags in posts.loc[posts['label']==1, 'hashtags'] for tag in tags]
normal_tags = [tag for tags in posts.loc[posts['label']==0, 'hashtags'] for tag in tags]
len(set(flagged_tags)), len(set(normal_tags))
(124, 1406)

Part 19: Real Top Hashtags in Flagged Posts#

Once you know flagged posts are tagged a little more often, the next question is what they are tagged with. The code runs Counter on the flagged tag list and takes the top 10. The output is led by trump with 6 mentions, then sjw with 4, and libtard, liberal, politics, hate and bigotry with 3 each. The theme is clearly political and identity-related, very different from the wellness tags leading the full dataset. The counts are small, so treat this as a direction to investigate rather than a firm conclusion.

flagged_top = Counter(flagged_tags).most_common(10)
flagged_top
[('trump', 6),
 ('sjw', 4),
 ('libtard', 3),
 ('liberal', 3),
 ('politics', 3),
 ('hate', 3),
 ('bigotry', 3),
 ('tcot', 2),
 ('helpcovedolphins', 2),
 ('race', 2)]

Part 20: Real Hashtags Unique to Each Group#

Tags used only by one group can act as warning signals or as clear markers of normal content. The code uses set difference, set(a) - set(b), to find tags that appear in one group but never in the other. The output is (108, 1390): 108 of the 124 flagged-post tags never appear in normal posts, and 1,390 normal-post tags never appear in flagged ones. So most flagged vocabulary is distinctive, but because many of those tags are used once, they would make a noisy filter on their own.

flagged_only = set(flagged_tags) - set(normal_tags)
normal_only = set(normal_tags) - set(flagged_tags)
len(flagged_only), len(normal_only)
(108, 1390)

Part 21: Real Trend Across Post Order#

Without a timestamp, you can still look for drift across the dataset by using post order as a rough timeline. The code sorts by id, resets the index, and uses pd.cut(..., bins=10, labels=False) to split the row positions into 10 equal-width batches numbered 0 to 9. It then takes the mean of has_hashtag in each batch as a percentage. The output ranges from 64.2% in the last batch to 79.8% in batch 4, with most batches between about 70% and 79%. Only the final batch stands out as noticeably lower.

posts_sorted = posts.sort_values('id').reset_index(drop=True)
posts_sorted['batch'] = pd.cut(posts_sorted.index, bins=10, labels=False)
hashtag_rate_by_batch = posts_sorted.groupby('batch')['has_hashtag'].mean().round(3) * 100
hashtag_rate_by_batch
batch
0    73.7
1    78.7
2    75.5
3    71.6
4    79.8
5    70.2
6    72.6
7    73.4
8    75.5
9    64.2
Name: has_hashtag, dtype: float64

Part 22: Visualizing the Real Batch Trend#

A line chart makes batch-to-batch movement easier to judge than a column of numbers. The code calls .plot(kind='line', marker='o') directly on the batch Series, so the batch numbers become the x-axis. The figure is saved to hashtag_trend.png and closed, so nothing is displayed here. When you open it, look for whether the line wanders around a stable level or shows a real trend. Based on the numbers, expect a fairly flat line with a drop at the final batch.

plt.figure(figsize=(9, 5))
hashtag_rate_by_batch.plot(kind='line', marker='o', color='darkcyan')
plt.xlabel('Real Sequential Batch')
plt.ylabel('Real Hashtag Usage Rate (%)')
plt.title('Real Hashtag Usage Rate Across the Dataset')
plt.tight_layout()
plt.savefig('hashtag_trend.png', dpi=120)
plt.close()

Part 23: Real Mention Extraction for Comparison#

Mentions (@user) are the other main tagging habit on social platforms, so comparing them with hashtags adds context. The code reuses the same regular expression idea with @(\w+) to extract mentions, counts them per post, and correlates mention count with hashtag count. The output is -0.144, a weak negative relationship. Posts with more mentions tend, very slightly, to have fewer hashtags, which fits the idea that replies to people and broadcast-style tagged posts are somewhat different kinds of content.

posts['mentions'] = posts['tweet'].apply(lambda t: re.findall(r'@(\w+)', str(t).lower()))
posts['mention_count'] = posts['mentions'].apply(len)
hashtag_mention_corr = posts['hashtag_count'].corr(posts['mention_count'])
round(hashtag_mention_corr, 3)
np.float64(-0.144)

Part 24: Real Hashtag Vocabulary Growth#

Vocabulary growth shows whether new topics keep appearing as you move through the data or whether the set of tags saturates. The loop walks through posts in id order, finds tags not yet seen using set difference, records how many are new, then adds them to seen with .update(). The output (1514, 1514) confirms that the new-tag counts add up to exactly the distinct vocabulary found earlier. The new_tag_counts list itself is the interesting part: plotting it cumulatively would show how quickly new tags keep arriving.

seen = set()
new_tag_counts = []
for tags in posts_sorted['hashtags']:
    new_tags = set(tags) - seen
    new_tag_counts.append(len(new_tags))
    seen.update(tags)
sum(new_tag_counts), len(seen)
(1514, 1514)

Part 25: Saving the Real Hashtag Frequency Table#

Saving results lets other people and later lessons reuse your work without rerunning everything. The code turns the full .most_common() list into a two-column DataFrame and writes it with to_csv(..., index=False), which stops pandas adding an extra unnamed index column. It then reads the file back and checks that the row count equals the number of distinct hashtags. The output True confirms nothing was lost in the round trip. Reloading and checking like this is a good habit for any exported file.

hashtag_freq_table = pd.DataFrame(hashtag_counts.most_common(), columns=['hashtag', 'count'])
hashtag_freq_table.to_csv('hashtag_trend_frequency.csv', index=False)
reloaded_freq = pd.read_csv('hashtag_trend_frequency.csv')
reloaded_freq.shape[0] == len(hashtag_counts)
True

Part 26: Saving the Real Co-occurrence Table#

The pair counts are the most valuable output of this lesson, so they deserve their own file. The list comprehension unpacks each ((tag_a, tag_b), count) item into a flat row of three values, and the result is saved as hashtag_cooccurrence.csv. The output shape (3991, 3) matches the 3,991 pairs counted earlier. A table in this form, often called an edge list, can be loaded directly into network tools if you later want to draw tag relationships as a graph.

cooccur_table = pd.DataFrame([(a, b, c) for (a, b), c in cooccurrence.most_common()], columns=['hashtag_a', 'hashtag_b', 'count'])
cooccur_table.to_csv('hashtag_cooccurrence.csv', index=False)
cooccur_table.shape
(3991, 3)

Part 27: Real Sanity Check, Counts are Consistent#

Sanity checks catch silent mistakes before you present results. This step adds up every value in hashtag_counts and checks it equals total_mentions, the 2,220 mentions counted at the start. The output True confirms the Counter accounted for every mention and none were dropped or double-counted. Simple equality checks like this cost one line and are worth adding whenever two parts of your analysis should agree.

sum(hashtag_counts.values()) == total_mentions
True

Part 28: Real Sanity Check, Rates in Range#

Percentages outside 0 to 100 would mean a calculation went wrong, so it is worth checking automatically. The code builds two boolean comparisons on hashtag_rate_by_label, combines them with & (which needs the brackets around each side), and wraps the result in all(), which returns True only if every value passes. The output True confirms both label rates are valid percentages. A common mistake is writing and instead of &, which raises an error on a pandas Series.

all((0 <= hashtag_rate_by_label) & (hashtag_rate_by_label <= 100))
True

Part 29: Real Top Cluster Summary#

A compact summary table is what you would actually hand to a stakeholder. The code builds a small DataFrame from a dictionary, one row per cluster, using the lengths of the two filtered subsets. The output shows the Financial cluster with 15 posts and the Wellness cluster with 31. Presenting the clusters this way makes it clear that wellness talk is about twice as common as the finance tags, while both are small relative to the 944 posts overall.

cluster_summary = pd.DataFrame({'cluster': ['Financial', 'Wellness'], 'post_count': [len(financial_posts), len(wellness_posts)]})
cluster_summary
cluster post_count
0 Financial 15
1 Wellness 31

Part 30: Real Recap Print#

Ending with a one-sentence recap is a good habit: it forces you to state the headline finding plainly. The code uses an f-string to drop live values into the sentence, so the text updates automatically if the data changes. The output reads: across 944 posts there are 1,514 distinct hashtags, and the top pair ('positive', 'thankful') co-occurs 16 times. Using variables rather than typed numbers avoids the common mistake of a summary that no longer matches the analysis.

print(f'Across {len(posts)} real posts, we found {distinct_hashtags} distinct real hashtags, with the top pair {top_pairs[0][0]} genuinely co-occurring {top_pairs[0][1]} times.')
Across 944 real posts, we found 1514 distinct real hashtags, with the top pair ('positive', 'thankful') genuinely co-occurring 16 times.

Wrap-Up: What You Learned#

  • Regular expressions are the real simplest reliable way to pull hashtags and mentions out of raw real social text.
  • Co-occurrence counting, not just frequency counting, is what actually reveals real topic clusters hiding in hashtag data.
  • Real hashtag vocabularies follow a real long-tail pattern, a handful of tags dominate while most are used only once.
  • Comparing real hashtag usage across a content label can surface real differences in how genuine and flagged content gets tagged.
  • Without a real timestamp field, sequential post order is an honest, if imperfect, real proxy for tracking trend movement.
  • Next video: real web traffic analytics and attribution, tracing how real visitors reach a site in the first place.

Practice on your own

  • Take the new_tag_counts list from the vocabulary growth step, compute its running total with pd.Series(new_tag_counts).cumsum() and plot it. Does the curve flatten or keep climbing?
  • Repeat the top-10 co-occurring pairs analysis for flagged posts only (label == 1) and compare the pairs with those from the full dataset.
  • Build a top-15 bar chart for mentions instead of hashtags using the mentions column, and check what share of posts mention at least one user, split by label.

The next video in the series moves on to web traffic analytics and attribution, tracing how visitors reach a site in the first place.

🧠 Test yourself

Pick an answer to check it. Your best score is saved in this browser.

  1. What does this code return?

    import re
    print(re.findall(r'#(\w+)', 'Love #Gold and #Silver!'.lower()))
    
  2. Why does the lesson count co-occurrence instead of only counting hashtag frequency?

  3. Why does the lesson use sorted(set(tags)) before combinations(..., 2)?

  4. What long-tail pattern does the lesson find in hashtag usage?

  5. The dataset has no timestamp. How does the lesson study a trend over time?

💻 Practice exercises

Try each one in the notebook or in your own editor before opening the solution.

  1. Create 10 short tweets with hashtags. Write extract_hashtags(text) using re.findall, apply it to a DataFrame, count hashtag frequency with Counter, print the top 3 and the percentage of posts that have at least one hashtag.

    💡 Hint

    re.findall(r'#(\w+)', text.lower()) returns the tag names.

    ✅ Show solution
    import re
    import pandas as pd
    from collections import Counter
    
    posts = pd.DataFrame({'tweet': [
        'Gold prices up #gold #forex', 'Feeling #thankful today', 'No tags in this one',
        'Silver and #gold rally', 'Be #positive #thankful', 'Just chatting',
        'Trading #forex all day', '#positive vibes only', 'Check my #blog about #gold', 'Lunch time',
    ]})
    
    def extract_hashtags(text):
        return re.findall(r'#(\w+)', str(text).lower())
    
    posts['hashtags'] = posts['tweet'].apply(extract_hashtags)
    all_tags = [t for tags in posts['hashtags'] for t in tags]
    counts = Counter(all_tags)
    print(counts.most_common(3))
    print('Posts with a hashtag:', round((posts['hashtags'].apply(len) > 0).mean() * 100, 1), '%')
    
  2. Using 12 tweets, compute hashtags per post, describe the distribution, count co-occurring hashtag pairs with combinations, print the top 3 pairs, and report the percentage of distinct hashtags that are used only once.

    💡 Hint

    Use sorted(set(tags)) before combinations so pair order is stable.

    ✅ Show solution
    import re
    import pandas as pd
    from collections import Counter
    from itertools import combinations
    
    posts = pd.DataFrame({'tweet': [
        'Gold up #gold #forex #blog', '#thankful #positive', 'Feeling #positive #thankful #affirmation',
        'Trading #forex #gold', 'Plain text', '#gold', '#positive #i_am', '#forex #gold #silver',
        '#blog post', 'Another #thankful #positive day', 'Nothing here', '#silver #gold',
    ]})
    posts['hashtags'] = posts['tweet'].apply(lambda t: re.findall(r'#(\w+)', t.lower()))
    posts['hashtag_count'] = posts['hashtags'].apply(len)
    print(posts['hashtag_count'].describe().round(2))
    
    cooccurrence = Counter()
    for tags in posts['hashtags']:
        for a, b in combinations(sorted(set(tags)), 2):
            cooccurrence[(a, b)] += 1
    print(cooccurrence.most_common(3))
    
    counts = Counter(t for tags in posts['hashtags'] for t in tags)
    single_use_pct = round(sum(1 for c in counts.values() if c == 1) / len(counts) * 100, 1)
    print('Single-use hashtags:', single_use_pct, '%')
    
  3. Create 16 labelled tweets (label 0 normal, 1 flagged) with hashtags. Find posts in a 'financial' cluster ({'gold','forex'}) and a 'wellness' cluster ({'positive','thankful'}), check the overlap, find hashtags unique to flagged vs normal posts, and compute the hashtag-use rate across 4 sequential batches of posts.

    💡 Hint

    Use set intersection for cluster membership and pd.cut(range index, 4, labels=False) for batches.

    ✅ Show solution
    import re
    import pandas as pd
    
    posts = pd.DataFrame({
        'id': range(1, 17),
        'label': [0,0,1,0,0,1,0,0,1,0,0,0,1,0,0,1],
        'tweet': ['#gold news', '#positive day', '#forex scam #gold', 'plain', '#thankful #positive',
                  '#forex #blog', 'hello', '#gold #positive', '#forex #forex', 'plain again',
                  '#thankful', 'nothing', '#blog #gold', '#positive', 'quiet', '#forex #silver'],
    })
    posts['hashtags'] = posts['tweet'].apply(lambda t: re.findall(r'#(\w+)', t.lower()))
    
    financial = {'gold', 'forex'}
    wellness = {'positive', 'thankful'}
    fin_posts = posts[posts['hashtags'].apply(lambda tags: bool(set(tags) & financial))]
    well_posts = posts[posts['hashtags'].apply(lambda tags: bool(set(tags) & wellness))]
    overlap = set(fin_posts['id']) & set(well_posts['id'])
    print('Financial:', len(fin_posts), 'Wellness:', len(well_posts), 'Overlap:', len(overlap))
    
    flagged_tags = {t for tags in posts.loc[posts['label'] == 1, 'hashtags'] for t in tags}
    normal_tags = {t for tags in posts.loc[posts['label'] == 0, 'hashtags'] for t in tags}
    print('Only in flagged:', sorted(flagged_tags - normal_tags))
    print('Only in normal:', sorted(normal_tags - flagged_tags))
    
    posts_sorted = posts.sort_values('id').reset_index(drop=True)
    posts_sorted['has_hashtag'] = posts_sorted['hashtags'].apply(len) > 0
    posts_sorted['batch'] = pd.cut(posts_sorted.index, bins=4, labels=False)
    print((posts_sorted.groupby('batch')['has_hashtag'].mean() * 100).round(1))
    

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.