Mathew K Analytics

Lesson 22 · Real-World Data Analytics

Sentiment Analysis in Python with VADER: Social Media Posts | Data Analytics #22

Score the sentiment of real social media posts in Python with VADER. We find the most positive and negative posts, classify sentiment, test whether it…

What you'll learn

Datasets used in this lesson

Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.

📓 Full notebook

Download .ipynb

Data Analytics 100, Video 22: Sentiment Analysis on Real Text Data#

  • Video twenty-two of the hundred-video real-world data analytics series.
  • Scoring real genuine sentiment on the exact same real social posts scraped last video, using a real established lexicon.
  • Let's get into it.

This lesson is for learners who have structured some text data and now want to measure what it actually says. It works on the same 944 social posts scraped in the previous video, and asks a practical marketing question: how positive or negative is this conversation, and does negative tone line up with posts that moderators flagged?

The data comes from social_posts_raw.csv and has an id, a label (1 for a flagged post, 0 for a normal one) and the post text in tweet. Sentiment is scored with VADER, an established lexicon-based tool. A lexicon is a dictionary of words with hand-assigned positive or negative scores, so no model training is needed.

In this lesson you will:

  • score every post with VADER and turn scores into Positive, Neutral and Negative classes
  • compare sentiment for flagged and normal posts and test the gap with a t-test
  • relate sentiment to hashtags, length and word count, and judge a simple sentiment-based flagging rule with precision and recall

Prerequisites: you should be comfortable with pandas apply, groupby and value_counts, and with the scraping lesson that came just before this one.

Part 1: Real Sentiment, Not a Guess#

Sentiment is easy to guess by eye but hard to measure consistently across hundreds of posts, so a repeatable scoring tool is needed. This step imports pandas, numpy and matplotlib, plus SentimentIntensityAnalyzer from the vaderSentiment package, and loads the posts from social_posts_raw.csv. There is no output yet. If the import fails, the package is not installed; pip install vaderSentiment fixes that. Note the package name differs from the class name, which trips many people up.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
posts = pd.read_csv('social_posts_raw.csv')

Part 2: Real Analyzer Setup#

Before scoring every post, test the tool on one so you understand what it returns. The code creates the analyser and calls polarity_scores on the fourth post (iloc[3], since counting starts at 0). The output has four numbers: neg, neu and pos are the shares of the text that read as negative, neutral and positive, here 0.0, 0.719 and 0.281. compound is a single overall score from -1 (very negative) to +1 (very positive), here 0.7249, a clearly positive post.

analyzer = SentimentIntensityAnalyzer()
sample_scores = analyzer.polarity_scores(posts['tweet'].iloc[3])
sample_scores
{'neg': 0.0, 'neu': 0.719, 'pos': 0.281, 'compound': 0.7249}

Part 3: Real Compound Score for Every Post#

The compound score is the one number you will use throughout, so it needs to exist for every post. The code uses .apply with a small lambda function that scores each tweet and keeps only ['compound']. The describe() output shows an average of 0.214 and a median of 0.224, so the conversation leans positive overall. Scores span almost the full range, from -0.931 to 0.959, and the 25th percentile is exactly 0, which hints that many posts contain no sentiment words at all.

posts['compound'] = posts['tweet'].apply(lambda t: analyzer.polarity_scores(t)['compound'])
posts['compound'].describe().round(3)
count    944.000
mean       0.214
std        0.489
min       -0.931
25%        0.000
50%        0.224
75%        0.637
max        0.959
Name: compound, dtype: float64

Part 4: Real Most Positive Posts#

Reading the top-scoring posts tells you whether the scores make sense to a human. nlargest(5, 'compound') returns the five rows with the highest compound score, and the column list keeps just the text and score. The output shows scores around 0.95 to 0.96, for posts about a happy baby, love and Father's Day, which fit the label well. You can also see odd characters such as ð in the text: these are emojis that were saved with the wrong text encoding, a common issue in scraped data.

most_positive = posts.nlargest(5, 'compound')[['tweet', 'compound']]
most_positive
tweet compound
235 the happiest baby ive ever known💓 #cute #sm... 0.9590
814 good evening my darling instagram babies. #b... 0.9590
191 ilovethesecret #lawofattraction #quiz #love ... 0.9565
376 do what you love to do simply for the love of ... 0.9538
472 happy fathers day. you are love of my life &am... 0.9538

Part 5: Real Most Negative Posts#

The most negative posts are just as important to check, because this is where a lexicon can misfire. nsmallest(5, 'compound') returns the five lowest scores, between -0.93 and -0.89. The top one refers to shocking events in Orlando and another to a medical diagnosis, so the tool is picking up genuine distress. One entry is a video titled mad, which shows a weakness of word-based scoring: a word can carry negative weight even when the post is not angry.

most_negative = posts.nsmallest(5, 'compound')[['tweet', 'compound']]
most_negative
tweet compound
655 shocking events in orlando - now will the usa ... -0.9314
568 i got the call yesterday. mom was diagnosed w... -0.9246
335 watching the @user leadership embrace & ki... -0.9136
83 carrying a gun wouldn't of helped if you can't... -0.9041
253 watch fancy tails's vine "mad #mad #teeth #b... -0.8910

Part 6: Real Sentiment Classification#

A score is useful, but stakeholders usually want categories. pd.cut sorts each compound score into bins defined by the edges [-1.01, -0.05, 0.05, 1.01], labelled Negative, Neutral and Positive. The ±0.05 cut-offs are the thresholds commonly recommended for VADER, and the outer edges sit slightly beyond -1 and 1 so no score falls outside. The output shows 506 positive, 233 neutral and 205 negative posts, so just over half the posts read as positive.

bins = [-1.01, -0.05, 0.05, 1.01]
labels = ['Negative', 'Neutral', 'Positive']
posts['sentiment_class'] = pd.cut(posts['compound'], bins=bins, labels=labels)
posts['sentiment_class'].value_counts()
sentiment_class
Positive    506
Neutral     233
Negative    205
Name: count, dtype: int64

Part 7: Visualizing Real Sentiment Distribution#

A bar chart makes the class balance easy to communicate. The code counts the classes, uses reindex(labels) to put the bars in the logical order Negative, Neutral, Positive rather than by size, and plots them in red, grey and green. plt.savefig writes the chart to sentiment_distribution.png and plt.close() stops it showing in the notebook, so there is no output here. When you open the image, look for the Positive bar standing at roughly twice the height of each of the other two.

plt.figure(figsize=(8, 5))
posts['sentiment_class'].value_counts().reindex(labels).plot(kind='bar', color=['crimson', 'gray', 'seagreen'])
plt.xlabel('Real Sentiment Class')
plt.ylabel('Real Number of Posts')
plt.title('Real Sentiment Distribution Across Social Posts')
plt.xticks(rotation=0)
plt.tight_layout()
plt.savefig('sentiment_distribution.png', dpi=120)
plt.close()

Part 8: Real Sentiment vs Real Content Label#

This step asks whether sentiment relates to something independent: the moderation label. groupby('label')['compound'].mean() gives the average compound score for each label. The output shows normal posts average 0.24, while flagged posts average -0.11. So flagged posts lean negative on average and normal posts lean positive. That is a meaningful gap, but an average hides a lot of spread, so the next steps look at it in more detail.

sentiment_by_label = posts.groupby('label')['compound'].mean()
sentiment_by_label.round(3)
label
0    0.24
1   -0.11
Name: compound, dtype: float64

Part 9: Real Sentiment Class Within Flagged Posts#

An average can be pulled around by a few extreme posts, so it helps to count classes within the flagged group. The code filters to label == 1 with .loc and counts sentiment classes. The output shows 30 negative, 24 positive and 17 neutral flagged posts, 71 in total. Negative is the largest group, but more than half of flagged posts are not negative at all. Flagged content can be sarcastic or upbeat in tone, which a sentiment score alone will not catch.

flagged_sentiment = posts.loc[posts['label'] == 1, 'sentiment_class'].value_counts()
flagged_sentiment
sentiment_class
Negative    30
Positive    24
Neutral     17
Name: count, dtype: int64

Part 10: Real Statistical Difference Check#

A difference in averages could be luck, so a statistical test checks whether it is likely to be real. stats.ttest_ind compares the mean compound score of flagged and normal posts; equal_var=False runs Welch's version, which does not assume both groups have the same spread, a sensible choice here as the groups differ greatly in size. The t-statistic is -6.129 and the p-value rounds to 0.0. A p-value is the chance of seeing a gap this large if there were truly no difference, so a value this small means the gap is very unlikely to be chance.

from scipy import stats
flagged_scores = posts.loc[posts['label'] == 1, 'compound']
normal_scores = posts.loc[posts['label'] == 0, 'compound']
t_stat, p_value = stats.ttest_ind(flagged_scores, normal_scores, equal_var=False)
round(t_stat, 3), round(p_value, 5)
(np.float64(-6.129), np.float64(0.0))

Part 11: Real Hashtag-Level Sentiment#

Hashtags group posts by topic, so averaging sentiment per hashtag shows which topics feel positive. The code extracts hashtags with .str.findall(r'#\w+'), then explode('hashtags') gives each hashtag its own row, so a post's score counts once for every hashtag it carries. It averages compound per hashtag and keeps only hashtags used at least 5 times, since an average over one or two posts is unreliable. The top of the list is #thankful at 0.88, followed by #beautiful and #quote.

posts['hashtags'] = posts['tweet'].str.findall(r'#\w+')
hashtag_sentiment = posts.explode('hashtags').dropna(subset=['hashtags']).groupby('hashtags')['compound'].mean()
hashtag_counts = posts.explode('hashtags')['hashtags'].value_counts()
common_hashtags = hashtag_counts[hashtag_counts >= 5].index
hashtag_sentiment.loc[common_hashtags].sort_values(ascending=False).head(10)
hashtags
#thankful           0.882205
#beautiful          0.870300
#quote              0.837017
#peace              0.826433
#lawofattraction    0.816900
#positive           0.814004
#blessed            0.807143
#home               0.775880
#cute               0.753071
#altwaystoheal      0.751717
Name: compound, dtype: float64

Part 12: Real Most Negative Common Hashtags#

The other end of the same table shows which common hashtags carry negative tone. Sorting the filtered averages in ascending order puts the lowest first. The output shows #orlando at -0.344 and #christinagrimmie at -0.337, both linked to tragic news events, then #trump at -0.139. #gold, #blog and #silver share exactly the same score of 0.071, which suggests they appear together in the same group of near-identical posts, so treat them as one source rather than three topics.

hashtag_sentiment.loc[common_hashtags].sort_values().head(10)
hashtags
#orlando            -0.344000
#christinagrimmie   -0.336800
#trump              -0.139433
#forex               0.033346
#gold                0.071360
#blog                0.071360
#silver              0.071360
#rip                 0.142500
#dog                 0.186740
#girl                0.275067
Name: compound, dtype: float64

Part 13: Real Sentiment vs Real Post Length#

Do longer posts sound more positive or more negative? .str.len() counts characters per post, and .corr() computes the Pearson correlation, a number from -1 to +1 measuring how closely two columns move together in a straight line. The output is -0.14, a weak negative relationship: longer posts are very slightly less positive on average. A value this close to zero means length tells you little about tone on its own.

posts['post_length'] = posts['tweet'].str.len()
length_sentiment_corr = posts['post_length'].corr(posts['compound'])
round(length_sentiment_corr, 3)
np.float64(-0.14)

Part 14: Real Sentiment vs Real Hashtag Count#

The same correlation check is now applied to how many hashtags a post uses. .str.len() on the list column counts hashtags per post, and .corr() relates that count to the compound score. The output is 0.191, a weak positive relationship: posts with more hashtags tend to be slightly more positive. That fits what the hashtag tables showed, since upbeat tags such as #love and #positive are common. Remember that correlation does not mean adding hashtags makes a post happier.

posts['hashtag_count'] = posts['hashtags'].str.len()
hashtag_sentiment_corr = posts['hashtag_count'].corr(posts['compound'])
round(hashtag_sentiment_corr, 3)
np.float64(0.191)

Part 15: Real Word-Level Positive and Negative Scores#

Looking inside the lexicon shows where the scores come from. analyzer.lexicon is a Python dictionary mapping words to sentiment ratings, which in VADER run roughly from -4 to +4. The code computes its length, but only the last expression is displayed: the ratings for three words. The output shows love at 3.2, hate at -2.7 and good at 1.9. So love is weighted more strongly than good, which is why posts full of love dominate the most positive list.

lexicon = analyzer.lexicon
len(lexicon)
lexicon['love'], lexicon['hate'], lexicon['good']
(3.2, -2.7, 1.9)

Part 16: Real Full Score Breakdown, One Post#

Pairing a post's text with its full score breakdown helps you reason about how VADER arrives at a number. The code shows the same fourth post scored earlier alongside its four scores. The text is #model i love u take with u all the time..., followed by garbled emoji characters. The scores match the earlier result, with a compound of 0.7249, and the positive weight comes mainly from love. The garbled emojis are almost certainly not being read as emojis, so any sentiment they carried is lost.

example_post = posts['tweet'].iloc[3]
full_scores = analyzer.polarity_scores(example_post)
example_post, full_scores
('#model   i love u take with u all the time in urð\x9f\x93±!!! ð\x9f\x98\x99ð\x9f\x98\x8eð\x9f\x91\x84ð\x9f\x91\x85ð\x9f\x92¦ð\x9f\x92¦ð\x9f\x92¦  ',
 {'neg': 0.0, 'neu': 0.719, 'pos': 0.281, 'compound': 0.7249})

Part 17: Real Neutral Posts Deep Dive#

Neutral posts are worth inspecting because they often hide mis-scored content. The code filters rows classed as Neutral and lists the first five texts. The output includes bihday your majesty, where the misspelling hides the positive word, a post that is just a list of @user tags, and a price index update. One post about Orlando, with #heabreaking (also misspelled), lands as Neutral despite being sad. Lexicon tools only know correctly spelled words, so cleaning text first can change results.

neutral_posts = posts[posts['sentiment_class'] == 'Neutral']
neutral_posts['tweet'].head(5).tolist()
['  bihday your majesty',
 ' @user camping tomorrow @user @user @user @user @user @user @user dannyâ\x80¦',
 ' â\x86\x9d #ireland consumer price index (mom) climbed from previous 0.2% to 0.5% in may   #blog #silver #gold #forex',
 'we are so selfish. #orlando #standwithorlando #pulseshooting #orlandoshooting #biggerproblems #selfish #heabreaking   #values #love #',
 'i get to see my daddy today!!   #80days #gettingfed']

Part 18: Real Sentiment Volatility, Standard Deviation#

Standard deviation measures how spread out scores are around the average, which tells you whether the conversation is uniform or polarised. .std() on the compound column gives 0.489, the same value seen in the earlier summary. Given an average of only 0.214 on a scale from -1 to 1, that is a wide spread: posts range from strongly negative to strongly positive rather than clustering near the mean. Tracking this number over time can show when a conversation becomes more divided.

sentiment_std = posts['compound'].std()
round(sentiment_std, 3)
np.float64(0.489)

Part 19: Real Positive-to-Negative Ratio#

A positive-to-negative ratio is a simple headline metric for brand or community health. Comparing the class column with a label gives True or False per post, and .sum() counts the Trues. Dividing positives by negatives gives 2.47, so there are about two and a half positive posts for every negative one, matching the 506 and 205 counts. Ratios are easy to compare across weeks or brands, but they ignore neutral posts and how strong each post is.

n_positive = (posts['sentiment_class'] == 'Positive').sum()
n_negative = (posts['sentiment_class'] == 'Negative').sum()
pos_neg_ratio = round(n_positive / n_negative, 2)
pos_neg_ratio
np.float64(2.47)

Part 20: Visualizing Real Sentiment vs Real Label#

Overlapping histograms let you compare the full shape of two distributions, not just their averages. plt.hist is called twice with 30 bins each, once for normal and once for flagged posts, and alpha=0.6 makes the bars partly transparent so both stay visible. The chart is saved to sentiment_by_label.png and closed, so nothing appears here. When you open it, expect the flagged bars to be much smaller simply because there are only 71 flagged posts, and look for them shifting to the left of the normal ones.

plt.figure(figsize=(9, 5))
plt.hist(normal_scores, bins=30, alpha=0.6, label='Real Normal Posts', color='seagreen')
plt.hist(flagged_scores, bins=30, alpha=0.6, label='Real Flagged Posts', color='crimson')
plt.xlabel('Real Compound Sentiment Score')
plt.ylabel('Real Number of Posts')
plt.title('Real Sentiment Distribution, Flagged vs Normal Posts')
plt.legend()
plt.tight_layout()
plt.savefig('sentiment_by_label.png', dpi=120)
plt.close()

Part 21: Real Sentiment-Based Simple Classifier#

If flagged posts are more negative, could sentiment alone flag them? This step tries a simple rule: predict a flag whenever compound is below -0.3, with .astype(int) turning True and False into 1 and 0. Comparing predictions with the real label and taking .mean() gives accuracy, the share of correct predictions: 81.2%. That sounds good, but with only 71 of 944 posts flagged, predicting no flags at all would score about 92.5%, so this rule is actually worse than doing nothing.

predicted_flag = (posts['compound'] < -0.3).astype(int)
accuracy = (predicted_flag == posts['label']).mean()
round(accuracy * 100, 1)
np.float64(81.2)

Part 22: Real Precision and Real Recall of This Rule#

Precision and recall show what accuracy hides. Precision is the share of posts the rule flagged that really were flagged; recall is the share of truly flagged posts the rule caught. The code counts true positives, false positives and false negatives with boolean & conditions, then divides. Precision is 0.165, so only about one in six rule flags is correct. Recall is 0.366, so the rule finds roughly a third of flagged posts. Always check both on imbalanced data like this.

true_positives = ((predicted_flag == 1) & (posts['label'] == 1)).sum()
false_positives = ((predicted_flag == 1) & (posts['label'] == 0)).sum()
false_negatives = ((predicted_flag == 0) & (posts['label'] == 1)).sum()
precision = true_positives / (true_positives + false_positives) if (true_positives + false_positives) > 0 else 0
recall = true_positives / (true_positives + false_negatives) if (true_positives + false_negatives) > 0 else 0
round(precision, 3), round(recall, 3)
(np.float64(0.165), np.float64(0.366))

Part 23: Real Cleaning Before Re-Scoring#

The masked @user tags are not real words, so it is worth confirming they are not distorting scores. The code removes every @user with .str.replace(..., regex=False), treating it as plain text, re-scores the posts and averages the absolute difference between old and new compound scores. The output is 0.0, so the tags have no effect at all because VADER's lexicon has no entry for them. Checks like this are quick and let you state confidently that a cleaning step does or does not matter.

posts['tweet_no_mentions'] = posts['tweet'].str.replace('@user', '', regex=False)
posts['compound_clean'] = posts['tweet_no_mentions'].apply(lambda t: analyzer.polarity_scores(t)['compound'])
(posts['compound'] - posts['compound_clean']).abs().mean().round(4)
np.float64(0.0)

Part 24: Saving the Real Scored Dataset#

Saving the scored data lets later work start from the results instead of re-running the analyser. The code selects the identifying columns plus the new compound, sentiment_class, hashtag_count and post_length features and writes them with to_csv(index=False), which avoids saving the row index as an extra column. Reading the file back and comparing row counts gives True, so all 944 rows were written. Note that sentiment_class will reload as plain text rather than a category.

output_cols = ['id', 'label', 'tweet', 'compound', 'sentiment_class', 'hashtag_count', 'post_length']
posts[output_cols].to_csv('sentiment_scored_posts.csv', index=False)
reloaded = pd.read_csv('sentiment_scored_posts.csv')
reloaded.shape[0] == posts.shape[0]
True

Part 25: Real Sanity Check, Compound Score Bounds#

VADER's compound score should always lie between -1 and 1. If it does not, something has gone wrong in the scoring code. .between(-1.0, 1.0) returns True for each score inside the range (inclusive of both ends), and .all() returns True only if every one passes. The output is True. Range checks like this cost one line and catch silent bugs early, such as accidentally storing the wrong column or mixing up score types.

posts['compound'].between(-1.0, 1.0).all()
np.True_

Part 26: Real Sanity Check, Classification Consistency#

This check confirms that the pd.cut classification matches the intended thresholds. Two conditions are tested: every Positive post has a compound above 0.05, and every Negative post is below -0.05. Only the last line is displayed, so the True in the output confirms the Negative check; the Positive check ran but was not shown. To see both results at once, put them in one tuple or wrap each in print().

(posts.loc[posts['sentiment_class'] == 'Positive', 'compound'] > 0.05).all()
(posts.loc[posts['sentiment_class'] == 'Negative', 'compound'] < -0.05).all()
np.True_

Part 27: Real Sentiment Summary by Word Count Bucket#

Grouping by word count shows whether sentiment changes with post size in a way a single correlation might miss. .str.split().str.len() counts words, pd.cut groups them into buckets of up to 5, 6 to 10, 11 to 15 and more than 15 words, and groupby(..., observed=True) averages compound per bucket, showing only buckets that contain posts. The pattern is not a straight line: 6 to 10 word posts average 0.326, while posts over 15 words average just 0.033. Very long posts are the most neutral or mixed on average.

posts['word_count'] = posts['tweet'].str.split().str.len()
word_bins = pd.cut(posts['word_count'], bins=[0, 5, 10, 15, 100])
posts.groupby(word_bins, observed=True)['compound'].mean().round(3)
word_count
(0, 5]       0.167
(5, 10]      0.326
(10, 15]     0.285
(15, 100]    0.033
Name: compound, dtype: float64

Part 28: Real Extreme Sentiment Count#

Strong emotion often drives engagement, so counting extreme posts is useful. Comparing compound with 0.8 and -0.8 gives True or False per post, and .sum() counts the Trues. The output shows 126 strongly positive posts against 21 strongly negative ones, a ratio of about 6 to 1, much more lopsided than the 2.47 ratio across all classes. So while negative posts exist, intensely negative ones are rare in this sample.

extreme_positive = (posts['compound'] > 0.8).sum()
extreme_negative = (posts['compound'] < -0.8).sum()
extreme_positive, extreme_negative
(np.int64(126), np.int64(21))

Part 29: Real Overall Sentiment Health Score#

A single health score is a convenient top-line figure for dashboards and reports. Here it is simply the mean compound score across all posts, rounded to three decimals. The output is 0.214, the same mean seen in the earlier summary: mildly positive overall. Keep in mind that a mean can look healthy even when a vocal minority is very negative, so report it alongside the class counts or the extreme-post counts.

overall_health = round(posts['compound'].mean(), 3)
overall_health
np.float64(0.214)

Part 30: Real Recap Print#

The recap pulls the main findings into one sentence built from variables, so it stays correct if the data changes. The output reports 944 posts scored, an average sentiment of 0.214, flagged posts averaging -0.11 against 0.24 for normal posts, and a p-value shown as 0.0. That p-value has been rounded to five decimals, so it means less than 0.000005, not exactly zero. When writing for others, phrase it as p < 0.001 instead.

print(f'Scored {len(posts)} real posts: average sentiment {overall_health}, flagged posts averaged {round(sentiment_by_label[1],3)} versus {round(sentiment_by_label[0],3)} for normal posts, a real statistically significant gap (p={round(p_value,5)}).')
Scored 944 real posts: average sentiment 0.214, flagged posts averaged -0.11 versus 0.24 for normal posts, a real statistically significant gap (p=0.0).

Wrap-Up: What You Learned#

  • A real lexicon-based sentiment tool scores text by summing real per-word polarity values from a real curated dictionary, no machine learning required.
  • Comparing real sentiment against an independent real content label, and testing that gap statistically, is how you check whether sentiment genuinely tracks something real.
  • Real hashtags, real post length, and real word count can all be correlated against sentiment to find real patterns in how people write.
  • A real sentiment-only classifier caught only some real flagged content, an honest reminder that sentiment and moderation are related but genuinely different signals.
  • Cleaning real anonymized tokens before re-scoring is a small real step that keeps sentiment scoring focused on genuinely meaningful words.
  • Next video: real customer churn and retention analysis, shifting from real text sentiment to real subscription behavior and retention.

Practice on your own

  • Rebuild the sentiment classes with stricter cut-offs of -0.3 and 0.3 using pd.cut, and compare the new Positive, Neutral and Negative counts with 506, 233 and 205.
  • Try thresholds of -0.1, -0.5 and -0.7 for the simple flagging rule and record precision and recall for each. Note which threshold gives the best balance.
  • Lower the minimum hashtag count from 5 to 3 in the hashtag sentiment step and see which new hashtags enter the most negative ten.

The next video moves from text to behaviour, with a customer churn and retention analysis based on subscription data. Watch the video walkthrough to see each step explained live.

🧠 Test yourself

Pick an answer to check it. Your best score is saved in this browser.

  1. What does a lexicon-based tool like VADER do to score a post?

  2. The lesson classifies compound scores with bins=[-1.01, -0.05, 0.05, 1.01] and labels Negative, Neutral, Positive. Which class does a compound score of exactly 0.05 get?

  3. Why does the lesson run a t-test on compound scores for flagged vs normal posts?

  4. The simple rule compound < -0.3 caught only some flagged posts. What is the lesson's takeaway?

  5. Why does the lesson remove the @user tokens before re-scoring posts?

💻 Practice exercises

Try each one in the notebook or in your own editor before opening the solution.

  1. Build a tiny sentiment lexicon dictionary (e.g. love: 3, great: 2, good: 1, bad: -1, hate: -3, awful: -2). Write a function that scores a sentence by summing the values of its words, and another that labels the score Positive (> 1), Negative (< -1) or Neutral. Apply both to 6 short sentences and print them.

    💡 Hint

    Lower-case the text, split on spaces and use dict.get(word, 0).

    ✅ Show solution
    lexicon = {'love': 3, 'great': 2, 'good': 1, 'bad': -1, 'hate': -3, 'awful': -2}
    
    def score(text):
        return sum(lexicon.get(word, 0) for word in text.lower().split())
    
    def label(s):
        if s > 1:
            return 'Positive'
        if s < -1:
            return 'Negative'
        return 'Neutral'
    
    sentences = [
        'I love this great product',
        'This is good',
        'Awful service and bad packaging',
        'I hate waiting',
        'It arrived on Tuesday',
        'Great price and good quality',
    ]
    for s in sentences:
        sc = score(s)
        print(f'{sc:>3}  {label(sc):<9}  {s}')
    
  2. Create a 12-row DataFrame of posts with a hand-typed compound score and a content label (0 = normal, 1 = flagged). Classify scores with pd.cut using bins [-1.01, -0.05, 0.05, 1.01], print the class counts, the mean compound per label, and the Welch t-test p-value between flagged and normal scores.

    💡 Hint

    Use groupby('label')['compound'].mean() and stats.ttest_ind(..., equal_var=False).

    ✅ Show solution
    import pandas as pd
    from scipy import stats
    
    posts = pd.DataFrame({
        'compound': [0.8, 0.5, 0.1, -0.6, 0.0, 0.7, -0.8, -0.4, -0.2, 0.3, -0.7, -0.5],
        'label':    [0,   0,   0,   0,    0,   0,   1,    1,    0,    0,   1,    1],
    })
    labels = ['Negative', 'Neutral', 'Positive']
    posts['sentiment_class'] = pd.cut(posts['compound'], bins=[-1.01, -0.05, 0.05, 1.01], labels=labels)
    print(posts['sentiment_class'].value_counts().reindex(labels))
    print(posts.groupby('label')['compound'].mean().round(3))
    flagged = posts.loc[posts['label'] == 1, 'compound']
    normal = posts.loc[posts['label'] == 0, 'compound']
    t, p = stats.ttest_ind(flagged, normal, equal_var=False)
    print('t =', round(t, 3), 'p =', round(p, 4))
    
  3. Create 12 posts with text (including #hashtags), a hand-typed compound score and a label. Extract hashtags with a regex, compute mean sentiment per hashtag (explode), and evaluate the rule compound < -0.3 as a flag predictor: accuracy, precision and recall (guard against divide-by-zero), plus the positive-to-negative post ratio.

    💡 Hint

    Use str.findall(r'#\w+'), explode, and Boolean masks for TP/FP/FN.

    ✅ Show solution
    import pandas as pd
    
    posts = pd.DataFrame({
        'tweet': ['Love this #gold rally', 'Hate it #gold', 'So thankful #positive', 'Awful day #forex',
                  'Great news #positive #thankful', 'Meh #forex', 'Terrible #gold crash', 'Happy #positive',
                  'Sad #forex loss', 'Nice #gold find', 'Worst ever #blog', 'Fine #blog'],
        'compound': [0.6, -0.7, 0.7, -0.6, 0.8, 0.0, -0.8, 0.6, -0.5, 0.4, -0.7, 0.1],
        'label':    [0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0],
    })
    posts['hashtags'] = posts['tweet'].str.findall(r'#\w+')
    tag_sentiment = posts.explode('hashtags').groupby('hashtags')['compound'].mean().sort_values()
    print(tag_sentiment.round(2))
    
    predicted = (posts['compound'] < -0.3).astype(int)
    tp = ((predicted == 1) & (posts['label'] == 1)).sum()
    fp = ((predicted == 1) & (posts['label'] == 0)).sum()
    fn = ((predicted == 0) & (posts['label'] == 1)).sum()
    precision = tp / (tp + fp) if (tp + fp) else 0
    recall = tp / (tp + fn) if (tp + fn) else 0
    print('Accuracy:', round((predicted == posts['label']).mean() * 100, 1), '%')
    print('Precision:', round(precision, 3), 'Recall:', round(recall, 3))
    
    n_pos = (posts['compound'] > 0.05).sum()
    n_neg = (posts['compound'] < -0.05).sum()
    print('Positive-to-negative ratio:', round(n_pos / n_neg, 2))
    

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.