<jupyter_start><jupyter_text>Week 8: Social media sentiment analysis This final week we are analyzing our social media data with sentiment analysis. There are a few ways to perform sentiment analysis:- use a dictionary lookup: words have corresponding sentiment values we use to calculate overall sentiment for a text chunk    - scores can range from -1 to +1 or another range    - we can add other rules for negations (e.g. the word 'not' flips the sign of the score for the next word)- use a machine learning model; train a classifier on labeled data    - label can be -1, 0, or +1 for negative, neutral, or positive    We will start with a demonstration of the keyword lookup method and also cover the machine learning method.  Load data First we need to load our data from our SQLite database we saved last week:<jupyter_code>import sqlite3
import pandas as pd

con = sqlite3.connect('co_reddit.sqlite')
df = pd.read_sql_query('SELECT * from posts;', con)
con.close()
df<jupyter_output><empty_output><jupyter_text>Since most of our text is missing for this subreddit, we'll just use the title of the posts. However, if you have text from each post or comments for each post, you could combine those with the title.  Keyword sentiment  We will start with a keyword sentiment analysis technique. Let's first load a dictionary of words and sentiment values:<jupyter_code>sentiment_df = pd.read_csv('AFINN-en-165.txt', sep='\t', names=['word', 'score'], index_col='word')
sentiment_df
sentiment_dict = sentiment_df.to_dict()['score']<jupyter_output><empty_output><jupyter_text>Here, we load the data into a dataframe. There is no header so we provide the column names with the `names` argument. Then we set the word as the index, which helps with the next step where we convert the dataframe to a dictionary. This has the column names as keys, then the values are dictionaries with the index value as keys and the column values as values. So we get a dictionary like this:```python{'abandon': -2, 'abandoned': -2, 'abandons': -2, 'abducted': -2, ...}```Now we can get the average sentiment for each string we have in our original dataframe, which is the title of each post. We get the scores for each word and take the average for each title:<jupyter_code>import numpy as np

title_sentiments = []
for title in df['title']:
    words = title.lower().split()
    this_titles_sentiments = []
    for w in words:
        if w in sentiment_dict.keys():
            this_titles_sentiments.append(sentiment_dict[w])
        else:
            this_titles_sentiments.append(0)
            
    
    title_sentiments.append(np.mean(this_titles_sentiments))
df['keyword_sentiment'] = title_sentiments
df['keyword_sentiment'].plot.hist(bins=30)
df['keyword_sentiment'].mean()<jupyter_output><empty_output><jupyter_text>We can easily create a new column in our original dataframe with sentiments and then use it to create plots. Since this lookup dictionary method doesn't have a lot of words,  we get a lot of 0 scores. These don't necessarily mean the sentiments are neutral, but may mean that no words were in the lookup dictionary. Overall, we can see the sentiments trend towards a positive value.A more interesting result might be had by looking at the sentiment for comments on posts. Last week in the advanced section we collected comments on posts. We can perform a similar analysis as above on the comments to see what the sentiment is on each post. Then we can combine the posts and comments dataframes to see the titles of the posts with some of the most positive and negative sentiments to help us understand what some of the most positive and negative topics are in the subreddit. This is left as an optional challenge for you in the assignment.Let's look at some of the top and bottom sentiment posts:<jupyter_code>df.sort_values(by='keyword_sentiment')[['title', 'keyword_sentiment']]
# full titles of the bottom 10 sentiments
df.sort_values(by='keyword_sentiment')['title'].to_list()[:10]
# full titles of the top 10 sentiments
df.sort_values(by='keyword_sentiment', ascending=False)['title'].to_list()[:10]<jupyter_output><empty_output><jupyter_text>Using Python packages for sentiment analysis There are a few Python packages that use rule-based and keyword-based sentiment analysis. We will look at VADER and textblob here, although there are probably more. We need to install these first: `conda install -c conda-forge textblob vadersentiment -y`. Another option we won't cover here is the `flair` package. We can use textblob and vader like so:<jupyter_code>from textblob import TextBlob

tb = TextBlob("April snowstorms make for some beautiful pictures...")
tb.sentiment<jupyter_output><empty_output><jupyter_text>Textblob requires us to use their TextBlob class on a string. Then we can get the sentiment score. It gives us polarity (-1 to =1) and subjectivite (0 to 1, objective to subjective). We can see this sentence is rated fully subjective and has a positive sentiment, which makes sense. We can apply it to the whole dataframe like so:<jupyter_code>def get_tb_sentiment(text):
    tb = TextBlob(text)
    return tb.sentiment[0]

df['tb_sentiment'] = df['title'].apply(get_tb_sentiment)<jupyter_output><empty_output><jupyter_text>Because we need to do a few steps to get the polarity (sentiment) score, we wrote a short function to do it. Then we use the pandas `apply` function to apply this to the whole dataframe.Note: If you have a lot of data and want to speed this up, look at the `swifter` package for parallelizing the `apply` function.Now we can look at the distribution of these sentiments:<jupyter_code>df['tb_sentiment'].plot.hist(bins=30)
df['tb_sentiment'].mean()<jupyter_output><empty_output><jupyter_text>We can see the distribution looks similar but is more spread out than our AFINN method. textblob has more words in its lookup dictionary than the AFINN dictionary we used.VADER is similar to use:<jupyter_code>from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

analyzer = SentimentIntensityAnalyzer()
analyzer.polarity_scores("April snowstorms make for some beautiful pictures...")
analyzer.polarity_scores("April snowstorms make for some beautiful pictures...")['compound']<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07074417119131268, 'fraction_numerical': 0.005429575215586075, 'mean_word_length': 5.045366795366795, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '401078', 'license': 'MIT', 'n_tokens_mistral': 1731, 'n_tokens_neox': 1680, 'n_words': 816, 'path': 'FTE/week8/Week_8_FTE.ipynb', 'repo_name': 'kscully-dotcom/msds600', 'size': '6262'}