<jupyter_start><jupyter_text>Sentiment AnalysisIn this notebook we will learn how to measure sentiment in text. Below is the overview of this notebook.0. Install required packages (only need to do this the first time we run the notebook)1. Load corpus of tweets2. Load sentiment classifier, which in this case is a BERT transformer3.  Measure tweet sentiment4. Analyze sentiment of tweets, compare to retweet counts5. Visualize UMAP transformer embeddingsBelow are some cool blogs you can read to learn more about the BERT transformer.http://jalammar.github.io/illustrated-bert/https://towardsdatascience.com/deconstructing-bert-distilling-6-patterns-from-100-million-parameters-b49113672f77https://github.com/jessevig/bertviz  Install requirementsThe pip command lets you install python packages.  We need two packages today.1) transformers - this lets us use pre-trained transformer models. 2) umap - this lets you embed high dimensional vectors in low dimensions.<jupyter_code>#install transformers
!pip install  transformers 

#install umap
!pip install umap-learn<jupyter_output><empty_output><jupyter_text>Clone GitHub RepositoryThis will clone the repository to your machine.  This includes the code and data files.  Then change into the directory of the repository.<jupyter_code>!git clone https://github.com/zlisto/sentiment_analysis

import os
os.chdir("sentiment_analysis")<jupyter_output><empty_output><jupyter_text>Import Packages<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
import codecs
import umap
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

print(f"device ={device}")<jupyter_output><empty_output><jupyter_text>Sentiment Classification with BERTWe will pass the tweets through a pre-trained sentiment classifier with a BERT core.  Then we will plot the tweets with UMAP and color them by their sentiment.  Hopefully the positive and negative are in different regions of the plot.  Download Pre-Traine Model and TokenizerWe will download the model and tokenizer from https://huggingface.co/nlptown/bert-base-multilingual-uncased-sentiment.  This is a pre-trained model in the huggingface library that was trained on product reviews in multiple languages.  The output sentiment is between 1 and 5.There are many other models on huggingface that you can find here: https://huggingface.co/models?pipeline_tag=text-classification.We will create a tokenizer for the model called `tokenizer` and create the model itself, which we call `model`.  Every model needs its own tokenizer which tells it how to map text into the proper input vectors.<jupyter_code>tokenizer = AutoTokenizer.from_pretrained("nlptown/bert-base-multilingual-uncased-sentiment")
model = AutoModelForSequenceClassification.from_pretrained("nlptown/bert-base-multilingual-uncased-sentiment")<jupyter_output><empty_output><jupyter_text>Define Sentiment Classifier and Transformer Embedding FunctionWhen we pass text through our transformer model, we get many pieces of data in the output. First, we get the sentiment of the text.  Second, we get the text embedding in the final layer of the transformer.  Remember, inside the transformer we turn the input text into a high dimensional vector. This is the transformer embedding, and it is designed to separate text based on sentiment.We will create a function called `sentiment_classifier` which takes as input a string `text`, a transformer model called `model`, and a tokenizer called `tokenizer`, and returns the sentiment and embedding of the text.  The raw sentiment output of the model is a probability for each sentiment value.  The function will return the average sentiment based on these probabilities.<jupyter_code>def sentiment_classifier(text,model,tokenizer):
    inputs = tokenizer.encode_plus(text, return_tensors='pt', add_special_tokens=True)

    token_type_ids = inputs['token_type_ids']
    input_ids = inputs['input_ids']

    output = model(input_ids, token_type_ids=token_type_ids,return_dict=True,output_hidden_states=True)
    logits = np.array(output.logits.tolist()[0])
    prob = np.exp(logits)/np.sum(np.exp(logits))
    sentiment = np.sum([(x+1)*prob[x] for x in range(len(prob))])  #use this line if you want the mean score
    embedding = output.hidden_states[12].detach().numpy().squeeze()[0]
    
    return sentiment, embedding<jupyter_output><empty_output><jupyter_text>Sentiment Classification ExampleNow we can test the model on some text.  Feel free to try any text you want here.  Just put your text in the list `Text`.<jupyter_code>Text = ["This class is kinda boring, but informative", 
        "This class is awesome", 
        "this class is dope", 
        "this class is stupid",
        "this class is fun",
       "this class is fun!",
       "this class is :(",
       "this class is :)",
]






for text in Text:
    sentiment,embedding = sentiment_classifier(text,model,tokenizer)
    print(f"Text: {text}\nSentiment:{sentiment:.2f}\n")<jupyter_output><empty_output><jupyter_text>Load Tweets Load the tweets from the file `"data/tweets_sentiment_embedding.csv"` into a dataframe `df`.<jupyter_code>df = pd.read_csv("data/tweets_sentiment_embedding.csv")

df = df[['screen_name','text','retweet_count']]
ntweets = len(df)
print(f"dataframe has {ntweets} tweets\n")
[print(f"Tweets from {x}") for x in df.screen_name.unique()];<jupyter_output><empty_output><jupyter_text>Calculate Sentiment and Transformer Embedding of TweetsWe use a for loop iterating over the rows in `df` to calculate the sentiment and transformer embedding of each tweet.  We store the sentiment in the list `Sentiment` and embeddings in the list `Embedding`.  We then add the sentiment to a column in `df` and save the resulting dataframe to a file `"data/lec_12_tweets_sentiment_embedding.csv"`.  This may take a long time on your machine unless you have a GPU.<jupyter_code>%%time
c = 0
Sentiment = []
Embedding = []
for index,row in df.iterrows():
    c+=1
    if c%1000==0:print(f"Tweet {c}/{len(df_tweets)}")
    sentiment,embedding = sentiment_classifier(row.text,model,tokenizer)
    Sentiment.append(sentiment)
    Embedding.append(embedding)

df['sentiment'] = Sentiment
df.to_csv("data/tweets_sentiment.csv")
df.head()<jupyter_output><empty_output><jupyter_text>UMAP Transformer Embedding of TweetsThe tranformer turns the input text into a high dimensional vector.  This is the transformer embedding, and it is designed to separate text based on sentiment.  However, we can't really visualize such a high-dimensional object.  But no worries, UMAP will let us embed this high-dimensional vector into 2 dimensions.  UMAP stands for Uniform Manifold Approximation and Projection for Dimension Reduction.  You can read more about UMAP on its website https://umap-learn.readthedocs.io/en/latest/.We apply UMAP to the transformer embedding `Embedding` to create the UMAP transformer embedding `umap_bert_embedding`.  Before doing this we have to convert `Embedding` from a list to an array.  Then we save the UMAP embedding values in `df` as `"umap_transformer_x"` and `"umap_transformer_y"`.  The dataframe is saved to the file `"data/lec_12_tweets_sentiment_embedding.csv"`.<jupyter_code>%%time

Embedding = np.array(Embedding)
umap_bert_embedding = umap.UMAP(n_components = 2, metric='cosine').fit_transform(Embedding)
df['umap_transformer_x'] = umap_bert_embedding[:,0]
df['umap_transformer_y'] = umap_bert_embedding[:,1]

df.to_csv('data/tweets_sentiment_embedding.csv')<jupyter_output><empty_output><jupyter_text>Analyze Tweet Sentiment  Load Tweets and SentimentOnce we save the tweet sentiments, the next time we run the notebook we can just load this data instead of recalculating the sentiment.  The sentiment is in the file `"data/tweets_sentiment_embedding.csv"`.<jupyter_code>df = pd.read_csv("data/tweets_sentiment_embedding.csv")

print(f"dataframe has {len(df)} tweets")
[print(f"Tweets from {x}") for x in df.screen_name.unique()];
df.sample(n=5)<jupyter_output><empty_output><jupyter_text>Average User SentimentWe make a bar plot of the average sentiment of each user.  We do this by grouping `df` by `"screen_name"` and then taking the mean of each group with the `mean` function.  We save this a new dataframe `df_plot` and then use the `barplot` function to make the bar plot.<jupyter_code>fig = plt.figure(figsize = (16,6))
sns.barplot(data = df, y= 'sentiment', x = 'screen_name' )
plt.ylim([2,4])
plt.ylabel("Mean sentiment",fontsize = 16)
plt.grid()
plt.ylim([0,5]);
plt.show()<jupyter_output><empty_output><jupyter_text>Sentiment Distribution per UserWe can make histograms of the tweet sentiment for each user.  We use a for loop to iterate through each screen name.We use the `histplot` function to make a histogram of the tweets of each user.  We also add a title to each plot so we know whose tweets they are.<jupyter_code>for screen_name in df.screen_name.unique():
  sns.histplot(data = df[df.screen_name==screen_name], x = 'sentiment')
  plt.title(f"Tweets of {screen_name}", fontsize = 14)
  plt.show()<jupyter_output><empty_output><jupyter_text>Look at Tweets with Extreme SentimentWe can select tweets of each user with very high or very low sentiment and print them out.  We do this by keeping the rows of `df` with the corresponding screen name, and then using `sort_values` to sort the user's tweets by sentiment.  We set `ascending = True` inside the `for` loop to get the most postive tweets, and set `ascending = False` to get the most negative tweets.  We set `ndisplay` equal to the number of tweets we want to print per user.To show all the funny Twitter characters, we need to use the `decode` function in the `codecs` module.<jupyter_code>ndisplay = 3
print(f"Top {ndisplay} most postive tweets per screen name")
for screen_name in df.screen_name.unique():
    df_display = df[(df.screen_name==screen_name)].sort_values(by = ['sentiment'], 
                                                               ascending = False)
    c=0
    print(f"\n{screen_name}")
    for index,row in df_display.iterrows():
        c+=1
        text = codecs.decode(row.text, 'unicode_escape')
        print(f"\tsentiment = {row.sentiment:.2f}: {text}")
        if c>=ndisplay:break

print("".join('-' * 200))

print(f"Top {ndisplay} Most Negative Tweets per Screen Name")
for screen_name in df.screen_name.unique():
    df_display = df[(df.screen_name==screen_name)].sort_values(by = ['sentiment'], ascending = True)
    c=0
    print(f"\n{screen_name}")
    for index,row in df_display.iterrows():
        c+=1
        text = codecs.decode(row.text, 'unicode_escape')
        print(f"\tsentiment = {row.sentiment:.2f}: {text}")
        if c>=ndisplay:break<jupyter_output><empty_output><jupyter_text>Box Plot of Retweet Count vs SentimentTo see this correlation of extreme sentiment and retweet count, we can make a box plot. The box plot (or box-and-whisker plot) shows the distribution of quantitative data in a way that facilitates comparisons between variables or across levels of a categorical variable. The box shows the quartiles of the dataset while the whiskers extend to show the rest of the distribution, except for points that are determined to be “outliers” using a method that is a function of the inter-quartile range.  More details on the `boxplot` function can be found here: https://seaborn.pydata.org/generated/seaborn.boxplot.html We first create a column called `star` that rounds the sentiment to the nearest integer.  Then we can make a boxplot of the retweet count versus the tweet star sentiment.  We will have to limit the y-axis in order to see the pattern clearly.  We set the maximum value on the y-axis equal to `ymax`, which we set to be an upper quantile of the retweet count.<jupyter_code>df['star'] = df.sentiment.round()

fig = plt.figure(figsize = (12,8))
sns.boxplot(data=df, x="star", y="retweet_count")
ymax = df.retweet_count.quantile(0.95)
plt.ylim([0,ymax])
plt.xlabel("Sentiment", fontsize = 16)
plt.ylabel("Retweet Count", fontsize = 16)
plt.grid()
plt.show()<jupyter_output><empty_output><jupyter_text>Box Plot per UserWe can also look at a boxplot for each individual user.<jupyter_code>for screen_name in df.screen_name.unique():
    fig = plt.figure(figsize = (12,8))
    df_plot = df[df.screen_name==screen_name]
    ax = sns.boxplot(data=df_plot, x="star", y="retweet_count")
    ymax = df_plot.retweet_count.quantile(q=0.9)
    plt.ylim([0,ymax])
    plt.xlabel("Sentiment", fontsize = 16)
    plt.ylabel("Retweet Count", fontsize = 16)
    plt.title(f"{screen_name}", fontsize = 16)
    plt.grid()
    plt.show()<jupyter_output><empty_output><jupyter_text>Visualize Transformer EmbeddingNow we will visulize the transformer embeddings using UMAP to see how the sentiment is distributed.  Scatter Plot of UMAP Transformer Tweet EmbeddingsWe can make a scatter plot of the UMAP transformer embeddings of the tweets.  We will color the data points by the user screen name.  We will also make another plot next to this plot where we color the data points by sentiment.  You set the column for the datapoint color with the `hue` parameter.  You can choose a color palette with the `palette` parameter.  There are many palettes you can choose from, but for discrete values like `"screen_name"` use the `"bright"` palette and for continous values like `"sentiment"` use the `"vlag"` palette.  Of course feel free to try other palettes. A complete list can be found here: https://seaborn.pydata.org/tutorial/color_palettes.html<jupyter_code>for screen_name in df.screen_name.unique():
  fig = plt.figure(figsize = (8,8))
  sns.scatterplot(data=df[df.screen_name == screen_name], 
                  x="umap_transformer_x", 
                  y="umap_transformer_y", 
                  hue="sentiment", 
                  palette="vlag", s=15)
  plt.title(f"UMAP Transformer Embedding for {screen_name}")
  plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08858687918030454, 'fraction_numerical': 0.006688487263412551, 'mean_word_length': 4.875836120401337, 'pattern_counts': {'":': 0, '<': 65, '<?xml version=': 0, '>': 67, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '91494', 'license': 'MIT', 'n_tokens_mistral': 4354, 'n_tokens_neox': 4183, 'n_words': 1531, 'path': 'Sentiment_Analysis.ipynb', 'repo_name': 'zlisto/sentiment_analysis', 'size': '14054'}