<jupyter_start><jupyter_text>This notebook contains sentiment predictions on Reddit text using a model trained on Metacritic/Amazon/IMDB reviews (no pre-training).<jupyter_code>%matplotlib inline
import string
import pandas as pd
import matplotlib.pyplot as plt
from textblob import TextBlob
from collections import Counter
import nltk
# nltk.download('stopwords')
from nltk.corpus import stopwords
import numpy as np
# read metacritic predictions
df = pd.read_csv('/home/ydu/BERT/bert_mgpu/predict/081419/no-pretrain-metatune/metatune.tsv', sep='\t') 
df['meta'] = df['1']*2-1
df = df.drop(['0', '1'], axis=1)
# read amazon predictions
tmp = pd.read_csv('/home/ydu/BERT/bert_mgpu/predict/081419/no-pretrain-amazontune/amazontune.tsv', sep='\t') 
df['amazon'] = tmp['1']*2-1
# read imdb predictions
tmp = pd.read_csv('/home/ydu/BERT/bert_mgpu/predict/081419/no-pretrain-imdbtune/imdbtune.tsv', sep='\t') 
df['imdb'] = tmp['1']*2-1
# add TextBlob
df['textblob'] = [TextBlob(text).sentiment.polarity for text in df['text']]
col=['text','amazon','meta','imdb','textblob','n_chars','n_words']
df = df[col]
df.drop(['n_words','n_chars'],axis=1)<jupyter_output><empty_output><jupyter_text>Observations BERT predictions for Reddit are fairly bipolar, which may be an indicator that they are fairly well calibrated for this dataset.<jupyter_code>fig, axs = plt.subplots(2, 2, figsize=(8,6))
fig.suptitle('Fine-tune on source dataset, no pre-training', fontsize=14)

axs[0][0].hist(df['textblob'])
axs[0][0].set_title('TextBlob')
axs[0][0].set_ylabel('count')
axs[0][0].set_ylim([0,600000])
axs[0][0].get_xaxis().set_visible(False)

axs[0][1].hist(df['amazon'])
axs[0][1].set_title('Amazon')
axs[0][1].set_ylim([0,600000])
axs[0][1].get_xaxis().set_visible(False)
axs[0][1].get_yaxis().set_visible(False)

axs[1][0].hist(df['meta'])
axs[1][0].set_title('Metacritic')
axs[1][0].set_xlabel('polarity')
axs[1][0].set_ylabel('count')
axs[1][0].set_ylim([0,600000])

axs[1][1].hist(df['imdb'])
axs[1][1].set_title('IMDB')
axs[1][1].set_xlabel('polarity')
axs[1][1].set_ylim([0,600000])
axs[1][1].get_yaxis().set_visible(False)<jupyter_output><empty_output><jupyter_text>There may be a similar bipolar relation relationship between the text length and the prediction<jupyter_code>df['n_words'] = df['text'].apply(lambda x: len(x.split()))
sample = df.sample(100000)
plt.plot(sample['textblob'], sample['n_words'], '.', label='TextBlob')
plt.plot(sample['amazon'], sample['n_words'], '.', alpha=0.8, label='Amazon')
plt.plot(sample['meta'], sample['n_words'], '.', alpha=0.6, label='Metacritic')
plt.plot(sample['imdb'], sample['n_words'], '.', alpha=0.4, label='IMDB')

plt.xlabel('Polarity')
plt.ylabel('# words')
plt.title("Polarity vs. Review Length")
plt.legend()<jupyter_output><empty_output><jupyter_text>Examples<jupyter_code>pd.set_option('display.max_colwidth', -1)<jupyter_output><empty_output><jupyter_text>Sentiment sorted by Amazon model prediction<jupyter_code>sample.sort_values('amazon').drop(['n_chars','n_words'], axis=1)<jupyter_output><empty_output><jupyter_text>Compare Amazon model prediction with others* There are situations that models disagree.-- Amazon thinks it's negative while Metacricit/IMDB think it's positive: 320/100,000 examples-- Amazon thinks it's positive while Metacritic/IMDB think it's negative: 4,544/100,000 examples (Amazon tends to pick up positive language)<jupyter_code>idx = (sample['amazon'] < -0.8) & ((sample['meta']>0.8) | (sample['imdb'] > 0.8))
sample.loc[idx].sort_values('amazon').drop(['n_words','n_chars','textblob'],axis=1)
idx2 = (sample['amazon'] > 0.8) & ((sample['meta']<-0.8) | (sample['imdb'] <- 0.8))
sample.loc[idx2].sort_values('amazon').drop(['n_words','n_chars','textblob'],axis=1)<jupyter_output><empty_output><jupyter_text>TextBlob "false positives"These are examples that Amazon BERT classified as relatively neutral (0 $\pm$ .1) ranked by TextBlob polarity. If we "trust" BERT's prediction, then we can view these as false positives. Some observations:* Compared to TextBlob BERT is fairly insensitive to gamer slang (e.g., nasty, sick) and personal ad-hominem attacks* There still is the length issue -- In some cases, short sentences with unambiguous sentiment fail to elicit high sentiment for BERT (e.g., "That was an annoying feature", "Imagine blackout at this time of night would be awesome.")<jupyter_code>idx3 = (sample['amazon'] > -.1) & (sample['amazon'] < .1)
sample.loc[idx3].sort_values('textblob').drop(['n_chars','n_words'],axis=1)<jupyter_output><empty_output><jupyter_text>TextBlob "false negatives"We can do the inverse and look at TextBlob "false negatives" -- examples TetBlob classifies as reltaively neutral (0 $\pm$ .1) ranked by BERT polarity. If we "trust" BERT's prediction, these are high-sentiment cases TextBlob fails to uncover. Some observations:* High emphasis on dollar value on negative end of the sentiment spectrum (likely due to Metacritic reviews emphasising whether a purchase was worth their money). TextBlob fails to uncover these cases because there aren't typically any sentiment-laden adjectives or they get averaged out across the comment.* The positive end of the sentiment spectrum isn't strictly positive. Seems to be more rationalized positivity (stuff like "despite it's flaws, I like this game"). I suspect this may be due to the way Metacritic reviews are written -- users have to point out some flaws as a reviewer, but end up awarding a high score, which we use as a proxy for positive sentiment.<jupyter_code>idx4 = (sample['textblob'] > -.1) & (sample['textblob'] < .1)
sample.loc[idx4].sort_values('amazon').drop(['n_chars','n_words'],axis=1)<jupyter_output><empty_output><jupyter_text>"Induced" sentiment for a given wordWe can look at the average sentiment for every comment that contains a specific word, even if that word is not intrinsically polarizing. We can then compare the scores assigned by TextBlob (often $\mu=0$ unless the word *is* intrinsically polarizing) to BERT to look at negative and positive sentiment surrounding the word.Fun words to try:* microtransactions, mtx* netcode, lag* activision* fortnite* hate, love* shit, lol(see also top500 list below)<jupyter_code>word = 'mtx'

sample.loc[sample['text'].apply(lambda x: word in x.lower().split()), 
           ['amazon', 'meta','imdb','textblob']].plot(kind='hist')
sample.loc[sample['text'].apply(lambda x: word in x.lower().split()), 
           ['amazon', 'meta','imdb','textblob']].mean()
sample.loc[sample['text'].apply(lambda x: word in x.lower().split()), ['text', 'amazon', 'meta','imdb']].sort_values('amazon')<jupyter_output><empty_output><jupyter_text>Ranking induced sentiment for top 500 most-common wordsInteresting to see that there are a number of non-adjectives on the lower end of the spectrum surround Activision, money, paying, etc., consistent with observations of TextBlob "false negatives"<jupyter_code>depunct = str.maketrans('', '', string.punctuation)
stop = set(stopwords.words('english'))
corpus = []

for text in sample['text']:
    corpus += [word for word in text.lower().translate(depunct).split() if word not in stop]
    
counts = Counter(corpus)
top500words = sorted(counts, key=counts.get)[::-1][:500]
# very inefficient code

top500polarity = {}
top500polarity_meta = {}

for word in top500words:
    top500polarity[word] = sample.loc[sample['text'].apply(lambda x: word in x.lower().split()), 
                                      'amazon'].mean()
    top500polarity_meta[word] = sample.loc[sample['text'].apply(lambda x: word in x.lower().split()), 
                                      'meta'].mean()
pd.DataFrame(list(top500polarity.items()), columns=['word', 'polarity']).sort_values('polarity')
pd.DataFrame(list(top500polarity_meta.items()), columns=['word', 'polarity']).sort_values('polarity')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15683032307300943, 'fraction_numerical': 0.025820401933350293, 'mean_word_length': 6.307620817843866, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 46, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '85856', 'license': 'Apache-2.0', 'n_tokens_mistral': 2711, 'n_tokens_neox': 2550, 'n_words': 673, 'path': 'BERT_analysis.ipynb', 'repo_name': 'abcd1927/bert-sentiment-analysis', 'size': '7862'}