<jupyter_start><jupyter_text>Exploring election tweets from Tweet the People project I want to build a Naive Bayes classifier that predicts whether a given tweet is about a Republican or Democrat presidential candidate.(For the purposes of this model, I'll assume that a tweet is "about a Republican" if it originally contained the words `donaldtrump`, `<NAME>`, `mikepence`, or `<NAME>`, and the same for the Democrat candidates. But I'll strip these actual terms out before building the model, otherwise the model wouldn't be building a prediction based on anything other than the presence of these terms.)Before I build this model, I'd like to know:- Is there a class imbalance that I need to take into account in my model?- Some basic stats about the tweets:    - Average no. words (and whether it differs by candidate)- Bonus: Check distribution of sentiment in each ticket- Bonus: I'll be stripping out hashtags, but I'm curious to know about the trends here.  Step 1: reading in data, basic data cleaning<jupyter_code>import re
import pandas as pd
import numpy as np
import random
from collections import Counter

import seaborn as sns

from scipy import stats

from spacy.lang.en import English
from spacy.tokenizer import Tokenizer

from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

# initialise NLP tools
SENT_ANALYSIS = SentimentIntensityAnalyzer()
nlp = English()

import warnings
warnings.filterwarnings('ignore')
dict_replace = {"-": " ", "\.": "", "\?": "\'", "\s+": " ", "&amp;" : "and"}
tweets = pd.read_csv('/Users/laraehrenhofer/Documents/Coding_Projects/git_repos/tweet-the-people-legacy/data/tweet_pg.csv')
# tweets = tweets[['text', 'politician', 'date', 'sentiment', 'ticket']]<jupyter_output><empty_output><jupyter_text>Some tweets were wrongly inserted with the date as the politician -- I'm going to delete these data points from the sample. (This removes about 30 tweets.)<jupyter_code>politicians = ['kamalaharris', 'donaldtrump', 'joebiden', 'mikepence']
def is_politician(politician):
    '''
    Returns boolean indicating whether input is a member of politicians list
    '''
    if politician in politicians:
        return politician
    else:
        return np.NaN
tweets['politician'] = tweets['politician'].apply(lambda x: is_politician(x))
tweets = tweets.dropna()<jupyter_output><empty_output><jupyter_text>Next: stripping out handles, hashtags, and other undesirables.<jupyter_code>def get_handles_hashtags(text):
    '''
    Returns separate lists of hashtags and user handles in the text
    '''
    handles = re.findall('\B\@\w+', text)
    hashtags = re.findall('\B\#\w+', text)
    return handles, hashtags
def strip_out_weird_symbols(text):
    '''
    Replaces special symbols that get messed up eg. replaces &amp; with and (as they are functionally identical)
    See dict_replace for replacements
    '''
    for key, value in dict_replace.items():
        sub = re.sub(key, value, text)
        text = sub
    text = re.sub(r'\\', '', text)
    text = re.sub('\s+', ' ', text)
    return text
def strip_out_handles_hashtags(text):
    '''
    Removes handles and hashtags
    '''
    text = re.sub(r'@\w+', '', text)
    text = re.sub(r'#\w+', '', text)
    return text
def strip_out_links(text):
    '''
    Removes links
    '''
    text = re.sub(r'https\S+', '', text)
    return text
def clean_text(text):
    '''
    - Removes handles & hashtags from text
    - Handles odd symbols
    - Tokenisation
    '''
    cleaned = strip_out_links(text)
    cleaned = strip_out_weird_symbols(cleaned)
    cleaned = strip_out_handles_hashtags(cleaned)
    doc = nlp(cleaned)
    cleaned = [token.orth_.lower() for token in doc if not token.is_punct]
    cleaned = [item for item in cleaned if not re.search('\s+', item)]
    text_len = len(cleaned)
    cleaned = ' '.join(cleaned)
    return cleaned, text_len
# re-do sentiment analysis

def analyse_sentiment(text):
    '''
    Simple lexical approach to sentiment analysis using VADER
    Returns compound sentiment score
    '''
    sentiment = SENT_ANALYSIS.polarity_scores(text)
    return sentiment['compound']
tweets[['handles', 'hashtags']] = tweets.apply(lambda row: pd.Series(get_handles_hashtags(row['text'])), axis=1)
tweets[['clean_text', 'text_len']] = tweets.apply(lambda row: pd.Series(clean_text(row['text'])), axis=1)
tweets['sentiment_2'] = tweets.apply(lambda row: pd.Series(analyse_sentiment(row['clean_text'])), axis=1)
# save this version of tweets data for later

tweets.to_csv('./tweet_minimal_newsentiment.csv', index=False)<jupyter_output><empty_output><jupyter_text>Step 2: check class imbalance Quick count of how many tweets per candidate and per ticket.<jupyter_code># count by candidate

cand = tweets.groupby(['politician']).count()
cand = cand['text']
cand
# comparison

cand.plot(kind='pie', subplots=True, figsize=(4, 4))
ticket = tweets.groupby(['ticket']).count()
ticket = ticket['text']
ticket
ticket.plot(kind='pie', subplots=True, figsize=(4, 4))
ticket[0]/sum(ticket)<jupyter_output><empty_output><jupyter_text>There's more tweets about Democrat than about Republican candidates, but that's quite probably due to the low rates of tweets about Mike Pence. Still, the discrepancy isn't monstrous (59% vs 41%), so I won't do any resampling to correct this.  Step 3: check different distributions in word length & sentiment<jupyter_code>dem = tweets[tweets['ticket']=='Democrat']
dem_len = dem['text_len']
rep = tweets[tweets['ticket']=='Republican']
rep_len = rep['text_len']
sns.distplot(dem_len, color='skyblue', kde=False)
sns.distplot(rep_len, color='red', kde=False)
# check for statistical difference in distributions

mean_dem = dem_len.mean()
mean_rep = rep_len.mean()

print(mean_dem, mean_rep)
stats.ttest_ind(dem_len, rep_len)<jupyter_output><empty_output><jupyter_text>I'm not sure what to make of the difference in tweet length. Bearing in mind that there are duplicates in the tweets (due to retweets) and the sheer number of Democrat-related tweets is higher, it's possible that if among the Dem sample there are some very popular and often-shared longer tweets, this would push up the mean and alter the distribution to the point of resulting in a low p-value. I'm saying this because of the 3 peaks in the Dem distribution, where it seems like three specific tweet lengths are high-frequency, suggesting many shares rather than people independently posting tweets around a certain length (where we would expect more of a normal distribution).  Step 4: check distributions of sentiment<jupyter_code>dem_sent = dem['sentiment']
rep_sent = rep['sentiment']
sns.distplot(dem_sent, color='skyblue', kde=False)
sns.distplot(rep_sent, color='red', kde=False)
stats.ttest_ind(dem_sent, rep_sent)<jupyter_output><empty_output><jupyter_text>No statistically significant differences between the two samples -- good news for further modelling.<jupyter_code>tweets['sent_diff'] = tweets['sentiment'] - tweets['sentiment_2']<jupyter_output><empty_output><jupyter_text>Step 5: Checking top hashtags for each sample<jupyter_code># I'm returning to this later and just reading in the minimal tweet csv:

tweets = pd.read_csv('./tweet_minimal.csv')
def flatten_list(tweets, ticket):
    '''
    Flattens df containing lists of hashtags into a single list
    '''
    flat = list(tweets[tweets['ticket'] == ticket]['hashtags'].values.flatten())
    flat = [re.findall(r'#\w+', item) for item in flat if item != '[]']
    flat = [item for sublist in flat for item in sublist]
    return flat
rep_hashtags = flatten_list(tweets, 'Republican')
dem_hashtags = flatten_list(tweets, 'Democrat')
len(rep_hashtags)
def count_hashtags(list_of_hashtags):
    '''
    turns list of hashtags into a df with sorted count of top 25 hashtags
    '''
    summary = pd.DataFrame.from_dict(Counter(list_of_hashtags), orient='index').reset_index()
    summary.columns = ['hashtag', 'count']
    summary = summary.sort_values(by='count', ascending=False).head(25)
    return summary
rep_sum = count_hashtags(rep_hashtags)
dem_sum = count_hashtags(dem_hashtags)
rep_sum
dem_sum<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11120668468911281, 'fraction_numerical': 0.003194888178913738, 'mean_word_length': 4.997789240972734, 'pattern_counts': {'":': 4, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '172874', 'license': 'MIT', 'n_tokens_mistral': 2627, 'n_tokens_neox': 2498, 'n_words': 816, 'path': 'election-tweet-classification/election_tweet_eda.ipynb', 'repo_name': 'lara-ehr/spaCy-explorers', 'size': '8148'}