<jupyter_start><jupyter_text>Introduction This notebook is an attempt to implement sentiment analysis using different methodsStarting with methods like Logistic Regression, Latent Dirilecht Analysis, Naive Baiyes etcWe will be using twitter datasets from NLTK  Imports<jupyter_code>import nltk
from nltk.stem import PorterStemmer
from nltk.tokenize import TweetTokenizer
from nltk.corpus import twitter_samples, stopwords

from collections import defaultdict

import re
import os  
import string
import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>Datasets and Few more Initializations<jupyter_code># Download the twitter dataset. This will be the dataset that will be used
nltk.download('twitter_samples')

# Download the english stopwords that will be removed from the tweets during preprocessing
nltk.download('stopwords')<jupyter_output>[nltk_data] Downloading package twitter_samples to /root/nltk_data...
[nltk_data]   Unzipping corpora/twitter_samples.zip.
[nltk_data] Downloading package stopwords to /root/nltk_data...
[nltk_data]   Unzipping corpora/stopwords.zip.<jupyter_text><jupyter_code># Get the positive and negative tweets from twitter_samples. These are of type list
all_positive_tweets = twitter_samples.strings('positive_tweets.json')
all_negative_tweets = twitter_samples.strings('negative_tweets.json')

# logging the type and lengths
type(all_positive_tweets), len(all_positive_tweets), len(all_negative_tweets)<jupyter_output><empty_output><jupyter_text>Let's take the last 1000 tweets as the test values for both positive and negative tweets.   That will leave the first 4000 tweets as the training data<jupyter_code># Separating positive tweets
test_pos = all_positive_tweets[4000:]
train_pos = all_positive_tweets[:4000]

# separating negative tweets
test_neg = all_negative_tweets[4000:]
train_neg = all_negative_tweets[:4000]

# Concatenating the positive and negative tweets to form the final test and train datasets
train_x = train_pos + train_neg
test_x = test_pos + test_neg

# Assigning labels 
# 1 = positive tweet and 
# 0 = negative tweet
train_y = [1]*len(train_pos) + [0]*len(train_neg)
test_y = [1]*len(test_pos) + [0]*len(test_neg)


# Showing the final lengths of train and test datasets
len(train_x), len(test_x)
# Showing few positive and negative tweets
print(train_x[:5])
print(train_x[-5:])<jupyter_output>['#FollowFriday @France_Inte @PKuchly57 @Milipol_Paris for being top engaged members in my community this week :)', '@Lamb2ja Hey James! How odd :/ Please call our Contact Centre on 02392441234 and we will be able to assist you :) Many thanks!', '@DespiteOfficial we had a listen last night :) As You Bleed is an amazing track. When are you in Scotland?!', '@97sides CONGRATS :)', 'yeaaaah yippppy!!!  my accnt verified rqst has succeed got a blue tick mark on my fb profile :) in 15 days']
['Amelia didnt stalk my twitter :(', 'oh, i missed the broadcast. : (', "i really can't stream on melon i feel useless :-(", 'I need to stop looking at old soccer pictures :(', 'Got an interview for the job that I want but they rang me Tuesday for the interview on Thursday but in on holiday :(']<jupyter_text>Text Preparing All tweets should be removed of stopwords, urls, hashtags, tweet_ats, stock market tickers converted into tokensSequence of steps for preparation:1.   remove urls in tweets2.   Remove  and @ symbols3.   Remove stopwords4.   Tokenize the tweet5.   Stem the words in the tweet**Note**: Stemming gets the words to the root form of the word (need not end up as a proper word in the language like lemmatization would result in)<jupyter_code>def process_tweet(tweet):
  # Getting rid of urls in the tweet
  # Below regular expression removes http https links including any with port numbers
  tweet = re.sub(r'http[s][:\/\w]*', '', tweet)

  # Getting rid of @, we will still retain who the tweet is being addressed to by just removing the symbol
  tweet = re.sub(r'@', '', tweet)

  # Getting rid of # symbol, retaining the hashtag name
  tweet = re.sub(r'#', '', tweet)

  # Get the list of stopwords from nltk 
  stopwords_list = set(stopwords.words('english'))
  #Let's print the stopwords as FYI
  #print("stopwords =", stopwords_list)

  # Get a Tokenizer instance
  tokenizer = TweetTokenizer(preserve_case=False, strip_handles=False, reduce_len=False)
  tokens = tokenizer.tokenize(tweet)

  # The tokens generated will have stopwords and words that need to be stemmed
  # Doing both in the next step

  # Initialize a stemmer
  stemmer = PorterStemmer() 
  
  # stem the token if it is not a stopword and it is not a punctuation
  clean_tweet=[stemmer.stem(token) for token in tokens if token not in stopwords_list and token not in string.punctuation ]
  
  return clean_tweet<jupyter_output><empty_output><jupyter_text>Method 1: Logistic Regression  Preparation for Feature Extraction:For every word, find the number of times it has been used in a positive tweet and number of times it has been used in a negative tweet<jupyter_code>def build_frequencies(tweets, labels):
  '''
  Function to create a list of tuples, each tuple of the form (token, sentiment)

  tweets: list of tweets
  labels: list of labels if it is positive of negative
  '''
  # Using a defaultdict so that in case a key is unavailable, it will return 0 
  # and it will also help in incrementing the frequencies
  freq_list = defaultdict()
  
  # Go through each tweet and label
  for tweet, label in zip(tweets, labels):
    # get tokens from the tweet
    tokens = process_tweet(tweet)
    
    # go through each token and increment the freq_list
    for token in tokens:
      freq_list[(token, label)] = freq_list.get((token, label),0) + 1

  return freq_list<jupyter_output><empty_output><jupyter_text>Extracting Features for LRWe will have two features for LR, for a given tweet, we will get the frequencies for each token used in a positive context and negative context```tweet1 Features = sum of positive_freq of words in tweet1, sum of negative_freq of words in tweet1```<jupyter_code>def extract_features(tweets, frequencies):
  '''
  
  tweets = list of tweets
  frequencies = resulting dictionary from the function build_frequencies
  '''
  feature_list = []
  for tweet in tweets:
    # Get the tokens from the tweet
    tokens = process_tweet(tweet)

    # define the features array. 
    # index 0 = positive sentiment
    # index 1 = negative sentiment
    features = np.zeros((2,))

    #go through the tokens and get the sum of positive and negative freqs
    for token in tokens:
      # add the token as a positive sentiment to the positive sentiment feature
      features[0] += frequencies.get((token, 1),0)

      # add the token as a negative sentiment to the negative sentiment feature
      features[1] += frequencies.get((token, 0),0)
    feature_list.append(features)

  return feature_list

print("stopwords =", set(stopwords.words('english')))
# get the frequencies from the training dataset
frequencies = build_frequencies(tweets = train_x, labels = train_y)

# get the features for training
training_features = extract_features(tweets = train_x, frequencies = frequencies)

# get the features for testing
testing_features = extract_features(tweets = test_x, frequencies = frequencies)

print("train data =", train_x[0])

print("processed tweet =", process_tweet(train_x[0]))

print("training_features:", training_features[0])<jupyter_output><empty_output><jupyter_text>Building LR ModelWe will implement LR model using Tensorflow as a single unit Dense layer with sigmoid activation function<jupyter_code>import tensorflow as tf
from tensorflow.keras import Model, Input
from tensorflow.keras.layers import Dense
def lr_model(batch_size=10):
  '''
  function to build logistic regression model using Tensorflow
  '''
  input = Input(shape = (2,), batch_size = batch_size, name='input')
  x = Dense(1, activation='sigmoid', name='dense')(input)

  model = Model(inputs = input, outputs = x, name='lr_model')
  model.summary()
  return model
lr_model = lr_model()
lr_model.compile(optimizer='adam'
                  , loss='binary_crossentropy'
                  , metrics = ['accuracy']
                 )


len(training_features)<jupyter_output><empty_output><jupyter_text>Training<jupyter_code>metrics = lr_model.fit(x=np.array(training_features)
             , y = np.array(train_y)
             , epochs = 30
             , shuffle=True
             , validation_split=0.1

             )

import matplotlib.pyplot as plt
plt.figure(figsize=(12,6))


plt.subplot(1,2,1)
plt.plot(metrics.history['loss'], label='train')
plt.plot(metrics.history['val_loss'], label='test')
plt.title('Loss')
plt.legend()

plt.subplot(1,2,2)
plt.plot(metrics.history['accuracy'], label='train')
plt.plot(metrics.history['val_accuracy'], label='test')
plt.title('Accuracy')
plt.legend()<jupyter_output><empty_output><jupyter_text>Predictions and Confusion Matrix<jupyter_code>y_pred = lr_model.predict(np.array(testing_features))

confusion_matrix = tf.math.confusion_matrix(test_y, y_pred, num_classes=2)

print(confusion_matrix)
y_pred[:5]<jupyter_output><empty_output><jupyter_text>Method 2: Linear Discrimant Analysis (LDA)In LDA we find the centroids of each class of tweets, use the gap between the centroids as the differentiating factor between the classes<jupyter_code># we need to represent the tweets as vectors, we do this using sklearn's TfidfVectorizer
from sklearn.feature_extraction.text import TfidfVectorizer

tweetVectorizer = TfidfVectorizer(tokenizer = process_tweet
                               , stop_words=list(stopwords.words('english'))
                               )

# Obtaining the vector representations of the tweets
tweetTfidf = tweetVectorizer.fit_transform(train_x)
type(tweetTfidf), tweetTfidf.shape
tweetTfidf = tweetTfidf.toarray()<jupyter_output><empty_output><jupyter_text>Calculating the centroids<jupyter_code># create a boolean mask to use to calculate the mean
positive_mask = [y == 1 for y in train_y]
print("length of mask = ", len(positive_mask))


# calculating the positive centroid
positive_centroid = tweetTfidf[positive_mask].mean(axis=0)
print("positive centroid = ", positive_centroid.shape, "\n", positive_centroid)

# calculating the negative centroid using a negation of the positive mask
negative_centroid = tweetTfidf[[~val for val in positive_mask]].mean(axis=0)
print("negative centroid =", negative_centroid.shape, "\n", negative_centroid)

# Finding the vector connecting the two centroids
difference = positive_centroid - negative_centroid<jupyter_output>length of mask =  8000
positive centroid =  (16652,) 
 [7.10403873e-05 8.27011206e-05 3.69501277e-04 ... 0.00000000e+00
 0.00000000e+00 9.05286295e-05]
negative centroid = (16652,) 
 [0. 0. 0. ... 0. 0. 0.]<jupyter_text>TestingPredictions are done by 1. finding the projection of the tweets onto the  vector connecting the positive and negative centroids, 2. scale the result between the range [0, 1], and 3. use a threshold like 0.5 or 0.7 to get a binary output prediction<jupyter_code># First we find the tfidfVector representation of the test data
# since the output is a sparse matrix, it is converted to dense
testTfidf = tweetVectorizer.transform(test_x).toarray()

# Calculating the projection of the test data onto the difference vector
testVectorProjection = np.dot(testTfidf, difference)

print("projection shape and few samples = ", testVectorProjection.shape, "\n", testVectorProjection[:5])

from sklearn.preprocessing import MinMaxScaler
# instantiate the scaler
scaler = MinMaxScaler(feature_range=(0,1))

# Scale the prediction between 0 and 1
scaled_y_pred = scaler.fit_transform(np.reshape(testVectorProjection, (-1, 1)))
print("sample scaled projecion =", scaled_y_pred[:5])

# Using threshold as 0.5 to
y_pred = [int(y > 0.7) for y in scaled_y_pred ]<jupyter_output><empty_output><jupyter_text>Confusion Matrix<jupyter_code>confusion_matrix = tf.math.confusion_matrix(test_y, y_pred, num_classes=2)
print(confusion_matrix)<jupyter_output>tf.Tensor(
[[992   8]
 [157 843]], shape=(2, 2), dtype=int32)<jupyter_text>Method 3: Naiive BayesNaiive Bayes assumes that every word is independent of each other$$sentiment(tweet) = logprior * log(\prod_i^N \frac{p(positive|word)}{p(negative|word)})$$$$sentiment(tweet) = logprior * \sum_i^N( log(p(positive|word)) - log(p(negative|word)) ) $$$$sentiment(tweet) = logprior * \sum_i^N(likelihood(word)) $$where N = Number of words in a tweetFirst we calculate the logprior, i.e., log of the ratio probability of positive tweets to negative tweets. This will indicate the bias in the dataset, if any.$$P(D_{pos}) = \frac{D_{pos}}{D}\tag{1}$$$$P(D_{neg}) = \frac{D_{neg}}{D}\ $$where * $D_{pos}$ is the number of positive tweets* $D_{neg}$ is the number of negative tweets* $D$ is the total number of tweets$$ logprior = log(\frac{P(D_{pos})}{P(D_{neg})})$$$$ logprior = log(\frac{D_{pos}}{D_{neg}})$$$$ logprior = log(D_{pos})-log(D_{neg})$$Next we find the likelihood of a word in the vocabulary of being positive or negative setiment. 1. probability of a word being a particular sentiment (using Laplacian smoothing to avoid log(0))2. likelihood of the word is the log of ratio of probabilities (positive to negative)$$ p(positive|word) = \frac{freq(word\_as\_positive) + 1}{N_{pos} + V}$$$$ p(negative|word) = \frac{freq(word\_as\_negative) + 1}{N_{neg} + V}$$$$ log\_likelihood(word) = log(p(positive|word)) - log(p(negative|word)) $$where * V = vocabulary size* $N_{pos}$ = Total Number of positive words* $N_{neg}$ = Total number of negative words**In conclusion**, likelihood is * \> (greater than) 1 if the word tends to be more positively used in the vocabulary* = 1 if the word is neutral* \< (less than) 1 if the word tends to be more negatively used in the vocabulary**And **, log likelihood is* \>(greater than) 0 if the word is positively used in the vocabulary* = 0 if the word is neutral* \<(less than) 0 if the word tends to be more negatively used in the vocabularyThe above steps are the training steps for Naiive Bayes model. The parameters of the model being1. logprior - float that signifies the bias within the dataset towards a sentiment2. log likelihood: dictionary of word and its likelihood of a sentiment  Calculating Model Parameters<jupyter_code>def calculate_naiive_bayes_params(tweets, labels):
  '''
  Function to calculate the log prior and log likelihood for a Naiive Bayes model
  tweets: list of tweets, i.e. training dataset
  labels: list of 1-D labels that signify the sentiment of the tweets. This should be the same length as the tweets.
          Assumption is positive sentiment = 1 and negative sentiment = 0

  Returns: a tuple (float, dictionary)
  log_prior: float, log prior of the training dataset provided
  log_likelihood: dictionary, key = token and value = log likelihood of the token
  '''
  # Calculating logprior, total number of positive tweets / total number of negative tweets
  # since positive tweet is represented as 1 and negative tweet is 0, we can put logprior as 
  log_prior = np.log(sum(labels)) - np.log(len(labels) - sum(labels))
  print("logprior = ", log_prior)

  # Next we are going to calculate the log likelihood of the words in the vocab

  # First build the frequencies dictionary key= (token, sentiment) value = frequency
  frequencies = build_frequencies(tweets, labels)

  # We need to know the vocabulary length V, total number of positive words and negative words

  # initialize a set for vocabulary
  vocab = set()
  # initialize the counts of positive and negative tokens
  n_pos = n_neg = 0
  # loop through the frequencies dictionary
  for token, sentiment in frequencies.keys():
    #add the token into the vocabulary set
    # since it is a set, it avoids duplication
    vocab.add(token)

    # increment n_pos and n_neg based on sentiment of the token
    if sentiment == 1: # positive sentiment
      n_pos += 1
    else: # negative sentiment
      n_neg += 1
  print("length of vocab =", len(vocab))
  print("total number of positive tokens =", n_pos)
  print("total number of negative tokens =", n_neg)

  # Going through the vocabulary to create log likehood dictionary
  log_likelihood = defaultdict()

  for token in vocab:
    # Find the likelihood that the token is positive. Include Laplacian smoothing
    likelihood_positive = (frequencies.get((token, 1),0) + 1)/(n_pos + len(vocab)) 

    # Find the likelihood that the token is negative. Include Laplacian smoothing
    likelihood_negative = (frequencies.get((token, 0),0) + 1)/(n_neg + len(vocab))

    # Find the log likelihood of the token
    log_likelihood[token] = np.log(likelihood_positive) - np.log(likelihood_negative)
  
  return log_prior, log_likelihood<jupyter_output><empty_output><jupyter_text>Predicting SentimentsGiven a test tweet, below steps will be followed to make a prediction using Naiive Bayes method1. Get the tokens from the tweet by calling process_tweet2. Find the log likelihood of each of the tokens using the dictionary from calculate_naiive_bayes_params function. If a token is not in the log_likelihood dictionary, assume that it is neutral3. Calculate the log likelihood of the test tweet using the formula: $$logprior + \sum  log likelihood(tokens\ in\ the\ test\ tweet)$$4. If log likelihood of the test tweet is greater than zero, then the sentiment of the tweet is positive, else it is negative<jupyter_code># Get the Naiive Bayes parameters
log_prior, log_likelihood = calculate_naiive_bayes_params(train_x, train_y)

# Initialize an empty list for the predictions
y_pred = []

# Go through each tweet in the testing dataset
for tweet in test_x:
  # Step 1: get the tokens from the tweet
  tokens = process_tweet(tweet)

  # Step 2 and 3: find sum of log likelihood of the tokens
  tweet_log_likelihood = np.sum([log_likelihood.get(token,0) for token in tokens])

  # Step 3: Calculate the sentiment value
  sentiment = log_prior + tweet_log_likelihood

  # Step 4: make final prediction
  if (sentiment >0):
    # Positive sentiment
    y_pred.append(1)
  else:
    # negative sentiment
    y_pred.append(0)<jupyter_output>logprior =  0.0
length of vocab = 16674
total number of positive tokens = 10587
total number of negative tokens = 8505<jupyter_text>Confusion Matrix<jupyter_code>confusion_matrix = tf.math.confusion_matrix(test_y, y_pred, num_classes=2)
print(confusion_matrix)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08787403279043342, 'fraction_numerical': 0.015096585682592933, 'mean_word_length': 4.84134007585335, 'pattern_counts': {'":': 0, '<': 66, '<?xml version=': 0, '>': 68, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '257604', 'license': 'CC0-1.0', 'n_tokens_mistral': 5941, 'n_tokens_neox': 5483, 'n_words': 2034, 'path': 'NLP_SentimentAnalysisMethods.ipynb', 'repo_name': 'mkkadambi/machine-learning', 'size': '18481'}