<jupyter_start><jupyter_text>Twitter Sentiment Analysis Installing some packages and downloading data from Kaggle<jupyter_code>!pip3 install ktrain
! pip install kaggle==1.5.6
! kaggle datasets download -d kazanova/sentiment140<jupyter_output>Requirement already satisfied: ktrain in c:\users\andre\anaconda3\lib\site-packages (0.28.3)
Requirement already satisfied: jieba in c:\users\andre\anaconda3\lib\site-packages (from ktrain) (0.42.1)
Requirement already satisfied: cchardet in c:\users\andre\anaconda3\lib\site-packages (from ktrain) (2.1.7)
Requirement already satisfied: matplotlib>=3.0.0 in c:\users\andre\anaconda3\lib\site-packages (from ktrain) (3.3.4)
Requirement already satisfied: sentencepiece in c:\users\andre\anaconda3\lib\site-packages (from ktrain) (0.1.96)
Requirement already satisfied: chardet in c:\users\andre\anaconda3\lib\site-packages (from ktrain) (4.0.0)
Requirement already satisfied: fastprogress>=0.1.21 in c:\users\andre\anaconda3\lib\site-packages (from ktrain) (1.0.0)
Requirement already satisfied: syntok in c:\users\andre\anaconda3\lib\site-packages (from ktrain) (1.3.3)
Requirement already satisfied: packaging in c:\users\andre\anaconda3\lib\site-packages (from ktrain) (20.9)
Requirement already s[...]<jupyter_text>Importing libraries<jupyter_code># File management libraries
import os
import zipfile

# Data manipulation libraries
import numpy as np
import pandas as pd

# Data visualization libraries
import matplotlib.pyplot as plt
import seaborn as sns
import plotly.express as px
import kaleido
from wordcloud import WordCloud, STOPWORDS

# Natural language processessing and text libraries
import nltk
import string
import re
from gensim.utils import simple_preprocess

# Scikit Learn libraries for splitting data and model evaluation
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Keras libraries for Neural Network development
from tensorflow.keras.preprocessing import text, sequence
from tensorflow.keras.utils import to_categorical
from tensorflow.keras import models, layers, optimizers, callbacks

# Library used to get the BERT Transformer model
import ktrain<jupyter_output><empty_output><jupyter_text>Reading the dataset Extracting the file and reading it on a Pandas DataFrame.<jupyter_code>with zipfile.ZipFile("sentiment140.zip","r") as zip_ref:
    zip_ref.extractall()

df = pd.read_csv('training.1600000.processed.noemoticon.csv', encoding = 'Latin-1', names=('target','id','date','flag','user','text'))<jupyter_output><empty_output><jupyter_text>Since we have a limited space on the GitHub repository, we have to delete the zip and csv file to reduce space.<jupyter_code>os.remove('training.1600000.processed.noemoticon.csv')
os.remove('sentiment140.zip')<jupyter_output><empty_output><jupyter_text>The positive sentiment is labeled as the number 4 and usually is the number 1. To make it clear for now on, we create a column called "sentiment" with 1 as the positive label and 0 as the negative label.<jupyter_code>df['sentiment'] = (df['target'] == 4)*1
df.head()<jupyter_output><empty_output><jupyter_text>Let's count the number of positives and negatives examples on the dataset. As we can se we have the same number for each sentiment.<jupyter_code>df.sentiment.value_counts()<jupyter_output><empty_output><jupyter_text>Data Prep When we are dealing with text-type data, we need to treat it in a way that we and the possible machine learning models that we make can use it in the best way. Two of the ways to handle text data are the removal of punctuation and stop words, which are quite common words that in general would not help us to differentiate a positive feeling from a negative one because they are present in both cases. Also, on Twitter texts are very common the presence of @mentions and hashtags that requires some treatment as well. a) Removing tweet patterns Removing @mentions, the "" symbol and URLs starting with "https" or "http".<jupyter_code>def twitter_text_cleaning(text):
  '''
  INPUT
  text - a text message that might have @mentions, "#" symbol or URLs
    
  OUTPUT
  text - the text without @mentions, "#" and URLs
  '''

  text = re.sub(r'@[A-Za-z0-9]+', '', text)
  text = re.sub(r'@[A-Za-zA-Z0-9]+', '', text)
  text = re.sub(r'@[A-Za-z0-9_]+','', text)
  text = re.sub(r'@[A-Za-z]+', '', text)
  text = re.sub(r'@[-)]+', '', text)
  text = re.sub(r'#', '', text )
  text = re.sub(r"http\S+", "", text)
  text = re.sub(r'https?\/\/\S+', '', text)
  text = re.sub(r'http?\/\/\S+', '', text)
  text = re.sub(r'https?\/\/.*[\r\n]*', '', text)
  text = re.sub(r'^https?\/\/.*[\r\n]*', '', text)
  text = re.sub(r'^https?:\/\/.*[\r\n]*', '', text)

  return text

df['cleaned_text'] = df['text'].apply(twitter_text_cleaning)<jupyter_output><empty_output><jupyter_text>b) Removing punctuations<jupyter_code>def remove_punc(message):
  '''
  INPUT
  message - a text message that might have puntuations in it
    
  OUTPUT
  message without punctuations
  '''

  return ''.join([char for char in message if char not in string.punctuation])

df['cleaned_text_wo_punctuation'] = df['cleaned_text'].apply(remove_punc)<jupyter_output><empty_output><jupyter_text>c) Removing stopwords<jupyter_code>nltk.download('stopwords')
stop_words = nltk.corpus.stopwords.words('english')
stop_words.extend(['from', 'subject', 're', 'edu', 'use', 'will', 'aap', 'co', 'day', 'user', 'today', 'week', 'year', 'https', 'amp', 'lol', 'one'])
print('Some examples of Stop Words: \n', stop_words[:10])
def stop_words_preprocess(text):
  '''
  INPUT
  text - a text message that might have stop words
    
  OUTPUT
  list of words without stop words examples
  '''

  return [word for word in simple_preprocess(text) if word not in stop_words and len(word) >= 3]
df['cleaned_text_wo_punctuation_and_stopwords'] = df['cleaned_text_wo_punctuation'].apply(stop_words_preprocess)
df['preprocessed_text'] = df['cleaned_text_wo_punctuation_and_stopwords'].apply(lambda x: " ".join(x))<jupyter_output><empty_output><jupyter_text>The final DataFrame with examples of every text preprocessing will be like<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>Analysis  1. What are the most common words in POSITIVE sentiment tweets? And what words are for the NEGATIVES?  A good way to visualize the most common words in sentences is by using Word Cloud visualization. In a Word Cloud the more frequent a word is the bigger the word will be on the picture.<jupyter_code>plt.figure(figsize= (10,10))
wc = WordCloud(
    max_words=500, 
    width=1600,
    height=800
).generate(' '.join(df[df['sentiment'] == 1]['preprocessed_text']))
wc.to_file('images/wordcloud_positive_words.png')
plt.imshow(wc)
plt.figure(figsize= (10,10))
wc = WordCloud(
    max_words=500, 
    width=1600,
    height=800
).generate(' '.join(df[df['sentiment'] == 0]['preprocessed_text']))
wc.to_file('images/wordcloud_negative_words.png')
plt.imshow(wc)<jupyter_output><empty_output><jupyter_text>2. Do users who tweet the most tend to have positive or negative sentiments? Who are these users? First, let's count the number of tweets from a user.<jupyter_code>sentiment_per_user_df = df.groupby('user').agg({'sentiment':'mean','text':'count'})\
.reset_index().rename(columns={"sentiment": "sentiment_average", 'text' : 'tweets_number'})

sentiment_per_user_df.head()<jupyter_output><empty_output><jupyter_text>Then, we define some ranges according to the number of tweets made by an user.<jupyter_code>def tweets_range(tweets_number):
  '''
  INPUT
  tweets_number - number of tweets

  OUTPUT
  tweet_range - interval of number of tweets
  '''

  if tweets_number == 1:
    return '1. Only 1 tweet'
  
  if tweets_number <= 5:
    return '2. Between 2 and 5 tweets'

  if tweets_number <= 10:
    return '3. Between 6 and 10 tweets'

  if tweets_number <= 20:
    return '4. Between 11 and 20 tweets'

  if tweets_number <= 50:
    return '5. Between 21 and 50 tweets'

  if tweets_number <= 100:
    return '6. Between 51 and 100 tweets'

  return '7. 101 or more tweets'

sentiment_per_user_df['tweets_range'] = sentiment_per_user_df['tweets_number'].apply(tweets_range)<jupyter_output><empty_output><jupyter_text>Now, let's count the number of users per tweets range.<jupyter_code>tweets_range_df = sentiment_per_user_df.groupby('tweets_range').agg({'tweets_number': 'count', 'sentiment_average': 'mean'}).reset_index()
tweets_range_df = tweets_range_df.rename(columns={"tweets_number": "users"})
tweets_range_df['%users'] = round(100*tweets_range_df['users']/tweets_range_df['users'].sum(),2)
tweets_range_df['sentiment_average'] = round(tweets_range_df['sentiment_average'],2)
tweets_range_df[['tweets_range', 'users','%users', 'sentiment_average']]<jupyter_output><empty_output><jupyter_text>Finally, we can find the high engaging users that has on average more positives than negatives tweets and the users that has more negatives than positives tweets.<jupyter_code>sentiment_per_user_df[sentiment_per_user_df['tweets_range'] == '7. 101 or more tweets'] \
.sort_values(["sentiment_average", "tweets_number"], ascending = (False, False)).head(5)
sentiment_per_user_df[sentiment_per_user_df['tweets_range'] == '7. 101 or more tweets'] \
.sort_values(["sentiment_average", "tweets_number"], ascending = (True, False)).head(5)<jupyter_output><empty_output><jupyter_text>3. Does the NUMBER OF WORDS in a tweet help determine whether it has positive or negative sentiment?<jupyter_code>df['num_words'] = df['cleaned_text_wo_punctuation_and_stopwords'].apply(lambda x : len(x))
def num_words_range(num_words):
    '''
    INPUT
    num_words - number of words in a tweet

    OUTPUT
    num_words_range - interval of number of words in a tweet
    '''

    if num_words <= 5 :
        return '1. Between 1 and 5 words'

    if num_words <= 10 :
        return '2. Between 6 and 10 words'

    return '3. 11 words or more'

df['num_words_range'] = df['num_words'].apply(num_words_range)
def create_relative_dataframe(df, range_col):
    '''
    INPUT
    df - dataframe with a range column and a categorical variabel
    range_col - column name with the ranges

    OUTPUT
    melted_df - relative dataframe with the percentage of the categorical variable per range
    '''
    
    grouped_df = df.groupby([range_col, 'sentiment']).text.count().reset_index()
    grouped_df = grouped_df.rename(columns={"text": "num_tweets"})
    grouped_df['sentiment'] = grouped_df['sentiment'].apply(lambda x: 'Positive' if x == 1 else 'Negative')

    pivot_df = grouped_df.pivot(index='sentiment', columns=range_col, values='num_tweets')
    relative_pivot_df = round(100*pivot_df.div(pivot_df.sum(axis=0), axis=1),1).reset_index()

    melted_df = relative_pivot_df.melt(id_vars=['sentiment'], value_vars=relative_pivot_df.columns[1:])
    melted_df = melted_df.rename(columns={"value": "%tweets"})

    return melted_df

def plot_stacked_bar_chart(df, range_col, chart_title = '', export_to = ''):  
    '''
    INPUT
    df - dataframe with a range column and a categorical variabel
    range_col - column name with the ranges
    chart_title - chart title (empty by default)
    export_to - path where the chart image must be saved as PNG

    OUTPUT
    A plot of the 100% stacked bar chart
    '''

    relative_df = create_relative_dataframe(df, range_col)
    colors = px.colors.qualitative.G10[:2]

    fig = px.bar(
        relative_df, 
        x=range_col,
        y='%tweets',
        color='sentiment',
        text='%tweets',
        width=1000,
        height=400,
        title=chart_title,
        color_discrete_sequence=[colors[1], colors[0]],
        category_orders={"sentiment": ["1", "0"]}
    )
    fig.update_traces(texttemplate='%{text}%', textposition='inside')
    fig.update_xaxes(showgrid=False,  title_text = "")
    fig.update_yaxes(showgrid=False)

    if export_to != '':
        fig.write_image(export_to, engine='kaleido')

    fig.show()
plot_stacked_bar_chart(
    df, 
    'num_words_range',
    chart_title='Percentage of positives and negatives sentiments per number of words range',
    export_to='images/sentiment_percentage_per_num_words_range.png'
)
sns.histplot(data=df, x="num_words", hue="sentiment", bins = 10,  multiple="dodge")
plt.title('Histogram of number of words per sentiment')
plt.savefig('images/histogram_number_words_per_sentiment.png', dpi='figure', backend=None)
plt.show()<jupyter_output><empty_output><jupyter_text>4. Does the tweet's LENGHT help determine whether it has positive or negative sentiment?<jupyter_code>df['tweet_lenght'] = df['text'].apply(lambda x : len(x))
sns.histplot(data=df, x="tweet_lenght", hue="sentiment", bins = 10,  multiple="dodge")
plt.title('Histogram of number of characters per sentiment')
plt.savefig('images/histogram_number_characters_per_sentiment.png', dpi='figure', backend=None)
plt.show()<jupyter_output><empty_output><jupyter_text>5. Is it possible to predict the sentiment of a new tweet using the existing ones?  a) Spliting the data in training, validation and test sets In order to reduce the time training the models and because we have limited CPU and GPU resources, we are going to reduce the dataset<jupyter_code>np.random.seed(42)
reduced_df = df.sample(int(len(df)*0.1))
print(f'New dataset size {len(reduced_df)} ({round(100*len(reduced_df)/len(df))}% of the original dataset)')<jupyter_output>New dataset size 160000 (10% of the original dataset)<jupyter_text>First, we select the columns which we are going to use to train the models (cleaned_text for features and sentiment for target)<jupyter_code>X = reduced_df['preprocessed_text']
y = reduced_df['sentiment']<jupyter_output><empty_output><jupyter_text>Splitting the dataset in training, validation and test set. The training set is used to train our models and the validation set it's used for measure how well our model perform on new data helping identify some underfitting or overfitting situations. Finally, the test set it's only used once when we are confident on the best model results and verify how it perform in another new data, but this time the test set was not used the choose the best model as we did with the validation set.<jupyter_code>X_train_val, X_test, y_train_val, y_test = train_test_split(X, y, test_size= 0.1, random_state=42)
X_train, X_val, y_train, y_val = train_test_split(X_train_val, y_train_val, test_size= 0.25, random_state=42)<jupyter_output><empty_output><jupyter_text>The number of examples and percentage of every set.<jupyter_code>print(f'Train set size: {X_train.shape[0]} ({round(100*X_train.shape[0]/reduced_df.shape[0],1)}%)')
print(f'Validation set size: {X_val.shape[0]} ({round(100*X_val.shape[0]/reduced_df.shape[0],1)}%)')
print(f'Test set size: {X_test.shape[0]} ({round(100*X_test.shape[0]/reduced_df.shape[0],1)}%)')<jupyter_output>Train set size: 108000 (67.5%)
Validation set size: 36000 (22.5%)
Test set size: 16000 (10.0%)<jupyter_text>b) Data prep using Keras Tokenizer We are using the Keras Tokenizer class in order to convert words into numbers, so our models could use it properly (some examples below).<jupyter_code>tokenizer = text.Tokenizer(num_words=2000000)
tokenizer.fit_on_texts(X_train)<jupyter_output><empty_output><jupyter_text>We have a total of 67182 distinct words on the training set<jupyter_code>total_words = len(tokenizer.word_index)
total_words<jupyter_output><empty_output><jupyter_text>Converting the text into numbers (tokens).<jupyter_code>train_sequences = tokenizer.texts_to_sequences(X_train)
val_sequences = tokenizer.texts_to_sequences(X_val)<jupyter_output><empty_output><jupyter_text>Using the padding function to make all the examples with the same input size.<jupyter_code>padded_train = sequence.pad_sequences(train_sequences, maxlen = 30)
padded_val = sequence.pad_sequences(val_sequences, maxlen = 30)<jupyter_output><empty_output><jupyter_text>Examples of the tokenizer and the padding preprocessing.<jupyter_code>X_train[:5]
train_sequences[:5]
padded_train[:5]<jupyter_output><empty_output><jupyter_text>For the target variable, we are using the One Hot Encoding technique to get the probability for each sentiment, and use it if we need.<jupyter_code>y_train_cat = to_categorical(y_train, 2)
y_val_cat = to_categorical(y_val, 2)<jupyter_output><empty_output><jupyter_text>Examples showing the One Hot Encoding technique.<jupyter_code>y_train[:5].values
y_train_cat[:5]<jupyter_output><empty_output><jupyter_text>c) Creating a LSTM Neural Networks<jupyter_code>lstm_model = models.Sequential()
lstm_model.add(layers.Embedding(input_dim = total_words, output_dim=1024))

lstm_model.add(layers.LSTM(256))

lstm_model.add(layers.Dense(128, activation='relu'))
lstm_model.add(layers.Dropout(0.2))
lstm_model.add(layers.Dense(2, activation='softmax'))

lstm_model.summary()<jupyter_output>Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 embedding (Embedding)       (None, None, 1024)        68794368  
                                                                 
 lstm (LSTM)                 (None, 256)               1311744   
                                                                 
 dense (Dense)               (None, 128)               32896     
                                                                 
 dropout (Dropout)           (None, 128)               0         
                                                                 
 dense_1 (Dense)             (None, 2)                 258       
                                                                 
=================================================================
Total params: 70,139,266
Trainable params: 70,139,266
No[...]<jupyter_text>We are using Adam optimizer and setting a callback to stop the model training early if our metrics does not get better per epoch.<jupyter_code>adam = optimizers.Adam(learning_rate=0.0001)
early = callbacks.EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)<jupyter_output><empty_output><jupyter_text>Since we have about the same number of positives and negatives examples we can use accuracy as classification metric to choose the best model. If that wasn't the case, we should use balanced accuracy in order to get a better view for classification of both classes.<jupyter_code>y_train.value_counts()
lstm_model.compile(
    optimizer = adam, 
    loss = 'categorical_crossentropy', 
    metrics = ['acc']
)<jupyter_output><empty_output><jupyter_text>Now, let's train our LSTM model<jupyter_code>lstm_model.fit(
    padded_train, y_train_cat, 
    epochs=20, 
    batch_size=100, 
    validation_data=(padded_val, y_val_cat),
    callbacks = [early]
)<jupyter_output>Epoch 1/20
1080/1080 [==============================] - 36s 28ms/step - loss: 0.5301 - acc: 0.7312 - val_loss: 0.4858 - val_acc: 0.7666
Epoch 2/20
1080/1080 [==============================] - 29s 27ms/step - loss: 0.4224 - acc: 0.8098 - val_loss: 0.4946 - val_acc: 0.7672
Epoch 3/20
1080/1080 [==============================] - 29s 27ms/step - loss: 0.3539 - acc: 0.8465 - val_loss: 0.5409 - val_acc: 0.7566
Epoch 4/20
1080/1080 [==============================] - 29s 27ms/step - loss: 0.2949 - acc: 0.8740 - val_loss: 0.5950 - val_acc: 0.7462
Epoch 5/20
1080/1080 [==============================] - 29s 27ms/step - loss: 0.2493 - acc: 0.8926 - val_loss: 0.6699 - val_acc: 0.7460
Epoch 6/20
1080/1080 [==============================] - 29s 27ms/step - loss: 0.2161 - acc: 0.9068 - val_loss: 0.7282 - val_acc: 0.7366<jupyter_text>With our trained model we can predict the sentiment and measure it's accuracy.<jupyter_code>lstm_train_pred = lstm_model.predict(padded_train)
lstm_val_pred = lstm_model.predict(padded_val)
lstm_train_pred = np.argmax(lstm_train_pred, axis=1)
lstm_val_pred = np.argmax(lstm_val_pred, axis=1)
accuracy_score(y_train, lstm_train_pred)
accuracy_score(y_val, lstm_val_pred)<jupyter_output><empty_output><jupyter_text>d) Creating a Transformer Deep Network with BERT Transformers are a type of Neural Networks that differs from others by using a "Attention" layer. It creates a relationship between each word and any other word ignoring the order of the words in the sentence but it comes with the cost of using a lot of memory to store all the neural network weights. BERT: Biderectional Encoder Representations from Transformer is a Transformer encoder trained on Wikipedia data. Since it is a a pre-trained model, we can fine tune it to retrain using our data. Creating our features and target variables.<jupyter_code>X = reduced_df[['preprocessed_text', 'target']]
y = reduced_df['sentiment']<jupyter_output><empty_output><jupyter_text>Sppliting our data as we did before.<jupyter_code>X_train_val, X_test, y_train_val, y_test = train_test_split(X, y, test_size= 0.1, random_state=42)
X_train, X_val, y_train, y_val = train_test_split(X_train_val, y_train_val, test_size= 0.25, random_state=42)
print(f'Train set size: {X_train.shape[0]} ({round(100*X_train.shape[0]/reduced_df.shape[0],1)}%)')
print(f'Validation set size: {X_val.shape[0]} ({round(100*X_val.shape[0]/reduced_df.shape[0],1)}%)')
print(f'Test set size: {X_test.shape[0]} ({round(100*X_test.shape[0]/reduced_df.shape[0],1)}%)')<jupyter_output>Train set size: 108000 (67.5%)
Validation set size: 36000 (22.5%)
Test set size: 16000 (10.0%)<jupyter_text>Creating one DataFrame for training and validation sets because ktrain package requires DataFrames instead of numpy series.<jupyter_code>train_df = X_train.copy()
train_df['sentiment'] = y_train

val_df = X_val.copy()
val_df['sentiment'] = y_val<jupyter_output><empty_output><jupyter_text>Getting the biggest sentence lenght to be used to preprocess the data.<jupyter_code>sentence_lenghts = X_train['preprocessed_text'].apply(lambda x: len(x))
max_sentence_lenght = sentence_lenghts.max()
max_sentence_lenght<jupyter_output><empty_output><jupyter_text>In order to train our BERT model, we need to use the same data preprocessing used to train it. So we are going to this preprocessing on our data.<jupyter_code>(x_train, y_train), (x_val, y_val), preproc = ktrain.text.texts_from_df(
    train_df=train_df, 
    text_column='preprocessed_text', 
    label_columns=['sentiment'], 
    val_df=val_df, 
    max_features=20000, 
    maxlen=max_sentence_lenght, 
    preprocess_mode='bert', 
    random_state=42, 
    verbose=1
  )<jupyter_output>['not_sentiment', 'sentiment']
         not_sentiment  sentiment
792113             1.0        0.0
919355             0.0        1.0
1105358            0.0        1.0
738376             1.0        0.0
187954             1.0        0.0
['not_sentiment', 'sentiment']
         not_sentiment  sentiment
84910              1.0        0.0
1240991            0.0        1.0
426435             1.0        0.0
793017             1.0        0.0
674790             1.0        0.0
downloading pretrained BERT model (uncased_L-12_H-768_A-12.zip)...
[██████████████████████████████████████████████████]
extracting pretrained BERT model...
done.

cleanup downloaded zip...
done.

preprocessing train...
language: en<jupyter_text>Let's import the BERT classifier and set the training and validation sets.<jupyter_code>bert_model = ktrain.text.text_classifier('bert', (x_train, y_train), preproc=preproc)
bert_learner = ktrain.get_learner(
    bert_model, 
    train_data=(x_train, y_train), 
    val_data=(x_val, y_val), 
    batch_size=32
)<jupyter_output>Is Multi-Label? False
maxlen is 128
done.<jupyter_text>Now, let's train our BERT model (be aware that it might take a while).<jupyter_code>bert_learner.fit_onecycle(2e-5, 5)<jupyter_output>begin training using onecycle policy with max lr of 2e-05...
Epoch 1/5
3375/3375 [==============================] - 1859s 546ms/step - loss: 0.4951 - accuracy: 0.7577 - val_loss: 0.4541 - val_accuracy: 0.7866
Epoch 2/5
3375/3375 [==============================] - 1837s 544ms/step - loss: 0.4338 - accuracy: 0.7980 - val_loss: 0.4462 - val_accuracy: 0.7927
Epoch 3/5
3375/3375 [==============================] - 1837s 544ms/step - loss: 0.3721 - accuracy: 0.8341 - val_loss: 0.4673 - val_accuracy: 0.7918
Epoch 4/5
3375/3375 [==============================] - 1837s 544ms/step - loss: 0.2414 - accuracy: 0.9003 - val_loss: 0.5815 - val_accuracy: 0.7876
Epoch 5/5
3375/3375 [==============================] - 1837s 544ms/step - loss: 0.1236 - accuracy: 0.9518 - val_loss: 0.7555 - val_accuracy: 0.7861<jupyter_text>Loading our predictor from the trained model.<jupyter_code>bert_predictor = ktrain.get_predictor(bert_learner.model, preproc)<jupyter_output><empty_output><jupyter_text>Predicting the sentiment for training and validation sets and calculating the accuracy.<jupyter_code>bert_train_pred = bert_predictor.predict(X_train['preprocessed_text'].values)
bert_val_pred = bert_predictor.predict(X_val['preprocessed_text'].values)
bert_train_pred = (np.array(bert_train_pred) == 'sentiment')*1
bert_val_pred = (np.array(bert_val_pred) == 'sentiment')*1
accuracy_score(np.argmax(y_train, axis=1), bert_train_pred)
accuracy_score(np.argmax(y_val, axis=1), bert_val_pred)<jupyter_output><empty_output><jupyter_text>Since we got a better accuracy using the BERT model on the validation set, we are going to use this model to predict on our test set and calculate the accuracy.<jupyter_code>bert_test_pred = bert_predictor.predict(X_test['preprocessed_text'].values)
bert_test_pred = (np.array(bert_test_pred) == 'sentiment')*1
accuracy_score(y_test, bert_test_pred)<jupyter_output><empty_output><jupyter_text>e) Using the model in other tweets Now we are going to test our model in some recent tweets and see how it performs. When a tweet is classified as "sentiment" it means it is a positive sentiment by the model and a "not_sentiment" classification means a negative sentiment. ![elon1.png](elon1.png)<jupyter_code>bert_predictor.predict('Tesla holiday software update goes to wide release today!')<jupyter_output><empty_output><jupyter_text>![biden1.png](biden1.png)<jupyter_code>bert_predictor.predict('Clean water for every American. We’ve never done that. And now we are. High-speed internet for every American.  We’ve never done that. And now we are. As a nation, we’ve only scratched the surface of what we’re capable of.')<jupyter_output><empty_output><jupyter_text>![tim1.png](tim1.png)<jupyter_code>bert_predictor.predict('Our hearts go out to all those affected by the devastating weather and tornadoes across the South and Midwest. Apple will be donating to support relief efforts on the ground.')<jupyter_output><empty_output><jupyter_text>![bill1.png](bill1.png)<jupyter_code>bert_predictor.predict('Just when it seemed like life would return to normal, we could be entering the worst part of the pandemic. Omicron will hit home for all of us. Close friends of mine now have it, and I’ve canceled most of my holiday plans.')
bert_predictor.predict('If there’s good news here, it’s that omicron moves so quickly that once it becomes dominant in a country, the wave there should last less than 3 months. Those few months could be bad, but I still believe if we take the right steps, the pandemic can be over in 2022.')
bert_predictor.predict('I know it’s frustrating to go into another holiday season with COVID looming over us. But it won’t be like this forever. Someday the pandemic will end, and the better we look after each other, the sooner that time will come.')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12584245235161967, 'fraction_numerical': 0.03587216464961229, 'mean_word_length': 4.852205258693808, 'pattern_counts': {'":': 5, '<': 187, '<?xml version=': 0, '>': 183, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 15, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '47', 'dir': 'jupyter-structured-clean-dedup', 'id': '363871', 'license': 'Unlicense', 'n_tokens_mistral': 9882, 'n_tokens_neox': 8955, 'n_words': 2488, 'path': 'twitter_sentiment_analysis.ipynb', 'repo_name': 'AndreGoncalves013/Twitter_Sentiment_Analysis', 'size': '27598'}