<jupyter_start><jupyter_text>Contextual Model for Real Time Tweet Sentiment PredictionSentiment Analysis has an important role in today’s world especially for private companieswhich hold lots of data. The massive amount of data generated by Twitter present a uniqueopportunity for sentiment analysis. However, it is challenging to build an accurate predictivemodel to identify sentiments, which may lack sufficient context due to the length limit. Inaddition, sentimental and regular ones can be hard to separate because of word ambiguity. Inthis notebook, I will be proposing the phases of text pre-processing, visual analysis and modeling.***I tried to keep code as simple as possible to remain understandable.***Proposed **BERT-CNN-BiLSTM-FC** learning pipeline, which consists of **four sequential modules**.BERT produces competitive results, and can be considered as one of the new electricity of naturallanguage processing tasks such as sentiment analysis, named entity recognition (NER), and topicmodeling. The combination of CNN and BiLSTM models requires a particular design, since eachmodel has a specific architecture and its own strengths:• BERT is utilized to transform word tokens from the raw Tweet messages to contextual wordembeddings.• CNN is known for its ability to extract as many features as possible from the text.• BiLSTM keeps the chronological order between words in a document, thus it has the abilityto ignore unnecessary words using the delete gate.• Fully Connected Layers give robustness to decrease unsteadiness of results in hard cases.  > If you find my work useful please don't forget to **Upvote!**  so it can reach more people.**References:**1) [A Sentiment-Aware Contextual Model for Real-Time Disaster Prediction Using Twitter Data](https://www.mdpi.com/1999-5903/13/7/163/htm) -> The idea comes from and really worth to check on, however, i am not using the same model.2) [Automatic identification of eyewitness messages on twitter during disasters](https://reader.elsevier.com/reader/sd/pii/S0306457319303590?token=<PASSWORD>20D4DCD96FDADE4A642A9805D728945987C1BBBE0FDAA8EC3684E372C7&originRegion=eu-west-1&originCreation=20210920022341)3) [Convolutional Neural Networks for Sentence Classification](http://arxiv.org/abs/1408.5882)4) [BERT: Pre-training of Deep Bidirectional Transformers for Language               Understanding](http://arxiv.org/abs/1810.04805)5) [LMAES' Notebook](https://www.kaggle.com/lmasca/disaster-tweets-using-bert-embeddings-and-lstm)6) [<NAME>'s Notebook](https://www.kaggle.com/paoloripamonti/twitter-sentiment-analysis)<jupyter_code># few of the imports are just for checking while coding not included in the rest of notebook.

# Most basic stuff for EDA.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# Core packages for text processing.
import string
import re

# Libraries for text preprocessing.
import nltk
nltk.download('stopwords')
nltk.download('punkt')
nltk.download('averaged_perceptron_tagger')
from nltk.corpus import stopwords

# Loading some sklearn packaces for modelling.
from sklearn.preprocessing import LabelEncoder
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer # not actively using
from sklearn.decomposition import LatentDirichletAllocation, NMF # not actively using
from sklearn.metrics import confusion_matrix, classification_report, accuracy_score
from sklearn.model_selection import train_test_split

# Utility
import logging
import itertools

# Core packages for general use throughout the notebook.
import random
import warnings
import time
import datetime

# For customizing our plots.
from matplotlib.ticker import MaxNLocator
import matplotlib.gridspec as gridspec
import matplotlib.patches as mpatches

# for build our model
import tensorflow as tf

from tensorflow.keras.layers import Embedding, LSTM, Dense, Bidirectional
from tensorflow.keras.layers.experimental.preprocessing import TextVectorization
from tensorflow.keras.models import Sequential
from tensorflow.keras.optimizers import Adam
from transformers import BertTokenizer, TFBertModel

# Setting some options for general use.
import os
stop = set(stopwords.words('english'))
plt.style.use('fivethirtyeight')
sns.set(font_scale=1.5)
pd.options.display.max_columns = 250
pd.options.display.max_rows = 250
warnings.filterwarnings('ignore')<jupyter_output>[nltk_data] Downloading package stopwords to /usr/share/nltk_data...
[nltk_data]   Package stopwords is already up-to-date!
[nltk_data] Downloading package punkt to /usr/share/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /usr/share/nltk_data...
[nltk_data]   Package averaged_perceptron_tagger is already up-to-
[nltk_data]       date!<jupyter_text>Read Dataset[Data source](www.kaggle.com/c/tweet-sentiment-extraction/data)Train data file format has 4 fields: * textID: The id of the tweet* text: the text of the tweet* selected_text: support phrases for sentiment labels* sentiment -> (negative, neutral, positive)Test data file format has 3 fields: * textID: The id of the tweet* text: the text of the tweet* sentiment -> (negative, neutral, positive)<jupyter_code># Read the data
df_train = pd.read_csv('../input/tweet-sentiment-extraction/train.csv')
df_test = pd.read_csv('../input/tweet-sentiment-extraction/test.csv')
# Raw data
df_train.head()
print("null text = ", pd.isna(df_train['text']).sum())
print("null sentiment = ", pd.isna(df_train['sentiment']).sum())
df_train = df_train.dropna(subset=['text'], axis=0)
print("null text = ", pd.isna(df_train['text']).sum())
df_train.describe()<jupyter_output><empty_output><jupyter_text>Visualizing the Data<jupyter_code>temp = df_train.groupby('sentiment').count()['text'].reset_index().sort_values(by='text',ascending=False)
temp.style.background_gradient(cmap='Purples')<jupyter_output><empty_output><jupyter_text>Cleaning Text<jupyter_code># Some basic helper functions to clean text by removing urls, emojis, html tags and punctuations.

def remove_URL(text):
    url = re.compile(r'https?://\S+|www\.\S+')
    return url.sub(r'', text)



def remove_html(text):
    html = re.compile(r'<.*?>|&([a-z0-9]+|#[0-9]{1,6}|#x[0-9a-f]{1,6});')
    return re.sub(html, '', text)


def remove_punct(text):
    table = str.maketrans('', '', string.punctuation)
    return text.translate(table)

# Applying helper functions
df_train['text_clean'] = df_train['text'].apply(lambda x: remove_URL(x))
df_train['text_clean'] = df_train['text_clean'].apply(lambda x: remove_html(x))
df_train['text_clean'] = df_train['text_clean'].apply(lambda x: remove_punct(x))

df_test['text_clean'] = df_test['text'].apply(lambda x: remove_URL(x))
df_test['text_clean'] = df_test['text_clean'].apply(lambda x: remove_html(x))
df_test['text_clean'] = df_test['text_clean'].apply(lambda x: remove_punct(x))
df_train.head()
print("TRAIN size:", len(df_train))
print("TEST size:", len(df_test))<jupyter_output>TRAIN size: 27480
TEST size: 3534<jupyter_text>________________  Setup environment to build model<jupyter_code>os.environ["WANDB_API_KEY"] = "0" ## to silence warning
try:
    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()
    tf.config.experimental_connect_to_cluster(tpu)
    tf.tpu.experimental.initialize_tpu_system(tpu)
    strategy = tf.distribute.experimental.TPUStrategy(tpu)
except ValueError:
    strategy = tf.distribute.get_strategy() # for CPU and single GPU
    print('Number of replicas:', strategy.num_replicas_in_sync)
# hyperparameters
max_length = 128
batch_size = 128
# Bert Tokenizer
model_name = "bert-base-uncased"
tokenizer = BertTokenizer.from_pretrained(model_name)<jupyter_output><empty_output><jupyter_text>Splitting the dev data<jupyter_code>train, dev = train_test_split(df_train, test_size=0.1, random_state=42)<jupyter_output><empty_output><jupyter_text>Label Encoder<jupyter_code>labels = df_train.sentiment.unique().tolist()
labels
encoder = LabelEncoder()
encoder.fit(df_train.sentiment.tolist())

y_train = encoder.transform(df_train.sentiment.tolist())
y_test = encoder.transform(df_test.sentiment.tolist())
y_dev = encoder.transform(dev.sentiment.tolist())

y_train = y_train.reshape(-1,1)
y_test = y_test.reshape(-1,1)
y_dev = y_dev.reshape(-1,1)

print("y_train",y_train.shape)
print("y_test",y_test.shape)
def bert_encode(data):
    tokens = tokenizer.batch_encode_plus(data, max_length=max_length, padding='max_length', truncation=True)
    
    return tf.constant(tokens['input_ids'])
train_encoded = bert_encode(df_train.text_clean)
dev_encoded = bert_encode(dev.text_clean)


train_dataset = (
    tf.data.Dataset
    .from_tensor_slices((train_encoded, y_train))
    .shuffle(128)
    .batch(batch_size)
)

dev_dataset = (
    tf.data.Dataset
    .from_tensor_slices((dev_encoded, y_dev))
    .shuffle(128)
    .batch(batch_size)
)<jupyter_output><empty_output><jupyter_text>Proposed Model<jupyter_code>def bert_tweets_model():

    bert_encoder = TFBertModel.from_pretrained(model_name)
    input_word_ids = tf.keras.Input(shape=(max_length,), dtype=tf.int32, name="input_ids")
    last_hidden_states = bert_encoder(input_word_ids)[0]   
    x = tf.keras.layers.SpatialDropout1D(0.2)(last_hidden_states)
    x = tf.keras.layers.Conv1D(64, 3, activation='relu')(x)
    x = tf.keras.layers.Bidirectional(LSTM(64, dropout=0.2, recurrent_dropout=0.2))(x)
    x = tf.keras.layers.Dense(128, activation='relu')(x)
    x = tf.keras.layers.Dropout(0.2)(x)
    x = tf.keras.layers.Dense(64, activation='relu')(x)
    x = tf.keras.layers.Dropout(0.3)(x)
    outputs = tf.keras.layers.Dense(3, activation='softmax')(x)
    model = tf.keras.Model(input_word_ids, outputs)
   
    
    return model
with strategy.scope():
    model = bert_tweets_model()
    adam_optimizer = tf.keras.optimizers.Adam(learning_rate=1e-5)
    model.compile(loss='sparse_categorical_crossentropy',optimizer=adam_optimizer,metrics=['accuracy'])

    model.summary()
tf.keras.utils.plot_model(model, show_shapes=True)
# Start train
history = model.fit(
    train_dataset,
    batch_size=batch_size,
    epochs=12,
    validation_data=dev_dataset,
    verbose=1)
    #callbacks=[tf.keras.callbacks.EarlyStopping(
    #            patience=6,
    #            min_delta=0.05,
    #            baseline=0.7,
    #            mode='min',
    #            monitor='val_accuracy',
    #            restore_best_weights=True,
    #            verbose=1)
    #          ])
def plot_graphs(history, string):
  plt.plot(history.history[string])
  plt.plot(history.history['val_'+string])
  plt.xlabel("Epochs")
  plt.ylabel(string)
  plt.legend([string, 'val_'+string])
  plt.show()
  
plot_graphs(history, "accuracy")
plot_graphs(history, "loss")<jupyter_output><empty_output><jupyter_text>Predict Manually Before Using Test DataDecoder to be able to see results as labelled negative, positive or neutral<jupyter_code>def decode_sentiment(score):
        if score == 0:
            return "negative"
        elif score == 1:
            return "neutral"
        else:
            return "positive"
def predict(text, include_neutral=True):
    start_at = time.time()
    # Tokenize text
    x_encoded = bert_encode([text])
    # Predict
    score = model.predict([x_encoded])[0]
    # Decode sentiment
    label = decode_sentiment(np.argmax(score))

    return {"label": label, "score": score,
       "elapsed_time": time.time()-start_at}  
predict("I hate the economy")
predict("I would prefer writing a crawler to create this dataset but i couldn't")
predict("I LOVE NLP")
predict("life is really strange isn't it? just the combination of laugh and sadness")
predict("ESL is the world's largest esports company, leading the industry across the most popular video games.\
        We're proud they've chosen us to help them deliver their launchers to gamers all over the world. Read the full review")
predict("Excited to present a tutorial on 'Modular and Parameter-Efficient Fine-Tuning for NLP Models' \
        at #EMNLP2022 with @PfeiffJo & @licwu.")
predict("Had a song stuck in my head. Thirty seconds later I'm listening to it, thanks to the internet,\
        and Apple/YouTube Music. In the bad old days I'd browse record stores for hours in the hope that the title might jog my memory.\
        It really is a wonderful time to be alive!")
predict("i don't say this lightly - hemingway's life ended by suicide. His life was actually a loss")
predict("these r not ur problems dear!!! these r ur x bf's commitng suicide")
predict("i hve no idea about that i love the uni or not")
predict("I found some old Reddit post in which one guy from english-speaking country complains that\
the names in The Witcher books are 'too difficult' and non- intuitive for english speaker.\
Man, let me introduce you to 'The books werent written only/for english speakers.'' #witcher")
predict("I forgot how cringy all the Slavic names sound read it English \
YOU'RE PRONOUNCING IT ALL WRONG MY EARS ARE HURTING AND I DON'T EVEN HAVE HEARING AIDS IN")
predict("fun fact: ai cannot predict everything right")
predict("our brain is just a machine as well")<jupyter_output><empty_output><jupyter_text>Evaluate<jupyter_code>test_encoded = bert_encode(df_test.text_clean)

test_dataset = (
    tf.data.Dataset
    .from_tensor_slices(test_encoded)
    .batch(batch_size)
)

predicted_tweets = model.predict(test_dataset, batch_size=batch_size)
predicted_tweets_binary = tf.cast(tf.round(predicted_tweets), tf.int32).numpy().flatten()
y_pred = []
for i in range(predicted_tweets.shape[0]):
    y_pred.append(np.argmax(predicted_tweets[i]))
print(classification_report(y_test, y_pred))<jupyter_output>precision    recall  f1-score   support

           0       0.81      0.73      0.77      1001
           1       0.72      0.79      0.75      1430
           2       0.84      0.80      0.82      1103

    accuracy                           0.78      3534
   macro avg       0.79      0.77      0.78      3534
weighted avg       0.78      0.78      0.78      3534<jupyter_text>_____ In next step, we will use our prepared model to predict tweets on twitter using Twint.  Fetching data from TwitterTo get started,* Import the twint package as follows.<jupyter_code>!pip install twint
import twint
import nest_asyncio
nest_asyncio.apply() 
!pip install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint
c = twint.Config()
c.Search = "GRAMMYs" #keyword for search
c.Limit = 100 #limit of the number of tweets which will be extracted
c.Store_csv = True 
c.Output = '#GRAMMYs_tweet_data.csv'
twint.run.Search(c)<jupyter_output>1473980883686367237 2021-12-23 11:36:42 +0000 <Adi_Sterling> Checkout Tunechi’s ‘drop the world’ performance at the 2010 Grammys with <NAME>.
1473980339450118147 2021-12-23 11:34:32 +0000 <vynguyenle1408> @Interzoo0 @crashingstyle89 @talkofthecharts yes i mean grammys
1473980033270222852 2021-12-23 11:33:19 +0000 <Interzoo0> @vynguyenle1408 @crashingstyle89 @talkofthecharts related to the Grammys?
1473979718307397635 2021-12-23 11:32:04 +0000 <iCiara> @hushmirrorball I think she will do something like this in the build up to the Grammys for Evermore. I’m afraid I do indeed need your closure, blondie. 🙏🏻
1473979352178212865 2021-12-23 11:30:37 +0000 <JideMawutor> Wizkid shot both Essence and the MIL Short Films in Ghana with Ghanaian crew.  It is not that we don't have the resources ohh. Do we know how to use it?  Essence is nominated for the Best Global Music Performance at the Grammys.  The short film is even better.
1473979044530171909 2021-12-23 11:29:24 +0000 <qorton4> @isaa[...]<jupyter_text>We stored the related tweets in the .csv or .json file which is really fast and coolSo how we will read from csv/json file to use for our purpose ?<jupyter_code>crawled_data = pd.read_csv("#GRAMMYs_tweet_data.csv")
#crawled_data = pd.read_json("tweet_data.json", lines=True)
pd.options.display.max_columns=36
crawled_data.head()<jupyter_output><empty_output><jupyter_text>_____As you can see above we have lots of features which extracted by twint. However, we only need the "tweet" feature which includes the text data of tweets for our purpose.<jupyter_code># prediction of the first 15 extracted tweets
for i in range(15):
    print(crawled_data["tweet"][i])
    print(predict(crawled_data["tweet"][i]))
    print("\n")<jupyter_output>Checkout Tunechi’s ‘drop the world’ performance at the 2010 Grammys with Travis Barker.
{'label': 'neutral', 'score': array([0.01953116, 0.8722475 , 0.10822131], dtype=float32), 'elapsed_time': 0.39107751846313477}


@Interzoo0 @crashingstyle89 @talkofthecharts yes i mean grammys
{'label': 'neutral', 'score': array([0.01343942, 0.95331293, 0.03324767], dtype=float32), 'elapsed_time': 0.390610933303833}


@vynguyenle1408 @crashingstyle89 @talkofthecharts related to the Grammys?
{'label': 'neutral', 'score': array([0.00696629, 0.97773516, 0.01529855], dtype=float32), 'elapsed_time': 0.3901488780975342}


@hushmirrorball I think she will do something like this in the build up to the Grammys for Evermore. I’m afraid I do indeed need your closure, blondie. 🙏🏻
{'label': 'neutral', 'score': array([0.01787372, 0.94304776, 0.03907843], dtype=float32), 'elapsed_time': 0.4018416404724121}


Wizkid shot both Essence and the MIL Short Films in Ghana with Ghanaian crew.  It is not that we don't have[...]
{'fraction_non_alphanumeric': 0.10404127257093723, 'fraction_numerical': 0.041501862997993695, 'mean_word_length': 4.915903696168193, 'pattern_counts': {'":': 3, '<': 58, '<?xml version=': 0, '>': 62, 'https://': 5, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '376870', 'license': 'Apache-2.0', 'n_tokens_mistral': 6195, 'n_tokens_neox': 5630, 'n_words': 1606, 'path': 'sentiment-analysis-for-multi-class-tweets.ipynb', 'repo_name': 'Toygarr/Sentiment-aware-Model-to-Classify-Tweets-in-Real-Time', 'size': '17489'}