<jupyter_start><jupyter_text>3. Mood Detection of Tweets - Word Embeddings and LSTMNow that we have our cleaned dataset, we will now continue to train a neural network to classify a tweet's mood.  Testing for CUDA-enabled TF with GPU Support**What this means:**Deep Learning requires a lot of matrix calculations and your CPU is not really meant for that sort of task. However, graphics cards (the hardware that your computer uses to make games run well) are able to perform these sort of tasks very well, so we need to check if the Jupyter Notebook has access to your (Nvidia) GPU, in order to train the model much more efficiently.<jupyter_code>import tensorflow as tf
if tf.test.gpu_device_name():
    print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))
else:
    print("Please install GPU version of TF")
print(tf.test.is_built_with_cuda())
print(tf.keras.__version__)
print(tf.__version__)<jupyter_output>True
2.2.4-tf
1.13.1<jupyter_text>Processing our mood tweets datasetDataset can also be accessed at: [future Kaggle link for my dataset]   Reading the CSV dataset with pandas<jupyter_code>import pandas as pd

df = pd.read_csv("datasets/mood_tweets.csv")
df.drop(df.columns[0], inplace=True, axis=1)
df["tweet"] = df["tweet"].astype(str)
df.head()
df.describe()<jupyter_output><empty_output><jupyter_text>Now let's analyse how the dataset is lain out and roughly how the data looks like<jupyter_code>pd.set_option('display.max_colwidth', -1) # option to be set so that the tweet's texts won't be truncated
df.sort_values("class", inplace=True)
display(df.head())
display(df.tail())
df["class"].unique()<jupyter_output><empty_output><jupyter_text>We will now change the classes into a numerical representation to make it easier for deep learning.<jupyter_code>classes_index = {
    0 : "suicidal",
    1 : "depressed",
    2 : "sad",
    3 : "happy",
    4 : "cheerful",
    5 : "overjoyed",
}

for val, class_name in classes_index.items():
    df.loc[(df["class"] == class_name), "class"] = val

display(df.head())
display(df.tail())<jupyter_output><empty_output><jupyter_text>Loading Glove word embeddings.Right now, we have our tweets in text form. In order for our neural network to take them in as inputs, they have to be numerical. Therefore, we will be performing 2 tasks:1. Tokenising the text2. Adding a second layer in the neural network to map the text to their respective word embeddingsWord embeddings are vector representations of distinct words in a vector space of many dimensions (in this case we are using 100 dimensions). This gives every word a distinct vector, and we can perform some very interesting vector operations on them.A lot of the code here is referenced from: https://github.com/keras-team/keras/blob/master/examples/pretrained_word_embeddings.pyNote: Please run the 2 cells below in a Linux environment / Google Colab so that the commands work properly. Windows might not have the proper binaries to perform these commands.<jupyter_code>!wget -qq http://nlp.stanford.edu/data/glove.6B.zip

!unzip glove.6B.zip<jupyter_output><empty_output><jupyter_text>The Glove word embeddings should be downloaded and unzipped. Check the output of the cell below to see if they are reflected correctly.<jupyter_code>!ls<jupyter_output>'1. Proof of Concept - Sentiment Analysis on Airline Tweets.ipynb'
'2. Creating Mood Tweets Dataset.ipynb'
'3. Mood Detection of Tweets - Word Embeddings and LSTM.ipynb'
 api-keys
 best_weights.hdf5
 datasets
 glove.6B.100d.txt
 glove.6B.200d.txt
 glove.6B.300d.txt
 glove.6B.50d.txt
 glove.6B.zip
 LICENSE
 mood_detection_model.h5
 mood_tweets.xlsx
 my_api_keys.py
 my_api_keys_SAMPLE.py
 __pycache__
 README.md<jupyter_text>Create a new dictionary to map the words in the embeddings to their 100-dimension embedding vector<jupyter_code>import os
import numpy as np
# NumPy is a very widely-used library in Python for mathematical operations.
# It supports many complex mathematical functions and can work on large arrays and matrices.

embeddings_index = {}
with open(os.path.join('glove.6B.100d.txt')) as f:
    for line in f:
        values = line.split()
        word = values[0]
        coefs = np.asarray(values[1:], dtype='float32')
        embeddings_index[word] = coefs

print("Found {} word vectors.".format(len(embeddings_index)))
# to see what word embeddings look like
embeddings_index["chris"]<jupyter_output><empty_output><jupyter_text>This array of numbers make up the representation of the word "chris" in 100d-space. This is beneficial as it gives every trained word a distinct vector representation, and we can calculate the cosine similarity between 2 different words to determine how similar / different the words are, in terms of contextual information.  Processing them for TensorFlowTensorFlow (& Keras) is the library/framework which we will use to train our neural network.We will process our dataframe above into TF-ready numpy arrays.<jupyter_code>import sys

from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.layers import Dense, Input, LSTM, Embedding, Bidirectional, Dropout
from tensorflow.keras.models import Model
from tensorflow.keras.initializers import Constant
# Setting some parameters

MAX_SEQUENCE_LENGTH = 40 # determines how many words of every tweet will be processed
# if the tweet length is smaller than this, the tweet will be padded until it has a length equivalent to this

MAX_NUM_WORDS = 70000 # how many unique words in the training text to be tokenised
EMBEDDING_DIM = 100 # number of dimensions in the word embeddings
TRAINING_SPLIT = 0.8 # proportion of data to be used to training
# Converting the dataframe into lists

texts = df["tweet"].tolist()
labels = df["class"].tolist()<jupyter_output><empty_output><jupyter_text>Tokenising the Tweets into a 2D integer tensorTokenising tweets will break up the text into individual words, and replace them with a numeric representation.e.g. - "I am a human" > "i", "am", "a", "human" > 1, 2, 3, 4- "I am a robot" > "i", "am", "a", "robot" > 1, 2, 3, 5<jupyter_code>tokenizer = Tokenizer(num_words=MAX_NUM_WORDS)
tokenizer.fit_on_texts(texts)
sequences = tokenizer.texts_to_sequences(texts)
word_index = tokenizer.word_index

print("{} distinct words in original text".format(len(word_index)))<jupyter_output>67094 distinct words in original text<jupyter_text>All our tensors have to be of the same length, so we need to set a maximum length, and if a certain tweet has fewer than the maximum length of words, it will be padded until it is the same size.<jupyter_code>import matplotlib.pyplot as plt
%matplotlib inline
seq_lengths = [len(x) for x in sequences]
plt.plot(seq_lengths)
max(seq_lengths)<jupyter_output><empty_output><jupyter_text>Wow, the longest tweet is 117 words long... I'm quite curious about what the tweet is about so let's see.<jupyter_code>texts[np.argmax(seq_lengths)]<jupyter_output><empty_output><jupyter_text>Okay back to the task, as seen from the graph, the tweets vary quite a lot in length, and maxes out at 117. We will now pad all of them to 117.<jupyter_code>MAX_SEQUENCE_LENGTH = max(seq_lengths)

data = pad_sequences(sequences, maxlen=MAX_SEQUENCE_LENGTH)<jupyter_output><empty_output><jupyter_text>Creating One Hot Encoding Labels<jupyter_code>labels = to_categorical(np.asarray(labels))
print('Shape of data tensor:', data.shape)
print('Shape of label tensor:', labels.shape)<jupyter_output>Shape of data tensor: (25758, 117)
Shape of label tensor: (25758, 6)<jupyter_text>Splitting the data into a train test splitWhat this means is that we will allocate 80% of our cleaned up dataset for training the neural network. After every epoch (cycle) of training, the model will be "examined" against the other 20%, named the validation set, to see how well our model generalises (whether our model is 'memorising' its training set, which is undesirable).<jupyter_code># we shuffle our sorted data, to make it random first
indices = np.arange(data.shape[0])
np.random.shuffle(indices)
data = data[indices]
labels = labels[indices]

# before we begin to split them
num_training_samples = int(TRAINING_SPLIT * data.shape[0])
x_train = data[:num_training_samples]
y_train = labels[:num_training_samples]
x_val = data[num_training_samples:]
y_val = labels[num_training_samples:]<jupyter_output><empty_output><jupyter_text>Preparing Embedding Matrix<jupyter_code>num_words = min(MAX_NUM_WORDS, len(word_index)) + 1
embedding_matrix = np.zeros((num_words, EMBEDDING_DIM))
for word, i in word_index.items():
    if i > MAX_NUM_WORDS:
        continue
    embedding_vector = embeddings_index.get(word)
    if embedding_vector is not None:
        # words not found in embedding index will be all-zeros.
        embedding_matrix[i] = embedding_vector

# load pre-trained word embeddings into an Embedding layer
# note that we set trainable = False so as to keep the embeddings fixed
embedding_layer = Embedding(num_words,
                            EMBEDDING_DIM,
                            embeddings_initializer=Constant(embedding_matrix),
                            input_length=MAX_SEQUENCE_LENGTH,
                            trainable=False)<jupyter_output><empty_output><jupyter_text>Defining and creating our model<jupyter_code>sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')
embedded_sequences = embedding_layer(sequence_input)
x = Bidirectional(LSTM(256))(embedded_sequences)
y = Dropout(0.5)(x)

preds = Dense(labels.shape[1], activation='softmax')(y)
model = Model(sequence_input, preds)

model.summary()
model.compile(loss='categorical_crossentropy',
              optimizer='rmsprop',
              metrics=['acc'])<jupyter_output><empty_output><jupyter_text>Training the model<jupyter_code>from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint

checkpoint = ModelCheckpoint("best_weights.hdf5", monitor='val_acc', verbose=1, save_best_only=True, mode='max')
earlystopping = EarlyStopping(patience=4, monitor='acc')
callbacks = [earlystopping, checkpoint]

model.fit(x_train, y_train,
          batch_size=32,
          epochs=50,
          validation_data=(x_val, y_val),
          callbacks=callbacks,
          verbose=2)

model.load_weights("best_weights.hdf5")

print("DONE")<jupyter_output>Train on 20606 samples, validate on 5152 samples
WARNING:tensorflow:From /opt/anaconda/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
Epoch 1/50

Epoch 00001: val_acc improved from -inf to 0.93109, saving model to best_weights.hdf5
 - 193s - loss: 0.6475 - acc: 0.7846 - val_loss: 0.2567 - val_acc: 0.9311
Epoch 2/50

Epoch 00002: val_acc improved from 0.93109 to 0.97186, saving model to best_weights.hdf5
 - 193s - loss: 0.1517 - acc: 0.9600 - val_loss: 0.1042 - val_acc: 0.9719
Epoch 3/50

Epoch 00003: val_acc did not improve from 0.97186
 - 191s - loss: 0.1092 - acc: 0.9697 - val_loss: 0.0961 - val_acc: 0.9705
Epoch 4/50

Epoch 00004: val_acc improved from 0.97186 to 0.97651, saving model to best_weights.hdf5
 - 192s - loss: 0.1118 - acc: 0.9709 - val_loss: 0.0812 - val_acc: 0.9765
Epoch 5/50

Epoch 00005: val_acc did [...]<jupyter_text>Making predictions with the modelOur model is now trained! We have come to the exciting part where we try to make predictions with our model and see how well it predicts whether a tweet sounds happy or sad. Give it a try with your own input!<jupyter_code>def predict_tweet(text_to_predict):
    test_sequence = tokenizer.texts_to_sequences([text_to_predict])
    test_data = pad_sequences(test_sequence, maxlen=MAX_SEQUENCE_LENGTH)
    predictions = model.predict(test_data)
    prediction = np.argmax(predictions)
    print("The person who tweeted this is {}\n".format(classes_index[prediction]))
    ranking = predictions.argsort()[-6:][::-1]
    for i in reversed(range(0, 6)):
        rank = ranking[0][i]
        print("{:>10} : {:.5f}%".format(classes_index[rank], predictions[0][rank]*100))
predict_tweet("If I just disappear from the world right now, will anyone even care?")
predict_tweet("Nobody can understand the stress and anxiety that exists within me right now. Nobody cares.")
predict_tweet("When I'm training my neural network, I can't watch netflix because it lags like crazy. Sigh")
predict_tweet("Today seems like a fine day. I can't wait to go to school today.")
predict_tweet("Wow your scrambled eggs are delicious!")
predict_tweet("I am so excited for Avengers Endgame actually")
model.save("mood_detection_model.h5")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08867880303148684, 'fraction_numerical': 0.026252050941479803, 'mean_word_length': 4.939675174013921, 'pattern_counts': {'":': 0, '<': 68, '<?xml version=': 0, '>': 74, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '326228', 'license': 'MIT', 'n_tokens_mistral': 4280, 'n_tokens_neox': 3947, 'n_words': 1359, 'path': '3. Mood Detection of Tweets - Word Embeddings and LSTM.ipynb', 'repo_name': 'chrisgzf/tweet-sentiment-analysis', 'size': '12799'}