<jupyter_start><jupyter_text>Sentiment analysis: IMDB reviews Table of Contents1&nbsp;&nbsp;About this notebook2&nbsp;&nbsp;Sentiment Analysis3&nbsp;&nbsp;Data4&nbsp;&nbsp;Imports5&nbsp;&nbsp;TensorFlow information6&nbsp;&nbsp;Set Paths7&nbsp;&nbsp;Import data8&nbsp;&nbsp;Split train and test set9&nbsp;&nbsp;Preprocess data10&nbsp;&nbsp;NLP10.1&nbsp;&nbsp;Tokenize vocabulary10.2&nbsp;&nbsp;Describe data10.3&nbsp;&nbsp;Shuffle data10.4&nbsp;&nbsp;Create validation set10.5&nbsp;&nbsp;Convolutional NN10.6&nbsp;&nbsp;Train10.7&nbsp;&nbsp;Accuracy and Loss for the Model10.8&nbsp;&nbsp;Predict11&nbsp;&nbsp;Use case12&nbsp;&nbsp;Embeddings13&nbsp;&nbsp;Word cloud  About this notebookThis notebook allows you to classify movie reviews extracted from the [IMBD](https://www.imdb.com/) website. By means of embeddings, it also allows you to visualize how the different words cluster with each other.![](img/embedding.png) Sentiment AnalysisSentiment analysis is a *Natural Language Processing* technique used to determine the "sentiment" of a corpus of text (e.g., whether the opinion expressed is either positive or negative).The model presented in this notebook, provides the following accuracies:- Train accuracy: 95.23%- Test accuracy: 83.88% DataThe dataset used to train the present model, correpsond to the [IMDB reviews](https://www.tensorflow.org/datasets/catalog/imdb_reviews) dataset. It is composed by:- Training set: 25,000 examples.- Test set: 25,000 examples.The following figure consists of a *word cloud* (see the code to create it at the end of the notebook).![](img/wordcloud.png)  Imports<jupyter_code># General imports
import numpy as np ; np.random.seed(1) # for reproducibility
import os
import joblib
import io
import zipfile
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import json
from pathlib import Path
import random
import shutil
from datetime import datetime
from sklearn.model_selection import train_test_split
import platform
from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator
%matplotlib inline

# TensorFlow
import tensorflow as tf
from tensorflow import keras
import tensorflow_datasets as tfds
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorboard.plugins import projector<jupyter_output><empty_output><jupyter_text>TensorFlow information<jupyter_code># Indicate the version of Tensorflow and whether it uses the CPU or the GPU
print("TensorFlow version:", tf.__version__)

if len(tf.config.list_physical_devices('GPU')) > 0:
    print("The GPU will be used for calculations.")
    
else:
    print("The CPU will be used for calculations.")<jupyter_output>TensorFlow version: 2.4.0
The CPU will be used for calculations.<jupyter_text>Set Paths<jupyter_code># Define paths
PATH_DATA = Path('data')
PATH_MODEL = Path('model')
PATH_OUTPUT = Path('output')<jupyter_output><empty_output><jupyter_text>Import data<jupyter_code># Download data
imdb, info = tfds.load("imdb_reviews", with_info=True, as_supervised=True)<jupyter_output><empty_output><jupyter_text>Split train and test set<jupyter_code># Split train and test data
train_data, test_data = imdb['train'], imdb['test']

# Initialize datasets
training_sentences = []
training_labels = []

testing_sentences = []
testing_labels = []

# Extract sentences and labels from the training data
for s,l in train_data:
    training_sentences.append(str(s.numpy()))
    training_labels.append(l.numpy())

for s,l in test_data:
    testing_sentences.append(str(s.numpy()))
    testing_labels.append(l.numpy())<jupyter_output><empty_output><jupyter_text>Preprocess data<jupyter_code># Transform to numpy arrays
training_sentences = np.array(training_sentences)
testing_sentences = np.array(testing_sentences)
training_labels = np.array(training_labels)
testing_labels = np.array(testing_labels)

# Save data
joblib.dump(training_labels, os.path.join(PATH_DATA, 'training_labels'))
joblib.dump(testing_labels, os.path.join(PATH_DATA, 'testing_labels'))
joblib.dump(training_sentences, os.path.join(PATH_DATA, 'training_sentences'))
joblib.dump(testing_sentences, os.path.join(PATH_DATA, 'testing_sentences'))
# Load data
training_labels = joblib.load(os.path.join(PATH_DATA, 'training_labels'))
testing_labels = joblib.load(os.path.join(PATH_DATA, 'testing_labels'))
training_sentences = joblib.load(os.path.join(PATH_DATA, 'training_sentences'))
testing_sentences = joblib.load(os.path.join(PATH_DATA, 'testing_sentences'))<jupyter_output><empty_output><jupyter_text>NLP Tokenize vocabulary<jupyter_code># Define hyper-parameters
vocab_size = 20000
embedding_dim = 32
max_length = 240
trunc_type='post'
oov_tok = "<OOV>"

# Tokenize the words in the sentences
tokenizer = Tokenizer(
    num_words = vocab_size, 
    oov_token = oov_tok
)

# Update internal vocabulary based on list of texts
tokenizer.fit_on_texts(training_sentences)

# Generate index of words
word_index = tokenizer.word_index

# Tokenize training sentences
training_sequences = tokenizer.texts_to_sequences(training_sentences)

# Add padding
training_padded = pad_sequences(
    training_sequences,
    maxlen = max_length, 
    truncating = trunc_type
)

# Repeat for test data
testing_sequences = tokenizer.texts_to_sequences(testing_sentences)
testing_padded = pad_sequences(
    testing_sequences,
    maxlen = max_length,
)
# Save tokenizer
joblib.dump(tokenizer, os.path.join(PATH_MODEL, 'tokenizer'))<jupyter_output><empty_output><jupyter_text>Describe data<jupyter_code># Describe the data
print("\nDictionary length =", len(word_index))
print("\nNumber of training instances =" , training_padded.shape[0])
print("Number of testing instances =" , testing_padded.shape[0])
print("\nTraining instance max length =" , training_padded.shape[1])
print("Testing instance max length =" , testing_padded.shape[1])
print("\n- First instance:\n")
print(training_sentences[0])
print("\n- Padded sequence for the first instance:\n")
print(training_padded[0])
print("\n- Label for the first instance:\n")
if training_labels[0] == 0:
    print("0: BAD review\n")
else:
    print("1: GOOD review\n")<jupyter_output>Dictionary length = 86539

Number of training instances = 25000
Number of testing instances = 25000

Training instance max length = 240
Testing instance max length = 240

- First instance:

b"This was an absolutely terrible movie. Don't be lured in by <NAME> or <NAME>. Both are great actors, but this must simply be their worst role in history. Even their great acting could not redeem this movie's ridiculous storyline. This movie is an early nineties US propaganda piece. The most pathetic scenes were those when the Columbian rebels were making their cases for revolutions. <NAME> appeared phony, and her pseudo-love affair with Walken was nothing but a pathetic emotional plug in a movie that was devoid of any real meaning. I am disappointed that there are movies like this, ruining actor's like <NAME>'s good name. I could barely sit through it."

- Padded sequence for the first instance:

[    0     0     0     0     0     0     0     0    [...]<jupyter_text>Shuffle data<jupyter_code># Transform to numpy arrays
training_sequences = np.array(training_sequences, dtype="object")
testing_sequences = np.array(testing_sequences, dtype="object")

# Shuffle train data
permut = np.random.permutation(training_padded.shape[0])
training_sentences = training_sentences[permut]
training_labels = training_labels[permut]
training_sequences = training_sequences[permut]
training_padded = training_padded[permut]

# Shuffle test data
permut = np.random.permutation(testing_padded.shape[0])
testing_sentences = testing_sentences[permut]
testing_labels = testing_labels[permut]
testing_sequences = testing_sequences[permut]
testing_padded = testing_padded[permut]<jupyter_output><empty_output><jupyter_text>Create validation set<jupyter_code># Create validation set
train_padded, validation_padded, train_labels, validation_labels = train_test_split(
    training_padded, 
    training_labels, 
    test_size=0.2, 
    random_state=42
)
# Describe data dimensions
print('\nTraining dimensions:', train_padded.shape)
print('Training labels size:', train_labels.shape[0])
print('\nValidation dimensions:', validation_padded.shape)
print('Validation labels size:', validation_labels.shape[0])
print('\nTest dimensions:', testing_padded.shape)
print('Test labels size:', testing_labels.shape[0])<jupyter_output>Training dimensions: (20000, 240)
Training labels size: 20000

Validation dimensions: (5000, 240)
Validation labels size: 5000

Test dimensions: (25000, 240)
Test labels size: 25000<jupyter_text>Convolutional NN<jupyter_code># Create CNN architecture
model = tf.keras.Sequential([
    tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(32)),
    tf.keras.layers.Dense(6, activation='relu'),
    tf.keras.layers.Dropout(0.30),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

# Print summary
model.summary()
# Compile model
model.compile(
    optimizer='adam', 
    loss='binary_crossentropy', 
    metrics=['accuracy']
)<jupyter_output><empty_output><jupyter_text>Train<jupyter_code># Set seed for random functions
tf.random.set_seed(1)

# Fit the model 
history = model.fit(
    x=train_padded,
    y=train_labels,
    batch_size=32,
    epochs=5,
    verbose=2,
    validation_data=(validation_padded, validation_labels)
)

# Get the dictionary containing each metric and the loss for each epoch
history_dict = history.history

# Save model and history
model.save(os.path.join(PATH_MODEL, 'imdb_model.h5'))
model.save_weights(os.path.join(PATH_MODEL, 'imdb_weights.h5'))
joblib.dump(history_dict, os.path.join(PATH_MODEL, 'imdb_history'))
    
# Load model and history
model = keras.models.load_model(os.path.join(PATH_MODEL, 'imdb_model.h5'))
model.load_weights(os.path.join(PATH_MODEL, 'imdb_weights.h5'))
history_dict = joblib.load(os.path.join(PATH_MODEL, 'imdb_history'))<jupyter_output><empty_output><jupyter_text>Accuracy and Loss for the Model<jupyter_code># Get metrics on training and test data
acc = history_dict['accuracy']
val_acc = history_dict['val_accuracy']
loss = history_dict['loss']
val_loss = history_dict['val_loss']

# Get number of epochs
epochs = range(1, len(acc)+1) 

# Set up plot
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 4));

# Plot training and validation accuracy per epoch
ax1.plot(epochs, acc, label='Training accuracy');
ax1.plot(epochs, val_acc, label='Validation accuracy');
ax1.set_title('Training and validation accuracy');
ax1.legend(loc=0);
ax1.set(xlabel='Epochs', ylabel='Accuracy');

# Plot training and validation loss per epoch
ax2.plot(epochs, loss, label='Training loss');
ax2.plot(epochs, val_loss, label='Validation loss');
ax2.set_title('Training and validation loss');
ax2.legend(loc=0);
ax2.set(xlabel='Epochs', ylabel='Loss');

# Save plot
plt.savefig('app/assets/acc.png', dpi=600, facecolor='w', edgecolor='w',);<jupyter_output><empty_output><jupyter_text>Predict<jupyter_code># Evaluate final accuracy of the model
train_scores = model.evaluate(training_padded, training_labels, verbose=0)
print("- Train accuracy: " + str(round(train_scores[1]*100, 2)) + '%')
test_scores = model.evaluate(testing_padded, testing_labels, verbose=0)
print("- Test accuracy: " + str(round(test_scores[1]*100, 2)) + '%')
# Print 5 test examples
test_set = testing_sentences[0:5]

# Tokenize
sequences = tokenizer.texts_to_sequences(test_set)

# Add padding
padded = pad_sequences(sequences, maxlen=max_length, truncating=trunc_type)

# Predict
predictions = model.predict(padded)

# Print results
for i in range(len(test_set)):
    print(test_set[i])
    if testing_labels[i] == 0:
        print("This is a BAD review")
    else:
        print("This is a GOOD review")
    if predictions[i] < 0.5:
        print("Prediction: BAD review\n")
    else:
        print("Prediction: GOOD review\n")<jupyter_output>b"Reeves plays <NAME>, a hero in 1850's Russia.<br /><br />This is a badly dubbed movie, with June Foray doing some of the voices. Unfortunately who ever was suppose to sync the voices to the lips was blind since the words never match the lip flaps. Anyone who says that Japanese films are bad have never watched this film.<br /><br />The film's plot is instantly forgettable and so I've forgotten it in the time its taken the movie to end and for me to sit and write this down. Perhaps it has more to do with the fact that the film is one of the many that Reeves made in hopes of moving away from action to more plot driven sort of films. It may have been a good thing for Reeves, but its deadly for the audience who have to slug through nooze fests such as this, where its all court intrigue with very little action.<br /><br />In Reeves defense, he was a good actor, he just had no real luck in picking films that were any good. They all looked great, but very few of them didn't put the those[...]<jupyter_text>Use case<jupyter_code># Create new instances
list_sentences = [
    "What a great movie", 
    "This show is garbage",
    "The best I've ever seen",
    "Awesome performance",
    "An incredible and remarkable disaster",
    "Incredible show"
]

# Tokenize
sequences = tokenizer.texts_to_sequences(list_sentences)

# Add padding
padded = pad_sequences(sequences, maxlen=max_length, truncating=trunc_type)

# Predict
predictions = model.predict(padded)

# Print results
for i in range(len(list_sentences)):
    print(list_sentences[i])
    if predictions[i] < 0.5:
        print("Prediction: BAD review\n")
    else:
        print("Prediction: GOOD review\n")<jupyter_output>What a great movie
Prediction: GOOD review

This show is garbage
Prediction: BAD review

The best I've ever seen
Prediction: GOOD review

Awesome performance
Prediction: GOOD review

An incredible and remarkable disaster
Prediction: GOOD review

Incredible show
Prediction: GOOD review<jupyter_text>Embeddings<jupyter_code># Embedding shape
embedding = model.layers[0]
weights = embedding.get_weights()[0] 
print('Embedding shape:', weights.shape) 
# Revert dictionary
reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])

# Save vectors and labels
out_v = io.open(os.path.join(PATH_OUTPUT, 'vectors.tsv'), 'w', encoding='utf-8')
out_m = io.open(os.path.join(PATH_OUTPUT, 'metadata.tsv'), 'w', encoding='utf-8')
labels = []
for word_num in range(1, vocab_size):
    word = reverse_word_index[word_num]
    embeddings = weights[word_num]
    labels.append(word)
    out_m.write(word + "\n")
    out_v.write('\t'.join([str(x) for x in embeddings]) + "\n")
out_v.close()
out_m.close()<jupyter_output><empty_output><jupyter_text>We can visualize the emebddings in this Tensorflow's [page](https://projector.tensorflow.org/?config=https://gist.githubusercontent.com/guillermo-lahuerta/6185a0ed9d82bf371a984cf7c2ec8547/raw/688afac9a363f872036640cf6e8ddf2fa036c576/config.json).<jupyter_code>%%html
<iframe src="https://projector.tensorflow.org/?config=https://gist.githubusercontent.com/guillermo-lahuerta/6185a0ed9d82bf371a984cf7c2ec8547/raw/688afac9a363f872036640cf6e8ddf2fa036c576/config.json" width="1100" height="600"></iframe><jupyter_output><empty_output><jupyter_text>Word cloud<jupyter_code># Remove irrelevant words
stopwords = set(STOPWORDS)
stopwords.update(["one", "film", "movie", "show"])

# Iterate over each label
comment_words = ' '
for val in labels: 
    val = str(val) 
    tokens = val.split() 
    for i in range(len(tokens)): 
        tokens[i] = tokens[i].lower() 
    for words in tokens: 
        comment_words = comment_words + words + ' '

# Create wordcloud
wordcloud = WordCloud(width = 5000, height = 4000, 
                background_color = 'white', 
                stopwords = stopwords, 
                min_font_size = 10).generate(comment_words) 

# Plot wordcloud
plt.figure(figsize = (12, 12));
plt.imshow(wordcloud);
plt.axis("off");
plt.show();

# Save image
wordcloud.to_file('img/wordcloud.png');
wordcloud.to_file('app/assets/wordcloud.png');<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10789897239073913, 'fraction_numerical': 0.020985514423672157, 'mean_word_length': 5.85405176071277, 'pattern_counts': {'":': 0, '<': 84, '<?xml version=': 0, '>': 83, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '348574', 'license': 'MIT', 'n_tokens_mistral': 5429, 'n_tokens_neox': 5114, 'n_words': 1399, 'path': 'Sentiment_analysis_IMDB.ipynb', 'repo_name': 'guillermo-lahuerta/Sentiment_analysis', 'size': '16207'}