<jupyter_start><jupyter_text>To create our embeddings, we’ll first use an embeddings layer, called `tf.keras.layers.Embedding`. This takes three arguments: the size of the tokenized vocabulary, the number of embedding dimensions to use, as well as the input length. The output of this layer needs to be reshaped to work with any fully-connected layers. Thus, `Flatten` or `GlobalAveragePooling1D` function is used.  **Import libraries**<jupyter_code>import tensorflow as tf
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>**Read and split dataset**<jupyter_code>dataset = pd.read_csv('combined_data.csv')

sentences = dataset['text'].tolist()
labels = dataset['sentiment'].tolist()

# Separate out the sentences and labels into training and test sets
training_size = int(len(sentences) * 0.8)

training_sentences = sentences[0:training_size]
testing_sentences = sentences[training_size:]
training_labels = labels[0:training_size]
testing_labels = labels[training_size:]

# Make labels into numpy arrays for use with the network later
training_labels_final = np.array(training_labels)
testing_labels_final = np.array(testing_labels)
dataset
training_sentences
training_labels_final<jupyter_output><empty_output><jupyter_text>**Tokenize**<jupyter_code>vocab_size = 1000
embedding_dim = 16
max_length = 100
trunc_type='post'
padding_type='post'
oov_tok = "<OOV>"

# Create model
tokenizer = Tokenizer(num_words = vocab_size, oov_token=oov_tok)
tokenizer.fit_on_texts(training_sentences)
word_index = tokenizer.word_index

# Create sequences for the train data and padding
sequences = tokenizer.texts_to_sequences(training_sentences)
padded = pad_sequences(sequences,maxlen=max_length, padding=padding_type, 
                       truncating=trunc_type)

# Create sequences for the train data and padding
testing_sequences = tokenizer.texts_to_sequences(testing_sentences)
testing_padded = pad_sequences(testing_sequences,maxlen=max_length, 
                               padding=padding_type, truncating=trunc_type)
print(padded.shape)
print(padded)
reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])
reverse_word_index
def decode_review(text):
    return ' '.join([reverse_word_index.get(i, '?') for i in text])

print(decode_review(padded[5]))
print(len(decode_review(padded[5])))
print(training_sentences[5])<jupyter_output>i have to <OOV> the plug to get it to line up right to get decent volume ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
238
I have to jiggle the plug to get it to line up right to get decent volume.<jupyter_text>**Train model**<jupyter_code># Build a basic sentiment network
# Note the embedding layer is first, 
# and the output is only 1 node as it is either 0 or 1 (negative or positive)
model = tf.keras.Sequential([
    tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(6, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
])
model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])
model.summary()<jupyter_output>Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding (Embedding)        (None, 100, 16)           16000     
_________________________________________________________________
flatten (Flatten)            (None, 1600)              0         
_________________________________________________________________
dense (Dense)                (None, 6)                 9606      
_________________________________________________________________
dense_1 (Dense)              (None, 1)                 7         
=================================================================
Total params: 25,613
Trainable params: 25,613
Non-trainable params: 0
_________________________________________________________________<jupyter_text>The TensorFlow team has suggest that the final network does not use a sigmoid activation layer when working with embeddings, especially when using just the two classes for sentiment analysis<jupyter_code>num_epochs = 50
model.fit(padded, training_labels_final, epochs=num_epochs, validation_data=(testing_padded, testing_labels_final))<jupyter_output>Epoch 1/50
50/50 [==============================] - 0s 6ms/step - loss: 0.6927 - accuracy: 0.5122 - val_loss: 0.7026 - val_accuracy: 0.4110
Epoch 2/50
50/50 [==============================] - 0s 4ms/step - loss: 0.6909 - accuracy: 0.5229 - val_loss: 0.7045 - val_accuracy: 0.4110
Epoch 3/50
50/50 [==============================] - 0s 3ms/step - loss: 0.6859 - accuracy: 0.5229 - val_loss: 0.6949 - val_accuracy: 0.4110
Epoch 4/50
50/50 [==============================] - 0s 4ms/step - loss: 0.6694 - accuracy: 0.5273 - val_loss: 0.6838 - val_accuracy: 0.4887
Epoch 5/50
50/50 [==============================] - 0s 4ms/step - loss: 0.6218 - accuracy: 0.6560 - val_loss: 0.6293 - val_accuracy: 0.7368
Epoch 6/50
50/50 [==============================] - 0s 4ms/step - loss: 0.4891 - accuracy: 0.8669 - val_loss: 0.5294 - val_accuracy: 0.7794
Epoch 7/50
50/50 [==============================] - 0s 4ms/step - loss: 0.3536 - accuracy: 0.8964 - val_loss: 0.4809 - val_accuracy: 0.7920
Epoch 8/50
50/50 [==[...]<jupyter_text>**visualizing the network**<jupyter_code>import io

# First get the weights of the embedding layer
e = model.layers[0]
weights = e.get_weights()[0]
print(weights.shape) # shape: (vocab_size, embedding_dim)
# Write out the embedding vectors and metadata
out_v = io.open('vecs.tsv', 'w', encoding='utf-8')
out_m = io.open('meta.tsv', 'w', encoding='utf-8')
for word_num in range(1, vocab_size):
  word = reverse_word_index[word_num]
  embeddings = weights[word_num]
  out_m.write(word + "\n")
  out_v.write('\t'.join([str(x) for x in embeddings]) + "\n")
out_v.close()
out_m.close()<jupyter_output><empty_output><jupyter_text>We can visualizing the network "sees" the sentiment related to each word. Head to http://projector.tensorflow.org/ and load these files.<jupyter_code># Download the files
try:
  from google.colab import files
except ImportError:
  pass
else:
  files.download('vecs.tsv')
  files.download('meta.tsv')
# Use the model to predict a review   
other_reviews = ['The weather is good today', 
                'I have a car model bugatti', 
                'I hate negative thinking',
                'Time travel machine is very exciting invention', 
                'Happy and sad is emotional status', 
                'The Earth is third planet from the Sun',
                'It is annoying', 
                'Neutral is consider positive or not',
                'The social media connect many people but causing other problem', 
                'The natural world is amazing']
other_reviews
# Create the sequences
padding_type='post'
sample_sequences = tokenizer.texts_to_sequences(other_reviews)
other_padded = pad_sequences(sample_sequences, padding=padding_type, maxlen=max_length)

print(len(other_padded))
print(other_padded)
classes = model.predict(other_padded)
print(len(classes))
classes
# The closer the class is to 1, the more positive the review is deemed to be
for x in range(len(other_reviews)):
  print(other_reviews[x])
  print(classes[x])
  print('\n')<jupyter_output>The weather is good today
[0.9969247]


I have a car model bugatti
[0.11933988]


I hate negative thinking
[0.00019339]


Time travel machine is very exciting invention
[0.81559587]


Happy and sad is emotional status
[0.99962616]


The Earth is third planet from the Sun
[0.13325784]


It is annoying
[0.47141936]


Neutral is consider positive or not
[0.00524831]


The social media connect many people but causing other problem
[0.2009104]


The natural world is amazing
[0.99917704]<jupyter_text>**Tweaking the Model**Based on the result obtained above, it seem like the model still has a lot of improvement to be made. We can use the same model but different value for the parameter.<jupyter_code># previously is 1000
vocab_size = 500

embedding_dim = 16

# previously use 100
max_length = 50

trunc_type='post'
padding_type='post'
oov_tok = "<OOV>"

tokenizer = Tokenizer(num_words = vocab_size, oov_token=oov_tok)
tokenizer.fit_on_texts(training_sentences)
word_index = tokenizer.word_index
training_sequences = tokenizer.texts_to_sequences(training_sentences)
training_padded = pad_sequences(training_sequences, maxlen=max_length, padding=padding_type, truncating=trunc_type)

testing_sequences = tokenizer.texts_to_sequences(testing_sentences)
testing_padded = pad_sequences(testing_sequences, maxlen=max_length, padding=padding_type, truncating=trunc_type)
model = tf.keras.Sequential([
    tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length),

    # previously using Flatten()
    tf.keras.layers.GlobalAveragePooling1D(),
    tf.keras.layers.Dense(6, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
])
model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])
model.summary()
num_epochs = 50
history = model.fit(training_padded, training_labels_final, epochs=num_epochs, validation_data=(testing_padded, testing_labels_final))
import matplotlib.pyplot as plt

def plot_graphs(history, string):
  plt.plot(history.history[string])
  plt.plot(history.history['val_'+string])
  plt.xlabel("Epochs")
  plt.ylabel(string)
  plt.legend([string, 'val_'+string])
  plt.show()
  
plot_graphs(history, "accuracy")
plot_graphs(history, "loss")
e = model.layers[0]
weights = e.get_weights()[0]
print(weights.shape) # shape: (vocab_size, embedding_dim)
reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])
out_v = io.open('vecs_2.tsv', 'w', encoding='utf-8')
out_m = io.open('meta_2.tsv', 'w', encoding='utf-8')
for word_num in range(1, vocab_size):
  word = reverse_word_index[word_num]
  embeddings = weights[word_num]
  out_m.write(word + "\n")
  out_v.write('\t'.join([str(x) for x in embeddings]) + "\n")
out_v.close()
out_m.close()

try:
  from google.colab import files
except ImportError:
  pass
else:
  files.download('vecs_2.tsv')
  files.download('meta_2.tsv')<jupyter_output><empty_output><jupyter_text>Now we can check the sentiment analysis result for the same `other_reviews`.<jupyter_code>other_sample_sequences = tokenizer.texts_to_sequences(other_reviews)
second_other_padded = pad_sequences(other_sample_sequences, padding=padding_type, maxlen=max_length)

new_classes = model.predict(second_other_padded)

for x in range(len(other_reviews)):
  print(other_reviews[x])
  print(new_classes[x])
  print('\n')<jupyter_output>The weather is good today
[0.9850309]


I have a car model bugatti
[0.7767314]


I hate negative thinking
[0.00864255]


Time travel machine is very exciting invention
[0.86802965]


Happy and sad is emotional status
[0.9911617]


The Earth is third planet from the Sun
[0.47298124]


It is annoying
[0.5221584]


Neutral is consider positive or not
[0.01714928]


The social media connect many people but causing other problem
[0.01414215]


The natural world is amazing
[0.9994548]
{'fraction_non_alphanumeric': 0.13486670151594354, 'fraction_numerical': 0.041644885868618225, 'mean_word_length': 4.895737031330252, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '242912', 'license': 'MIT', 'n_tokens_mistral': 4032, 'n_tokens_neox': 3632, 'n_words': 956, 'path': 'Text_Sentiment_Analysis/Basic_tf-tokernizer_layer.ipynb', 'repo_name': 'ee2110/Natural_Language_Processing-NLP-TensorFlow', 'size': '11478'}