<jupyter_start><jupyter_text>Using Pre-trained Word embeddings In this lab exercise, we will use a pretrained word embedding for our text classification task, instead of training our own embedding layer.   Setup<jupyter_code>import io
import os
import re
import shutil
import string
import tensorflow as tf
import numpy as np

from datetime import datetime
import tensorflow as tf
from tensorflow.keras import Model, Sequential
from tensorflow.keras.layers import Activation, Dense, Embedding, GlobalAveragePooling1D, Flatten
from tensorflow.keras.layers.experimental.preprocessing import TextVectorization<jupyter_output><empty_output><jupyter_text>Download the IMDb DatasetDownload the dataset using Keras file utility and do the clean-up as before<jupyter_code>url = "https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz"

dataset = tf.keras.utils.get_file("aclImdb_v1.tar.gz", url,
                                    untar=True, cache_dir='.',
                                    cache_subdir='')

dataset_dir = os.path.join(os.path.dirname(dataset), 'aclImdb')
train_dir = os.path.join(dataset_dir, 'train')
remove_dir = os.path.join(train_dir, 'unsup')
shutil.rmtree(remove_dir)<jupyter_output><empty_output><jupyter_text>Next, create a `tf.data.Dataset` using `tf.keras.preprocessing.text_dataset_from_directory`. Use the `train` directory to create both train and validation datasets with a split of 20% for validation.<jupyter_code>batch_size = 1024
seed = 123
train_ds = tf.keras.preprocessing.text_dataset_from_directory(
    'aclImdb/train', batch_size=batch_size, validation_split=0.2, 
    subset='training', seed=seed)
val_ds = tf.keras.preprocessing.text_dataset_from_directory(
    'aclImdb/train', batch_size=batch_size, validation_split=0.2, 
    subset='validation', seed=seed)
AUTOTUNE = tf.data.AUTOTUNE

train_ds = train_ds.cache().prefetch(buffer_size=AUTOTUNE)
val_ds = val_ds.cache().prefetch(buffer_size=AUTOTUNE)<jupyter_output><empty_output><jupyter_text>Text preprocessing We then initialize a TextVectorization layer with the desired parameters to vectorize our movie reviews.<jupyter_code># Vocabulary size and number of words in a sequence.
vocab_size = 10000
sequence_length = 100

# Use the text vectorization layer to normalize, split, and map strings to 
# integers. Note that the layer uses the custom standardization defined above. 
# Set maximum_sequence length as all samples are not of the same length.
vectorize_layer = TextVectorization(
    max_tokens=vocab_size,
    output_mode='int',
    output_sequence_length=sequence_length)

# Make a text-only dataset (no labels) and call adapt to build the vocabulary.
text_ds = train_ds.map(lambda x, y: x)
vectorize_layer.adapt(text_ds)<jupyter_output><empty_output><jupyter_text>Getting the pre-trained embedding modelWe will use the pre-trained GloVe embeddings available from [stanford site](https://nlp.stanford.edu/projects/glove/). The original zip file contains embedding of different dimensions (e.g. 50d, 100d, 200d, etc) and is more than 800MB in file size. To save downloading time, we have made a copy of 50d GloVe file available on our course website for download. If you want to experiment with other embedding dimensions, please feel free to download from the stanford site.<jupyter_code>glove_url = 'https://nyp-aicourse.s3-ap-southeast-1.amazonaws.com/pretrained-models/glove.6B.50d.zip'

glove_files = tf.keras.utils.get_file("glove.6B.50d.zip", glove_url,
                                    extract=True, cache_dir='.',
                                    cache_subdir='')<jupyter_output><empty_output><jupyter_text>Load the Embedding layerIf you examined the extracted files, you will notice filenames like glove.6B.100d, glove.6B.50d, etc.  The 100d and 50d represents the different embedding dimensions.  We will try the 50d embedding first.In the code below, we read the embeddings from the download file line by line to create the embeddings index and then initialize the Keras embedding layer with this embeddings index.<jupyter_code># Load up the GloVe word embedding data

embedding_dim = 50

print("Loading GloVe Word Embedding...")
embeddings_index = {}
with open('glove.6B.50d.txt', encoding="utf8") as f:
    for line in f:
        values = line.split()
        word = values[0]
        coefs = np.asarray(values[1:], dtype='float32')
        embeddings_index[word] = coefs
    f.close()<jupyter_output><empty_output><jupyter_text>Let's print out the embedding for the word `happy`.<jupyter_code>embeddings_index['happy']
# Construct the word embedding matrix that will be used in the 
# Embedding layer.

vocab = vectorize_layer.get_vocabulary()
glove_embedding_matrix = np.zeros((len(vocab), embedding_dim))
for i, word in enumerate(vocab):
    embedding_vector = embeddings_index.get(word)
    if embedding_vector is not None:
        glove_embedding_matrix[i] = embedding_vector
vocab_size = len(vocab)
embedding_layer = Embedding(vocab_size, 
                            embedding_dim, 
                            input_length = sequence_length,
                            weights=[glove_embedding_matrix],  
                            trainable=False)<jupyter_output><empty_output><jupyter_text>Create a classification modelWe will now create the model as before, but this time with embedding layer initialized with pretrained embedding.<jupyter_code>model = Sequential([
    vectorize_layer,
    embedding_layer,
    GlobalAveragePooling1D(),
    Dense(16, activation='relu'),
    Dense(8, activation='relu'),
    Dense(1, activation='sigmoid')
])<jupyter_output><empty_output><jupyter_text>Compile and train the model<jupyter_code>root_logdir = os.path.join(os.curdir, "tb_logs")

def get_run_logdir():    # use a new directory for each run
	import time
	run_id = time.strftime("run_%Y_%m_%d-%H_%M_%S")
	return os.path.join(root_logdir, run_id)

run_logdir = get_run_logdir()
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=run_logdir)<jupyter_output><empty_output><jupyter_text>Compile and train the model using the `Adam` optimizer and `BinaryCrossentropy` loss.<jupyter_code>model.compile(optimizer='adam',
              loss=tf.keras.losses.BinaryCrossentropy(from_logits=False),
              metrics=['accuracy'])
model.fit(
    train_ds,
    validation_data=val_ds, 
    epochs=15,
    callbacks=[tensorboard_callback])
%load_ext tensorboard
%tensorboard --logdir tb_logs<jupyter_output><empty_output><jupyter_text>With pretrained embedding layer, we reaches a validation accuracy of around 70%, worse than jointly train our embedding layer with the classification task.If we have enough data (like in our case), joinly train our own embedding layer will usually yield a better performance. Let's evaluate the model on our test dataset.<jupyter_code>test_ds = tf.keras.preprocessing.text_dataset_from_directory(
    'aclImdb/test', 
    batch_size=batch_size)
model.evaluate(test_ds)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08684436801375753, 'fraction_numerical': 0.01117798796216681, 'mean_word_length': 4.619162640901771, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '277408', 'license': 'Apache-2.0', 'n_tokens_mistral': 2230, 'n_tokens_neox': 2130, 'n_words': 595, 'path': 'week7/word_embeddings_glove.ipynb', 'repo_name': 'nyp-sit/it3103-tutors', 'size': '6978'}