<jupyter_start><jupyter_text>Data Science 2: Advanced Topics in Data Science  Section 5: Natural Language Processing**Harvard University****Spring 2021****Instructors**: <NAME>, <NAME>, and <NAME> **Authors**: <NAME><jupyter_code>## RUN THIS CELL TO PROPERLY HIGHLIGHT THE EXERCISES
import requests
from IPython.core.display import HTML
styles = requests.get("https://raw.githubusercontent.com/Harvard-IACS/2019-CS109B/master/content/styles/cs109.css").text
HTML(styles)<jupyter_output><empty_output><jupyter_text>Learning ObjectivesBy the end of this section, you should be able to:* Perform machine translation task to understand the following model architectures:  * Sequence2Sequence model  * Sequence2Sequence with Attention model  * Transformer* Familiarize with key concepts in `Tensorflow` such as:  * Building custom `Layers`  * Custom training loops using `GradientTape`  * Custom `loss` functions  * Adding learning rate schedulers in your `optimizer`  * View model predictions during training using custom `callbacks`* Build a Masked Language model using Mini BERT  * Familiarize with BERT(Bidirectional Encoder Representations from Transformers) architecture  * Undertand the postional encoding in transfomer based models  Notebook Contents- [**Neural Machine Translation**](nmt)    - [Overview](nmt)     - [Sequence 2 Sequence](nmt)    - [Sequence 2 Sequence with Attention](nmt)    - [Transfomer](nmt)- [**Mini BERT Language Model**](bert)    - [Overview](bert)     - [Data Generation for Language Model](bert)    - [Build Mini BERT](bert)    - [Break Out Room 🎊🎉](breakout)    - [Training with / without Postional Encoding](bert)- [**References**](references)  **Setup Notebook** **Imports**<jupyter_code>import os
import requests
import zipfile
import tarfile
import shutil
import math
import json
import time
import sys
import string
import re
import numpy as np
import pandas as pd
from glob import glob
import collections
import unicodedata
import seaborn as sns
import matplotlib.pyplot as plt
import matplotlib.cm as cm
%matplotlib inline

# NLTK
import nltk
from nltk.tokenize import word_tokenize, sent_tokenize
from nltk.translate.bleu_score import corpus_bleu, SmoothingFunction

# Tensorflow
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.models import Model, Sequential
from tensorflow.keras.layers.experimental.preprocessing import TextVectorization
from tensorflow.keras.utils import to_categorical
from tensorflow.python.keras import backend as K
from tensorflow.python.keras.utils.layer_utils import count_params

# sklearn
from sklearn.model_selection import train_test_split
# download nltk's punkt sentence tokenizer
nltk.download('punkt')<jupyter_output>[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Unzipping tokenizers/punkt.zip.<jupyter_text>**Verify Setup**<jupyter_code># Enable/Disable Eager Execution
# Reference: https://www.tensorflow.org/guide/eager
# TensorFlow's eager execution is an imperative programming environment that evaluates operations immediately, 
# without building graphs

#tf.compat.v1.disable_eager_execution()
#tf.compat.v1.enable_eager_execution()

print("tensorflow version", tf.__version__)
print("keras version", tf.keras.__version__)
print("Eager Execution Enabled:", tf.executing_eagerly())

# Get the number of replicas 
strategy = tf.distribute.MirroredStrategy()
print("Number of replicas:", strategy.num_replicas_in_sync)

devices = tf.config.experimental.get_visible_devices()
print("Devices:", devices)
print(tf.config.experimental.list_logical_devices('GPU'))

print("GPU Available: ", tf.config.list_physical_devices('GPU'))
print("All Physical Devices", tf.config.list_physical_devices())

# Better performance with the tf.data API
# Reference: https://www.tensorflow.org/guide/data_performance
AUTOTUNE = tf.data.experimental.AUTOTUNE<jupyter_output>tensorflow version 2.4.1
keras version 2.4.0
Eager Execution Enabled: True
WARNING:tensorflow:There are non-GPU devices in `tf.distribute.Strategy`, not using nccl allreduce.
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',)
Number of replicas: 1
Devices: [PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU')]
[]
GPU Available:  []
All Physical Devices [PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU')]<jupyter_text>**Utils**Here we define some helper functions that would be used in loading data, saving models and evaluating models.<jupyter_code>def download_file(packet_url, base_path="", extract=False, headers=None):
  if base_path != "":
    if not os.path.exists(base_path):
      os.mkdir(base_path)
  packet_file = os.path.basename(packet_url)
  with requests.get(packet_url, stream=True, headers=headers) as r:
      r.raise_for_status()
      with open(os.path.join(base_path,packet_file), 'wb') as f:
          for chunk in r.iter_content(chunk_size=8192):
              f.write(chunk)
  
  if extract:
    if packet_file.endswith(".zip"):
      with zipfile.ZipFile(os.path.join(base_path,packet_file)) as zfile:
        zfile.extractall(base_path)
    else:
      packet_name = packet_file.split('.')[0]
      with tarfile.open(os.path.join(base_path,packet_file)) as tfile:
        tfile.extractall(base_path)

class JsonEncoder(json.JSONEncoder):
  def default(self, obj):
    if isinstance(obj, np.integer):
        return int(obj)
    elif isinstance(obj, np.floating):
        return float(obj)
    elif isinstance(obj, decimal.Decimal):
        return float(obj)
    elif isinstance(obj, np.ndarray):
        return obj.tolist()
    else:
        return super(JsonEncoder, self).default(obj)

def compute_blue_scores(model, inputs, outputs, translate_fn):
  actual = []
  predicted = []
  for idx,test_text in enumerate(inputs):
    actual_op = outputs[idx]
    
    source_text, output_text = translate_fn(model,test_source_text=test_text)
    actual.append([actual_op.split()])
    predicted.append(output_text.split())
    
  smooth = SmoothingFunction().method4
  blue_scores = {
      "BLEU-1": corpus_bleu(actual, predicted,smoothing_function=smooth, weights=(1.0, 0, 0, 0)),
      "BLEU-2": corpus_bleu(actual, predicted,smoothing_function=smooth, weights=(0.5, 0.5, 0, 0)),
      "BLEU-3": corpus_bleu(actual, predicted,smoothing_function=smooth, weights=(0.3, 0.3, 0.3, 0)),
      "BLEU-4": corpus_bleu(actual, predicted,smoothing_function=smooth, weights=(0.25, 0.25, 0.25, 0.25))
  }

  return blue_scores

def evaluate_save_model(model,test_in, test_out, translate_fn, training_results,execution_time, learning_rate, epochs):
    
  # Get the model train history
  model_train_history = training_results.history
  # Get the number of epochs the training was run for
  num_epochs = len(model_train_history["loss"])

  # Plot training results
  fig = plt.figure(figsize=(20,5))
  axs = fig.add_subplot(1,3,1)
  axs.set_title('Loss')
  # Plot all metrics
  for metric in ["loss"]:
      axs.plot(np.arange(0, num_epochs), model_train_history[metric], label=metric)
  axs.legend()

  plt.show()
  
  # Evaluate on test data
  blue_scores = compute_blue_scores(model,test_in, test_out, translate_fn)
  print(blue_scores)

  # Ensure path exists
  if not os.path.exists("models"):
    os.mkdir("models")
  
  # Save model history
  with open(os.path.join("models",model.name+"_train_history.json"), "w") as json_file:
      json_file.write(json.dumps(model_train_history,cls=JsonEncoder))

  trainable_parameters = count_params(model.trainable_weights)
  non_trainable_parameters = count_params(model.non_trainable_weights)

  # Save model metrics
  metrics ={
      "trainable_parameters":trainable_parameters,
      "execution_time":execution_time,
      "loss":model_train_history["loss"][-1],
      "BLEU-1": blue_scores["BLEU-1"],
      "BLEU-2": blue_scores["BLEU-2"],
      "BLEU-3": blue_scores["BLEU-3"],
      "BLEU-4": blue_scores["BLEU-4"],
      "learning_rate":learning_rate,
      "epochs":epochs,
      "name": model.name,
      "id": int(time.time())
  }
  with open(os.path.join("models",model.name+"_metrics.json"), "w") as json_file:
      json_file.write(json.dumps(metrics,cls=JsonEncoder))<jupyter_output><empty_output><jupyter_text>---  **Neural Machine Translation** ---  **Overview**In this section, we're going to perform machine translation from english to french using an encoder decoder architecture. We will explore various model architectures.**The Task:** Perform Neural Machine Translation from english to french.  **Dataset**The dataset consists of English to French translations from http://www.manythings.org/   **Download**Download the datasets to colab.<jupyter_code>start_time = time.time()
headers = {
  'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'
}
download_file("http://www.manythings.org/anki/fra-eng.zip", base_path="datasets", extract=True, headers=headers)
execution_time = (time.time() - start_time)/60.0
print("Download execution time (mins)",execution_time)<jupyter_output>Download execution time (mins) 0.013792125384012859<jupyter_text>**Load Data*** Read-in data as dataframe.* Data in column`english` is our inputs* Data in column`french` is our outputs<jupyter_code># Read the fra-eng raw data 
english_french_data = pd.read_csv('datasets/fra.txt',delimiter='\t',header=None, names=["english","french","att"])
english_french_data = english_french_data[["english","french"]]
print("Size",english_french_data.shape)
english_french_data.head()
subset_data = False
if subset_data:
  english_french_data = english_french_data[:50000]

# Read in the english and french texts
data_en = english_french_data["english"].values
data_fr = english_french_data["french"].values

def unicode_to_ascii(s):
  return ''.join(c for c in unicodedata.normalize('NFD', s)if unicodedata.category(c) != 'Mn')

def preprocess_string(s):
  s = unicode_to_ascii(s)
  s = re.sub(r'([!.?])', r' \1', s)
  s = re.sub(r'[^a-zA-Z.!?]+', r' ', s)
  s = re.sub(r'\s+', r' ', s)
  return s

data_en = [preprocess_string(data) for data in data_en]
# Add start tag for decoder input
data_fr_in = ['<start> ' + preprocess_string(data) for data in data_fr]
# Add end tag for decoder output
data_fr_out = [preprocess_string(data) + ' <end>' for data in data_fr]

print("data_en len:",len(data_en))
print("data_fr_in len:",len(data_fr_in))
print("data_fr_out len:",len(data_fr_out))<jupyter_output>data_en len: 185583
data_fr_in len: 185583
data_fr_out len: 185583<jupyter_text>**View Text**Let's take a look at the data.<jupyter_code># Generate a random sample of index
data_samples = np.random.randint(0,high=len(data_en)-1, size=6)
for i,data_idx in enumerate(data_samples):
  print("English:",data_en[data_idx])
  print("French (input):",data_fr_in[data_idx])
  print("French (output):",data_fr_out[data_idx])
  print("\n")<jupyter_output>English: They found us .
French (input): <start> Elles nous ont trouves .
French (output): Elles nous ont trouves . <end>


English: I don t compromise .
French (input): <start> Je ne transige pas .
French (output): Je ne transige pas . <end>


English: I like to sit in the front of the bus .
French (input): <start> J aime m asseoir a l avant du bus .
French (output): J aime m asseoir a l avant du bus . <end>


English: It may rain in the evening .
French (input): <start> Il se peut qu il pleuve dans la soiree .
French (output): Il se peut qu il pleuve dans la soiree . <end>


English: Is it still raining ?
French (input): <start> Pleut il encore ?
French (output): Pleut il encore ? <end>


English: I wish you luck .
French (input): <start> Je te souhaite bonne chance .
French (output): Je te souhaite bonne chance . <end><jupyter_text>**Split Data** We split data into train and test by randomly selecting 20% as the test set.<jupyter_code>test_percent = 0.10

# Split data into train / validate
split_outputs = train_test_split(data_en, data_fr_in, data_fr_out, test_size=test_percent)
train_en, test_en, train_fr_in, test_fr_in, train_fr_out, test_fr_out = split_outputs

print("train_en count:",len(train_en))
print("test_en count:",len(test_en))<jupyter_output>train_en count: 167024
test_en count: 18559<jupyter_text>**Build Data Pipelines**  **Text Tokenization**<jupyter_code># Tokenize english
en_tokenizer = keras.preprocessing.text.Tokenizer(filters='')
en_tokenizer.fit_on_texts(data_en)
# Train
train_tokens_en = en_tokenizer.texts_to_sequences(train_en)
train_tokens_en = keras.preprocessing.sequence.pad_sequences(train_tokens_en,padding='post')
# Test
test_tokens_en = en_tokenizer.texts_to_sequences(test_en)
test_tokens_en = keras.preprocessing.sequence.pad_sequences(test_tokens_en,padding='post')


# Tokenize french input and output
fr_tokenizer = keras.preprocessing.text.Tokenizer(filters='')
fr_tokenizer.fit_on_texts(data_fr_in)
fr_tokenizer.fit_on_texts(data_fr_out)
# Train
train_tokens_fr_in = fr_tokenizer.texts_to_sequences(train_fr_in)
train_tokens_fr_in = keras.preprocessing.sequence.pad_sequences(train_tokens_fr_in,padding='post')
train_tokens_fr_out = fr_tokenizer.texts_to_sequences(train_fr_out)
train_tokens_fr_out = keras.preprocessing.sequence.pad_sequences(train_tokens_fr_out,padding='post')
# Test
test_tokens_fr_in = fr_tokenizer.texts_to_sequences(test_fr_in)
test_tokens_fr_in = keras.preprocessing.sequence.pad_sequences(test_tokens_fr_in,padding='post')
test_tokens_fr_out = fr_tokenizer.texts_to_sequences(test_fr_out)
test_tokens_fr_out = keras.preprocessing.sequence.pad_sequences(test_tokens_fr_out,padding='post')

vocabulary_size_en = len(en_tokenizer.word_index)+1
print("Vocabulary Size english:",vocabulary_size_en)
vocabulary_size_fr = len(fr_tokenizer.word_index)+1
print("Vocabulary Size french:",vocabulary_size_fr)

print("train_tokens_en",len(train_tokens_en))
print(train_tokens_en[:5,:5])
print("train_tokens_fr_in",len(train_tokens_fr_in))
print(train_tokens_fr_in[:5,:5])
print("test_tokens_fr_out",len(test_tokens_fr_out))
print(test_tokens_fr_out[:5,:5])<jupyter_output>Vocabulary Size english: 14412
Vocabulary Size french: 23568
train_tokens_en 167024
[[   2   36    3   33    4]
 [   2   65   25 1240 1039]
 [   3  555   25 1786    1]
 [   2   27    8   96   12]
 [  60   63   13 2440    1]]
train_tokens_fr_in 167024
[[   2    4   82   11   10]
 [   2   84   43    5  758]
 [   2   16   59  263   57]
 [   2    4   20   25   72]
 [   2  105   14 1491    1]]
test_tokens_fr_out 18559
[[  162     8    93  6661  2993]
 [    4    14    30     9   183]
 [ 1021    26    79    47     6]
 [   81   266     5    76 17136]
 [   31     8    21  2274   276]]<jupyter_text>**Create TF Datasets**<jupyter_code>BATCH_SIZE = 64
TRAIN_SHUFFLE_BUFFER_SIZE = len(train_tokens_en)

# Create TF Dataset
train_data = tf.data.Dataset.from_tensor_slices((train_tokens_en, train_tokens_fr_in, train_tokens_fr_out))
test_data = tf.data.Dataset.from_tensor_slices((test_en, test_fr_in, test_fr_out))

#############
# Train data
#############
train_data = train_data.shuffle(buffer_size=TRAIN_SHUFFLE_BUFFER_SIZE)
train_data = train_data.batch(BATCH_SIZE, drop_remainder=True)

print("train_data",train_data)<jupyter_output>train_data <BatchDataset shapes: ((64, 49), (64, 62), (64, 62)), types: (tf.int32, tf.int32, tf.int32)><jupyter_text>Take a batch of data and view some data to verify our data pipeline<jupyter_code>for batch in train_data.take(1):
  print("train_en",batch[0][0,:10])
  print("train_fr_in",batch[1][0,:10])
  print("train_fr_out",batch[2][0,:10])<jupyter_output>train_en tf.Tensor([   2   37 1756  257    1    0    0    0    0    0], shape=(10,), dtype=int32)
train_fr_in tf.Tensor([   2   19   62   28 3083   11    4  476    1    0], shape=(10,), dtype=int32)
train_fr_out tf.Tensor([  19   62   28 3083   11    4  476    1    3    0], shape=(10,), dtype=int32)<jupyter_text>**Neural Machine Translation Models**  **Sequence 2 Sequence*** We will start with a simple sequence to sequence model* The Encoder will consist of an Embedding and LSTM layer* The Decoder will also consist of an Embedding and LSTM layer* In addtion the Decoder will have a final logits layer with the size of the french vocabulary* Like we saw in lecture the final hidden state of the encoderis the initial state of the decoder  Lets look at the various parts to the Encoder Decoder architecture  Build ModelWe will create build custom `keras.layers` for the Encoder and Decoder<jupyter_code>class Encoder(keras.layers.Layer):
  def __init__(self, vocab_size, embedding_size, lstm_size):
    super(Encoder, self).__init__()

    self.embedding = keras.layers.Embedding(input_dim=vocab_size, output_dim=embedding_size, name="encoder_embedding")
    self.lstm = keras.layers.LSTM(units=lstm_size, return_sequences=True, return_state=True, name="encoder_lstm")
  
  def call(self, data):
    encoder_outputs = self.embedding(data)
    encoder_outputs = self.lstm(encoder_outputs)
    return encoder_outputs
class Decoder(keras.layers.Layer):
  def __init__(self, vocab_size, embedding_size, lstm_size):
    super(Decoder, self).__init__()

    self.embedding = keras.layers.Embedding(input_dim=vocab_size, output_dim=embedding_size, name="decoder_embedding")
    self.lstm = keras.layers.LSTM(units=lstm_size, return_sequences=True, return_state=True, name="decoder_lstm")
    self.logits = keras.layers.Dense(units=vocab_size)

  def call(self, data, initial_state):
    decoder_outputs = self.embedding(data)
    decoder_outputs = self.lstm(decoder_outputs, initial_state=initial_state)
    logits = self.logits(decoder_outputs[0])
    return logits, decoder_outputs[1], decoder_outputs[2]
class Sequence2Sequence(keras.Model):
  def __init__(self,source_vocab_size, target_vocab_size, embedding_size, lstm_size):
    super(Sequence2Sequence, self).__init__()

    # Encoder
    self.encoder = Encoder(source_vocab_size, embedding_size, lstm_size)
    # Decoder
    self.decoder = Decoder(target_vocab_size, embedding_size, lstm_size)
  
  def train_step(self, data):
    # Unpack data
    x1, x2, y = data

    with tf.GradientTape() as tape:
      # Encoder
      encoder_outputs = self.encoder(x1)
      encoder_states = encoder_outputs[1:]
      # Decoder
      decoder_outputs = self.decoder(x2, encoder_states)
      logits = decoder_outputs[0]

      # Calculate loss
      loss = self.loss(y, logits)

    # Compute gradients
    trainable_variables = self.encoder.trainable_variables + self.decoder.trainable_variables
    gradients = tape.gradient(loss, trainable_variables)

    # Update weights
    self.optimizer.apply_gradients(zip(gradients, trainable_variables))

    # Return a dict of performance
    results = {m.name: m.result() for m in self.metrics}
    results.update({"loss": loss})
    return results<jupyter_output><empty_output><jupyter_text>Utils<jupyter_code># Function to perform translation using the model
def translate_seq2seq(model,test_source_text=None, max_ouput_size=25):

  if test_source_text is None:
    test_source_text = data_en[np.random.choice(len(data_en))]
  test_source_seq = en_tokenizer.texts_to_sequences([test_source_text])

  # Encoder
  encoder_outputs = model.encoder(tf.constant(test_source_seq))
  encoder_states = encoder_outputs[1:]

  # Decoder
  decoder_input = tf.constant([[fr_tokenizer.word_index['<start>']]])
  decoder_state_h, decoder_state_c = encoder_outputs[1:]
  output_words = []
  output_seq = []
  for i in range(max_ouput_size):
    decoder_output, decoder_state_h, decoder_state_c = model.decoder(decoder_input, (decoder_state_h, decoder_state_c))
    decoder_input = tf.argmax(decoder_output, -1)
    output_seq.append(decoder_input.numpy()[0][0])
    output_words.append(fr_tokenizer.index_word[output_seq[-1]])
    if output_words[-1] == '<end>':
        break

  output_text = ' '.join(output_words)

  return test_source_text, output_text<jupyter_output><empty_output><jupyter_text>Train<jupyter_code>############################
# Training Params
############################
epochs = 20
embedding_size = 256
lstm_size = 512
learning_rate = 0.01

# Free up memory
K.clear_session()

# Build the model
model = Sequence2Sequence(vocabulary_size_en, vocabulary_size_fr, embedding_size, lstm_size)

# Optimizer
optimizer = keras.optimizers.Adam(lr=learning_rate, clipnorm=5.0)

# Loss
crossentropy = keras.losses.SparseCategoricalCrossentropy(from_logits=True)
def custom_loss(targets, logits):
  mask = tf.math.logical_not(tf.math.equal(targets, 0))
  mask = tf.cast(mask, dtype=tf.int64)
  loss = crossentropy(targets, logits, sample_weight=mask)
  return loss

# Callback
class DisplayTranslation(keras.callbacks.Callback):
  def on_epoch_end(self, epoch, logs=None):
    print("\nTranslation:")
    source_text, output_text = translate_seq2seq(self.model)
    print(source_text," => ", output_text)

# Compile
model.compile(optimizer=optimizer, loss=custom_loss)

# Train model
start_time = time.time()
training_results = model.fit(
        train_data, # train_data.take(20) while testing
        epochs=epochs,
        callbacks=[DisplayTranslation()],
        verbose=1)
execution_time = (time.time() - start_time)/60.0
print("Training execution time (mins)",execution_time)<jupyter_output>Epoch 1/20
2609/2609 [==============================] - 291s 105ms/step - loss: 0.5040

Translation:
You re safe .  =>  un coup de feu . <end>
Epoch 2/20
2609/2609 [==============================] - 273s 104ms/step - loss: 0.3864

Translation:
How about a cup of hot coffee ?  =>  comme je suis en train de faire ca ? <end>
Epoch 3/20
2609/2609 [==============================] - 275s 105ms/step - loss: 0.3500

Translation:
I was able to catch the last train because I walked very quickly .  =>  je suis en train de me dire que tu ne veux pas que tu sois la . <end>
Epoch 4/20
2609/2609 [==============================] - 273s 105ms/step - loss: 0.3290

Translation:
The road was obstructed by fallen trees .  =>  la maison est un crime . <end>
Epoch 5/20
2609/2609 [==============================] - 272s 104ms/step - loss: 0.3158

Translation:
I didn t want any of that .  =>  je ne suis pas sur que ca arrive . <end>
Epoch 6/20
2609/2609 [==============================] - 273s 105ms/step - loss:[...]<jupyter_text>Evaluate<jupyter_code>evaluate_save_model(model, test_en[:1000], test_fr_out[:1000],translate_seq2seq,
                    training_results, execution_time, learning_rate, epochs)
# Make some translations using the model
for i,data_idx in enumerate(data_samples):
  print("English:",data_en[data_idx])
  print("French (actual):",data_fr_out[data_idx])
  source_text, output_text = translate_seq2seq(model, test_source_text=data_en[data_idx])
  print("French (prediction):",output_text)
  print("\n")<jupyter_output>English: They found us .
French (actual): Elles nous ont trouves . <end>
French (prediction): portent croire de maniere . <end>


English: I don t compromise .
French (actual): Je ne transige pas . <end>
French (prediction): continues de faire face a autre chose . <end>


English: I like to sit in the front of the bus .
French (actual): J aime m asseoir a l avant du bus . <end>
French (prediction): continues de grignoter comme un adulte acharne sur la table . <end>


English: It may rain in the evening .
French (actual): Il se peut qu il pleuve dans la soiree . <end>
French (prediction): reflechir ce matin . <end>


English: Is it still raining ?
French (actual): Pleut il encore ? <end>
French (prediction): allonger mon dejeuner . <end>


English: I wish you luck .
French (actual): Je te souhaite bonne chance . <end>
French (prediction): promis pas que nous gardions nos distances . <end><jupyter_text>**Sequence 2 Sequence with Attention*** Next we will add Attention to the decoder* The Encoder will consist of an Embedding and LSTM layer* The Decoder will also consist of an Embedding and LSTM layer* In addition the Decoder will have an Attention layer* In addtion the Decoder will have a final logits layer with the size of the french vocabulary  Lets look at the various parts to the Encoder Decoder architecture  Build Model<jupyter_code>class Attention(keras.layers.Layer):
  def __init__(self, lstm_size):
    super(Attention, self).__init__()
    self.wa = keras.layers.Dense(lstm_size)

  def call(self, decoder_output, encoder_output):

    # Score
    score = tf.matmul(decoder_output, self.wa(encoder_output), transpose_b=True)

    # alignment = softmax(score)
    alignment = tf.nn.softmax(score, axis=2)

    # context vector is the weighted average sum of encoder output
    context = tf.matmul(alignment, encoder_output)

    return context, alignment
class Encoder(keras.layers.Layer):
  def __init__(self, vocab_size, embedding_size, lstm_size):
    super(Encoder, self).__init__()

    self.embedding = keras.layers.Embedding(input_dim=vocab_size, output_dim=embedding_size, name="encoder_embedding")
    self.lstm = keras.layers.LSTM(units=lstm_size, return_sequences=True, return_state=True, name="encoder_lstm")
  
  def call(self, data):
    encoder_outputs = self.embedding(data)
    lstm_out, state_h, state_c = self.lstm(encoder_outputs)
    return lstm_out, state_h, state_c
class Decoder(keras.layers.Layer):
  def __init__(self, vocab_size, embedding_size, lstm_size):
    super(Decoder, self).__init__()
    
    self.embedding = keras.layers.Embedding(input_dim=vocab_size, output_dim=embedding_size, name="decoder_embedding")
    self.lstm = keras.layers.LSTM(units=lstm_size, return_sequences=True, return_state=True, name="decoder_lstm")
    # Add Attention
    self.attention = Attention(lstm_size)

    self.wc = keras.layers.Dense(units=lstm_size, activation='tanh')
    self.ws = keras.layers.Dense(units=vocab_size)

  def call(self, data, initial_state, encoder_output):
    # Get LSTM outputs
    embed = self.embedding(data)
    lstm_out, state_h, state_c = self.lstm(embed, initial_state=initial_state)

    # Get context and alignment vectors using the Attention layer
    context, alignment = self.attention(lstm_out, encoder_output)

    # Combine the context vector and the LSTM output
    lstm_out = tf.concat([tf.squeeze(context, 1), tf.squeeze(lstm_out, 1)], 1)

    # Pass lstm_out through Dense layer
    lstm_out = self.wc(lstm_out)

    # Output logits of vocab_size
    logits = self.ws(lstm_out)

    return logits, state_h, state_c, alignment
class Sequence2SequenceAttention(keras.Model):
  def __init__(self,source_vocab_size, target_vocab_size, embedding_size, lstm_size):
    super(Sequence2SequenceAttention, self).__init__()

    # Encoder
    self.encoder = Encoder(source_vocab_size, embedding_size, lstm_size)
    # Decoder
    self.decoder = Decoder(target_vocab_size, embedding_size, lstm_size)

  def train_step(self, data):
    # Unpack data
    x1, x2, y = data

    loss = 0
    with tf.GradientTape() as tape:
      encoder_outputs = self.encoder(x1)
      encoder_states = encoder_outputs[1:]
      decoder_state_h, decoder_state_c = encoder_states

      # For every word in y
      for i in range(y.shape[1]):
          # Decoder input is one word at a time
          decoder_input = tf.expand_dims(x2[:, i], 1)
          logits, decoder_state_h, decoder_state_c, _ = self.decoder(decoder_input, (decoder_state_h, decoder_state_c), encoder_outputs[0])

          # The loss is now accumulated through the whole batch
          loss += self.loss(y[:, i], logits)

    variables = self.encoder.trainable_variables + self.decoder.trainable_variables
    gradients = tape.gradient(loss, variables)
    self.optimizer.apply_gradients(zip(gradients, variables))

    # Return a dict of performance
    results = {m.name: m.result() for m in self.metrics}
    results.update({"loss": loss / y.shape[1]})
    return results<jupyter_output><empty_output><jupyter_text>Utils<jupyter_code># Function to perform translation using the model
def translate_seq2seqattention(model,test_source_text=None, max_ouput_size=25):

  if test_source_text is None:
    test_source_text = data_en[np.random.choice(len(data_en))]
  test_source_seq = en_tokenizer.texts_to_sequences([test_source_text])

  # Encoder
  encoder_outputs = model.encoder(tf.constant(test_source_seq))

  # Decoder
  decoder_input = tf.constant([[fr_tokenizer.word_index['<start>']]])
  decoder_state_h, decoder_state_c = encoder_outputs[1:]

  output_words = []
  output_seq = []
  for i in range(max_ouput_size):
    logits, decoder_state_h, decoder_state_c, _ = model.decoder(decoder_input, 
                                                                (decoder_state_h, decoder_state_c), 
                                                                encoder_outputs[0])
    decoder_input = tf.expand_dims(tf.argmax(logits, -1), 0)
    output_seq.append(decoder_input.numpy()[0][0])
    output_words.append(fr_tokenizer.index_word[output_seq[-1]])
    if output_words[-1] == '<end>':
        break

  output_text = ' '.join(output_words)

  return test_source_text, output_text<jupyter_output><empty_output><jupyter_text>Train<jupyter_code>############################
# Training Params
############################
epochs = 20
embedding_size = 256
lstm_size = 512
learning_rate = 0.01

# Free up memory
K.clear_session()

# Build the model
model = Sequence2SequenceAttention(vocabulary_size_en, vocabulary_size_fr, embedding_size, lstm_size)

# Optimizer
optimizer = keras.optimizers.Adam(lr=learning_rate, clipnorm=5.0)
# Loss
crossentropy = keras.losses.SparseCategoricalCrossentropy(from_logits=True)
def custom_loss(targets, logits):
  mask = tf.math.logical_not(tf.math.equal(targets, 0))
  mask = tf.cast(mask, dtype=tf.int64)
  loss = crossentropy(targets, logits, sample_weight=mask)
  return loss

# Callback
class DisplayTranslation(keras.callbacks.Callback):
  def on_epoch_end(self, epoch, logs=None):
    print("\nTranslation:")
    source_text, output_text = translate_seq2seqattention(self.model)
    print(source_text," => ", output_text)

# Compile
model.compile(optimizer=optimizer, loss=custom_loss)

# Train model
start_time = time.time()
training_results = model.fit(
        train_data, # train_data.take(20) while testing
        epochs=epochs,
        callbacks=[DisplayTranslation()],
        verbose=1)
execution_time = (time.time() - start_time)/60.0
print("Training execution time (mins)",execution_time)<jupyter_output>Epoch 1/20
2609/2609 [==============================] - 842s 295ms/step - loss: 0.9762

Translation:
It was a bad movie .  =>  ca parviendrons . <end>
Epoch 2/20
2609/2609 [==============================] - 769s 295ms/step - loss: 1.0220

Translation:
You can t blame him .  =>  tu n aimait pas son cousin . <end>
Epoch 3/20
2609/2609 [==============================] - 768s 294ms/step - loss: 1.0424

Translation:
The accident happened two years ago .  =>  reflechissez tu pas de nous aider mes excuses aider . <end>
Epoch 4/20
2609/2609 [==============================] - 767s 294ms/step - loss: 1.0578

Translation:
I don t want you to be hurt .  =>  je ne pense pas faire ca . <end>
Epoch 5/20
2609/2609 [==============================] - 768s 294ms/step - loss: 1.0742

Translation:
This may just come in handy someday .  =>  est en danger une encyclopedie . <end>
Epoch 6/20
2609/2609 [==============================] - 768s 294ms/step - loss: 1.0799

Translation:
The woman gave birth to a bab[...]<jupyter_text>Evaluate<jupyter_code>evaluate_save_model(model, test_en[:1000], test_fr_out[:1000],translate_seq2seqattention,
                    training_results, execution_time, learning_rate, epochs)
# Make some translations using the model
for i,data_idx in enumerate(data_samples):
  print("English:",data_en[data_idx])
  print("French (actual):",data_fr_out[data_idx])
  source_text, output_text = translate_seq2seqattention(model, test_source_text=data_en[data_idx])
  print("French (prediction):",output_text)
  print("\n")<jupyter_output>English: They found us .
French (actual): Elles nous ont trouves . <end>
French (prediction): la raison . <end>


English: I don t compromise .
French (actual): Je ne transige pas . <end>
French (prediction): je ne fais pas . <end>


English: I like to sit in the front of the bus .
French (actual): J aime m asseoir a l avant du bus . <end>
French (prediction): je comprends comme destruction dit . <end>


English: It may rain in the evening .
French (actual): Il se peut qu il pleuve dans la soiree . <end>
French (prediction): d abord important . <end>


English: Is it still raining ?
French (actual): Pleut il encore ? <end>
French (prediction): qui prend chose qui est assis bonne idee de nouveau . <end>


English: I wish you luck .
French (actual): Je te souhaite bonne chance . <end>
French (prediction): je vous souhaite . <end><jupyter_text>**Transfomer*** The transformer will consist of an Encoder and Decoder* The Encoder consists of embedding, positional encoding, self attention, normalization, and feed forward layers* The Decoder consists of embedding, positional encoding, decoder self attention, encoder-decoder attention, normalization, and feed forward layers Lets look at the various parts to the Transfomer Encoder Decoder architecture  Build Model **Model Param**<jupyter_code>############################
# Model Params
############################
MODEL_SIZE = 128 # depth size of the model
NUM_LAYERS = 4 # number of layers (Multi-Head Attention + FFN)
NUM_HEADS = 8 # number of attention heads<jupyter_output><empty_output><jupyter_text>**Positional Encoding**<jupyter_code>def generate_positional_encoding(max_length, model_size):
    pos_enc = np.array(
        [
            [pos / np.power(10000, 2 * (j // 2) / model_size) for j in range(model_size)]
            if pos != 0
            else np.zeros(model_size)
            for pos in range(max_length)
        ]
    )
    pos_enc[1:, 0::2] = np.sin(pos_enc[1:, 0::2])  # dim 2i
    pos_enc[1:, 1::2] = np.cos(pos_enc[1:, 1::2])  # dim 2i+1
    return pos_enc<jupyter_output><empty_output><jupyter_text>Here we have an implementation of `MultiHeadSelfAttention` layer. But you can use the `keras.layers.MultiHeadSelfAttention` directly<jupyter_code>class MultiHeadSelfAttention(keras.layers.Layer):
  def __init__(self, model_size, num_heads):
    super(MultiHeadSelfAttention, self).__init__()

    self.key_size = model_size // num_heads # Key size
    self.num_heads = num_heads # number of attention heads

    # Query weights
    self.wq = keras.layers.Dense(units=model_size)
    # Key weights
    self.wk = keras.layers.Dense(units=model_size)
    # Value weights
    self.wv = keras.layers.Dense(units=model_size)
    # Output weights
    self.wo = keras.layers.Dense(units=model_size)

  def call(self, query, value, mask=None):
    
    ############################################
    # Step 1: Create query, key, value vectors
    ############################################
    query = self.wq(query) # query has shape (batch, query_len, model_size)
    key = self.wk(value) # key has shape (batch, value_len, model_size)
    value = self.wv(value) # value has shape (batch, value_len, model_size)
    batch_size = query.shape[0]
    # Each heads query, key, value needs to be size = key_size
    # Reshape to (batch, query_len, num_heads, key_size)
    query = tf.reshape(query, [batch_size, -1, self.num_heads, self.key_size])
    key = tf.reshape(key, [batch_size, -1, self.num_heads, self.key_size])
    value = tf.reshape(value, [batch_size, -1, self.num_heads, self.key_size])
    # Tranpose to (batch, num_heads, query_len, key_size) for matmul
    query = tf.transpose(query, [0, 2, 1, 3])
    key = tf.transpose(key, [0, 2, 1, 3])
    value = tf.transpose(value, [0, 2, 1, 3])
    
    ########################################
    # Step 2+3: Compute Self-Attention Score
    ########################################
    # Score is dot product of query vector and key vector
    # Score is then divided by the square root of key_size : This leads to having more stable gradients
    score = tf.matmul(query, key, transpose_b=True) / tf.math.sqrt(tf.dtypes.cast(self.key_size, dtype=tf.float32))
    
    # Mask out scores if a mask is provided
    if mask is not None:
        score *= mask
        # Apply a very large negative value so the softmax in the next step will zero it out
        score = tf.where(tf.equal(score, 0), tf.ones_like(score) * -1e9, score)
    
    ##################################################
    # Step 4: Normalize Score to get Alignment vector
    ##################################################
    # Pass the score through a softmax operation
    # Softmax normalizes the scores so they’re all positive and add up to 1
    # Alignment vector shape = (batch, h, query_len, value_len)
    alignment = tf.nn.softmax(score, axis=-1)
    
    ################################
    # Step 5: Compute Context vector
    ################################
    # Computed by multiplying each value vector with alignment vector
    # Context vector shape = (batch, h, query_len, key_size)
    context = tf.matmul(alignment, value)
    
    ################################
    # Step 6: Compute Final Output
    ################################
    # Align all the context vectors (from all heads)
    context = tf.transpose(context, [0, 2, 1, 3])
    context = tf.reshape(context, [batch_size, -1, self.key_size * self.num_heads])
    # Multiply context vectors with weight matric wo
    z = self.wo(context)
    
    # Final Z matrix, alignment of all heads
    return z, alignment<jupyter_output><empty_output><jupyter_text>**Encoder**<jupyter_code>class Encoder(keras.layers.Layer):
  def __init__(self, vocab_size, model_size, num_layers, num_heads):
    super(Encoder, self).__init__()

    self.model_size = model_size # depth size of the model
    self.num_layers = num_layers # number of layers (Multi-Head Self Attention + FFN)
    self.num_heads = num_heads # number of attention heads

    # Embedding Layer
    self.embedding = keras.layers.Embedding(vocab_size, self.model_size)
    self.embedding_dropout = keras.layers.Dropout(0.1)

    ###################
    # Attention Blocks (repeated num_layer times)
    ###################

    # Multi-Head Self Attention + Normalization
    self.attention = [MultiHeadSelfAttention(self.model_size, self.num_heads) for _ in range(self.num_layers)]
    self.attention_dropout = [keras.layers.Dropout(0.1) for _ in range(self.num_layers)]
    self.attention_norm = [keras.layers.LayerNormalization(epsilon=1e-6) for _ in range(self.num_layers)]

    # Feedforward + Normalization
    self.dense_1 = [keras.layers.Dense(self.model_size * 4, activation='relu') for _ in range(self.num_layers)]
    self.dense_2 = [keras.layers.Dense(self.model_size) for _ in range(self.num_layers)]
    self.ffn_dropout = [keras.layers.Dropout(0.1) for _ in range(self.num_layers)]
    self.ffn_norm = [keras.layers.LayerNormalization(epsilon=1e-6) for _ in range(self.num_layers)]

  def call(self, data, training=True, encoder_mask=None):
    # Embedding output
    embedding_output = self.embedding(data)
    embedding_output *= tf.math.sqrt(tf.cast(self.model_size, tf.float32))
    # Add Positionsl Encoding to embedding output
    embedding_output = embedding_output + generate_positional_encoding(data.shape[1],self.model_size)
    embedding_output = self.embedding_dropout(embedding_output, training=training)

    # Sub layer input
    sub_layer_input = embedding_output
    alignments = [] # alignment (attention) vectors for all layers

    # Repeat for each layer
    for i in range(self.num_layers):
      # Pass in query, value, mask to Attention layer
      sub_layer_output, alignment = self.attention[i](sub_layer_input, sub_layer_input, encoder_mask)
      sub_layer_output = self.attention_dropout[i](sub_layer_output, training=training)
      # Add residual connection 
      sub_layer_output = sub_layer_input + sub_layer_output
      # Normalization
      sub_layer_output = self.attention_norm[i](sub_layer_output)
      alignments.append(alignment)

      # Feedforward Network
      ffn_input = sub_layer_output
      ffn_output = self.dense_2[i](self.dense_1[i](ffn_input))
      ffn_output = self.ffn_dropout[i](ffn_output, training=training)
      # Add residual connection 
      ffn_output = ffn_input + ffn_output
      # Normalization
      ffn_output = self.ffn_norm[i](ffn_output)

      # Pass FFN output to the next layer
      sub_layer_input = ffn_output

    # Final encoder output and all alignment vectors
    return ffn_output, alignments<jupyter_output><empty_output><jupyter_text>**Decoder**<jupyter_code>class Decoder(keras.layers.Layer):
  def __init__(self, vocab_size, model_size, num_layers, num_heads):
    super(Decoder, self).__init__()
    
    self.model_size = model_size # depth size of the model
    self.num_layers = num_layers # number of layers (Multi-Head Self Attention + FFN)
    self.num_heads = num_heads # number of attention heads

    # Embedding Layer
    self.embedding = keras.layers.Embedding(vocab_size, model_size)
    self.embedding_dropout = keras.layers.Dropout(0.1)

    ###################
    # Attention Blocks (repeated num_layer times)
    ###################

    # Bottom Multi-Head Self Attention + Normalization
    self.attention_bottom = [MultiHeadSelfAttention(self.model_size, self.num_heads) for _ in range(self.num_layers)]
    self.attention_bottom_dropout = [keras.layers.Dropout(0.1) for _ in range(self.num_layers)]
    self.attention_bottom_norm = [keras.layers.LayerNormalization(epsilon=1e-6) for _ in range(self.num_layers)]

    # Middle Multi-Head Self Attention + Normalization
    self.attention_middle = [MultiHeadSelfAttention(self.model_size, self.num_heads) for _ in range(self.num_layers)]
    self.attention_middle_dropout = [keras.layers.Dropout(0.1) for _ in range(self.num_layers)]
    self.attention_middle_norm = [keras.layers.LayerNormalization(epsilon=1e-6) for _ in range(self.num_layers)]

    # Feedforward + Normalization
    self.dense_1 = [keras.layers.Dense(units=self.model_size * 4, activation='relu') for _ in range(self.num_layers)]
    self.dense_2 = [keras.layers.Dense(units=self.model_size) for _ in range(self.num_layers)]
    self.ffn_dropout = [keras.layers.Dropout(0.1) for _ in range(self.num_layers)]
    self.ffn_norm = [keras.layers.LayerNormalization(epsilon=1e-6) for _ in range(self.num_layers)]

    # Final output/logits layer
    self.dense = keras.layers.Dense(units=vocab_size)

  def call(self, data, encoder_output, training=True, encoder_mask=None):
    # Embedding output
    embedding_output = self.embedding(data)
    embedding_output *= tf.math.sqrt(tf.cast(self.model_size, tf.float32))
    # Add Positionsl Encoding to embedding output
    embedding_output = embedding_output + generate_positional_encoding(data.shape[1],self.model_size)
    embedding_output = self.embedding_dropout(embedding_output)

    bottom_sub_layer_input = embedding_output
    bottom_alignments = [] # bottom alignment (attention) vectors for all layers
    middle_alignments = [] # middle alignment (attention) vectors for all layers

    # Repeat for each layer
    for i in range(self.num_layers):
      # Bottom Multi-Head Sub Layer
      sequence_length = bottom_sub_layer_input.shape[1]

      if training:
        mask = tf.linalg.band_part(tf.ones((sequence_length, sequence_length)), -1, 0)
      else:
        mask = None

      # Pass in query, value, mask to Attention layer
      bottom_sub_layer_output, bottom_alignment = self.attention_bottom[i](bottom_sub_layer_input, bottom_sub_layer_input, mask)
      bottom_sub_layer_output = self.attention_bottom_dropout[i](bottom_sub_layer_output, training=training)
      # Add residual connection 
      bottom_sub_layer_output = bottom_sub_layer_input + bottom_sub_layer_output
      # Normalization
      bottom_sub_layer_output = self.attention_bottom_norm[i](bottom_sub_layer_output)
      bottom_alignments.append(bottom_alignment)

      # Middle Multi-Head Sub Layer
      middle_sub_layer_input = bottom_sub_layer_output
      # Pass in query, value, mask to Attention layer
      middle_sub_layer_output, middle_alignment = self.attention_middle[i](middle_sub_layer_input, encoder_output, encoder_mask)
      middle_sub_layer_output = self.attention_middle_dropout[i](middle_sub_layer_output, training=training)
      # Add residual connection 
      middle_sub_layer_output = middle_sub_layer_output + middle_sub_layer_input
      # Normalization
      middle_sub_layer_output = self.attention_middle_norm[i](middle_sub_layer_output)
      middle_alignments.append(middle_alignment)

      # Feedforward Network
      ffn_input = middle_sub_layer_output
      ffn_output = self.dense_2[i](self.dense_1[i](ffn_input))
      ffn_output = self.ffn_dropout[i](ffn_output, training=training)
      # Add residual connection 
      ffn_output = ffn_output + ffn_input
      # Normalization
      ffn_output = self.ffn_norm[i](ffn_output)

      # Pass FFN output to the next layer
      bottom_sub_layer_input = ffn_output

    # Compute final output logits (batch_size, length, model_size)
    logits = self.dense(ffn_output)

    # Final decoder output and bottom and middle alignment vectors
    return logits, bottom_alignments, middle_alignments<jupyter_output><empty_output><jupyter_text>**Transfomer Model**<jupyter_code>class Transformer(keras.Model):
  def __init__(self, source_vocab_size, target_vocab_size, model_size, num_layers, num_heads):
    super(Transformer, self).__init__()

    # Encoder
    self.encoder = Encoder(source_vocab_size, model_size, num_layers, num_heads)
    # Decoder
    self.decoder = Decoder(target_vocab_size, model_size, num_layers, num_heads)

  def train_step(self, data):
    # Unpack data
    x1, x2, y = data

    with tf.GradientTape() as tape:
        encoder_mask = 1 - tf.cast(tf.equal(x1, 0), dtype=tf.float32)
        # Add two more dimensions to make it broadcastable when computing attention heads
        encoder_mask = tf.expand_dims(encoder_mask, axis=1)
        encoder_mask = tf.expand_dims(encoder_mask, axis=1)
        # Encoder
        encoder_output, _ = self.encoder(x1, encoder_mask=encoder_mask)
        # Decoder
        decoder_output, _, _ = self.decoder(x2, encoder_output, encoder_mask=encoder_mask)
        # Compute loss
        loss = self.loss(y, decoder_output)

    variables = self.encoder.trainable_variables + self.decoder.trainable_variables
    gradients = tape.gradient(loss, variables)
    self.optimizer.apply_gradients(zip(gradients, variables))

    # Return a dict of performance
    results = {m.name: m.result() for m in self.metrics}
    results.update({"loss": loss})
    return results<jupyter_output><empty_output><jupyter_text>Utils<jupyter_code># Function to perform translation using the model
def translate_transformer(model,test_source_text=None, max_ouput_size=25):
  if test_source_text is None:
    test_source_text = data_en[np.random.choice(len(data_en))]
  test_source_seq = en_tokenizer.texts_to_sequences([test_source_text])

  # Encoder
  en_output, _ = model.encoder(tf.constant(test_source_seq), training=False)
  # Decoder
  decoder_input = tf.constant([[fr_tokenizer.word_index['<start>']]], dtype=tf.int64)

  output_words = []
  output_seq = []
  for i in range(max_ouput_size):
    de_output, _,_ = model.decoder(decoder_input, en_output, training=False)
    new_word = tf.expand_dims(tf.argmax(de_output, -1)[:, -1], axis=1)
    output_seq.append(new_word.numpy()[0][0])
    output_words.append(fr_tokenizer.index_word[output_seq[-1]])

    # Add the predicted word to create a new input sequence
    decoder_input = tf.concat((decoder_input, new_word), axis=-1)

    # Check for end of sentence
    if output_words[-1] == '<end>':
        break

  output_text = ' '.join(output_words)

  return test_source_text, output_text

class WarmupThenDecaySchedule(tf.keras.optimizers.schedules.LearningRateSchedule):
    """ Learning schedule for training the Transformer
    Attributes:
        model_size: d_model in the paper (depth size of the model)
        warmup_steps: number of warmup steps at the beginning
    """
    def __init__(self, model_size, warmup_steps=4000):
        super(WarmupThenDecaySchedule, self).__init__()

        self.model_size = model_size
        self.model_size = tf.cast(self.model_size, tf.float32)

        self.warmup_steps = warmup_steps

    def __call__(self, step):
        step_term = tf.math.rsqrt(step)
        warmup_term = step * (self.warmup_steps ** -1.5)

        return tf.math.rsqrt(self.model_size) * tf.math.minimum(step_term, warmup_term)<jupyter_output><empty_output><jupyter_text>Train<jupyter_code>############################
# Training Params
############################
epochs = 20

# Free up memory
K.clear_session()

# Build the model
model = Transformer(vocabulary_size_en, vocabulary_size_fr, MODEL_SIZE, NUM_LAYERS, NUM_HEADS)

# Optimizer
lr = WarmupThenDecaySchedule(MODEL_SIZE)
optimizer = keras.optimizers.Adam(lr,beta_1=0.9,beta_2=0.98,epsilon=1e-9)
# Loss
crossentropy = keras.losses.SparseCategoricalCrossentropy(from_logits=True)
def custom_loss(targets, logits):
  mask = tf.math.logical_not(tf.math.equal(targets, 0))
  mask = tf.cast(mask, dtype=tf.int64)
  loss = crossentropy(targets, logits, sample_weight=mask)
  return loss

# Callback
class DisplayTranslation(keras.callbacks.Callback):
  def on_epoch_end(self, epoch, logs=None):
    print("\nTranslation:")
    source_text, output_text = translate_transformer(self.model)
    print(source_text," => ", output_text)

# Compile
model.compile(optimizer=optimizer, loss=custom_loss)

# Train model
start_time = time.time()
training_results = model.fit(
        train_data, # train_data.take(20) while testing
        epochs=epochs,
        callbacks=[DisplayTranslation()],
        verbose=1)
execution_time = (time.time() - start_time)/60.0
print("Training execution time (mins)",execution_time)
evaluate_save_model(model, test_en[:1000], test_fr_out[:1000],translate_transformer,
                    training_results, execution_time, 0.0, epochs)<jupyter_output><empty_output><jupyter_text>Evaluate<jupyter_code><jupyter_output><empty_output><jupyter_text>**Compare all Models**<jupyter_code>models_store_path = "models"

models_metrics_list = glob(models_store_path+"/*_metrics.json")

all_models_metrics = []
for mm_file in models_metrics_list:
  with open(mm_file) as json_file:
    model_metrics = json.load(json_file)
    all_models_metrics.append(model_metrics)

# Load metrics to dataframe
view_metrics = pd.DataFrame(data=all_models_metrics)

# Format columns
view_metrics['trainable_parameters'] = view_metrics['trainable_parameters'].map('{:,.0f}'.format)
view_metrics['execution_time'] = view_metrics['execution_time'].map('{:,.2f} mins'.format)
view_metrics['loss'] = view_metrics['loss'].map('{:,.2f}'.format)
view_metrics['BLEU-1'] = view_metrics['BLEU-1'].map('{:,.4f}'.format)
view_metrics['BLEU-2'] = view_metrics['BLEU-2'].map('{:,.4f}'.format)
view_metrics['BLEU-3'] = view_metrics['BLEU-3'].map('{:,.4f}'.format)
view_metrics['BLEU-4'] = view_metrics['BLEU-4'].map('{:,.4f}'.format)

# Filter columns
view_metrics = view_metrics[["trainable_parameters","execution_time","loss","BLEU-1","BLEU-2","BLEU-3","BLEU-4",
                             "learning_rate","epochs","name"]]
view_metrics = view_metrics.sort_values(by=['loss'],ascending=True)
view_metrics.head()<jupyter_output><empty_output><jupyter_text>---  **Mini BERT Language Model** ---  **Overview**In this section, we're going to be working with some text related to machine & deep learning. The data is a very small set of defintions pulled form wikipedia. The primary goal will be to understand BERT(Bidirectional Encoder Representations from Transformers) architecture.**The Task:** Build a masked language model using a simplified version of the BERT. **Language Model**: A model that understands language and how words appear in context to one another. The model is trained using unsupervised approaches such as next word prediction in a sentence or next sentence prediction.**Review BERT**:* Masked Language Model* Made up of only the Encoder with stacked transformer blocks* Bidirectional language* Good for fill in the blanks  **Dataset**  **Download Dataset**Download the datasets to colab.<jupyter_code>start_time = time.time()
download_file("https://storage.googleapis.com/cs109b/nlp/deep_learning_terms.txt", base_path="datasets", extract=False)
execution_time = (time.time() - start_time)/60.0
print("Download execution time (mins)",execution_time)<jupyter_output>Download execution time (mins) 0.0040332555770874025<jupyter_text>**Load Data*** Read-in data as lists.<jupyter_code>data_text_file = os.path.join("datasets","deep_learning_terms.txt")
data_text = []
with open(data_text_file) as file:
  for line in file:
    data_text.append(line)

print("data_text len:", len(data_text))

# Get sentences
sentences = []
for text in data_text:
  sentences.extend(sent_tokenize(text))

# Add a few DEMO sentence
sentences.append("pavlos taught positional encoding to shivas")
print("sentences len:", len(sentences))<jupyter_output>data_text len: 20
sentences len: 183<jupyter_text>**View Text**Let's take a look at the data.<jupyter_code>display(sentences[:10])<jupyter_output><empty_output><jupyter_text>**Build Data Pipelines**  **Text Vectorization**<jupyter_code>MASK_TOKEN = "[mask]"
max_len = max([len(x.split()) for x in sentences])
print("max_len",max_len)

def standardize_text(input_text):
  # Convert to lowercase
  output_text = tf.strings.lower(input_text) 
  return tf.strings.regex_replace(
        output_text, "[%s]" % re.escape("!#$%&'()*+,-./:;<=>?@\^_`{|}~"), ""
    )

text_data = tf.data.Dataset.from_tensor_slices(sentences)

# Initialize Text Vectorizer
text_vectorizer = TextVectorization(
    standardize=standardize_text,
    output_mode="int",
    output_sequence_length=max_len
)

# Generate Text Vector
text_vectorizer.adapt(sentences)

# Get Vocabulary
vocabulary = text_vectorizer.get_vocabulary()
vocabulary_size = len(vocabulary)
# Add the mask token
vocabulary = vocabulary[2 : vocabulary_size - 1] + [MASK_TOKEN]
text_vectorizer.set_vocabulary(vocabulary)
vocabulary = text_vectorizer.get_vocabulary()
vocabulary_size = len(vocabulary)
print("Vocabulary Size:",vocabulary_size)
print(vocabulary[-1])

# Generate word index
word_index = dict(zip(vocabulary, range(vocabulary_size)))
index_word = dict(zip(range(vocabulary_size), vocabulary))

print("vocabulary:",len(vocabulary),vocabulary)
print("word_index:",word_index)

# Get mask token id for masked language model
mask_token_id = word_index[MASK_TOKEN]
print("Mask token Id:",mask_token_id)
# 1) Convert input text to tokens
all_data_tokens = text_vectorizer(sentences)
print("Input text:\n")
print(sentences[0])
print(sentences[1])
print(sentences[3])
print("all_data_tokens shape:",all_data_tokens.shape)
print("Tokenized text:", all_data_tokens[:5,:20])<jupyter_output>Input text:

Deep learning also known as deep structured learning is part of a broader family of machine learning methods based on artificial neural networks with representation learning.
Learning can be supervised, semi-supervised or unsupervised.
Artificial neural networks (ANNs) were inspired by information processing and distributed communication nodes in biological systems.
all_data_tokens shape: (183, 66)
Tokenized text: tf.Tensor(
[[  35    9   37   75   11   35  192    9    7  150    3    4 1048  879
     3   23    9  101  109   30]
 [   9   14   18   55   97   12   83    0    0    0    0    0    0    0
     0    0    0    0    0    0]
 [ 961  428   20   11   35   16   17   35  420   17   61   16   17    6
   244   16   17   44  254  170]
 [  33   16   17  256  178  220   36   90   57    6  934 1013  206    8
    79   60    0    0    0    0]
 [ 256   44  268  394   25   79  251    0    0    0    0    0    0    0
     0    0    0    0    0    0]], shape=(5, 20), dtype=int64)<jupyter_text>**Generate Training Data**For the training we need inputs and lables but we only have input texts. In lecture we learnt that language models are trained in a smei supervised way where we generate inputs and labels from the input text. To generate inputs and lables for training we will mask random words from the input text. Then our labels will be the masked text. The model will be trained to predict the masked word<jupyter_code># Since we have a very small dataset, we will just replicate the data
encoded_texts = all_data_tokens.numpy()
print(encoded_texts.shape)
encoded_texts = np.vstack([encoded_texts]*50)
print(encoded_texts.shape)
# 15% BERT masking
inp_mask = np.random.rand(*encoded_texts.shape) < 0.15
print("inp_mask:",inp_mask.shape)
#print("inp_mask:",inp_mask[0,:25])

# Exclude masking special tokens
inp_mask[encoded_texts <= 2] = False

# Set targets to -1 by default, it means ignore
labels = -1 * np.ones(encoded_texts.shape, dtype=int)

# Set labels for masked tokens
labels[inp_mask] = encoded_texts[inp_mask]

# Prepare input
encoded_texts_masked = np.copy(encoded_texts)

# Set input to [MASK] which is the last token for the 90% of tokens
# This means leaving 10% unchanged
inp_mask_2mask = inp_mask & (np.random.rand(*encoded_texts.shape) < 0.90)
encoded_texts_masked[
    inp_mask_2mask
] = mask_token_id  # mask token is the last in the dict

# Set 10% to a random token
inp_mask_2random = inp_mask_2mask & (np.random.rand(*encoded_texts.shape) < 1 / 9)
encoded_texts_masked[inp_mask_2random] = np.random.randint(
    3, mask_token_id, inp_mask_2random.sum()
)

# Prepare sample_weights to pass to .fit() method
sample_weights = np.ones(labels.shape)
sample_weights[labels == -1] = 0

# y_labels would be same as encoded_texts i.e input tokens
y_labels = np.copy(encoded_texts)

print("encoded_texts_masked:",encoded_texts_masked.shape)
print(encoded_texts_masked[:5,:10])
print("y_labels:",y_labels.shape)
print(y_labels[:5,:10])
print("sample_weights:",sample_weights.shape)
print(sample_weights[:5,:10])<jupyter_output>inp_mask: (9150, 66)
encoded_texts_masked: (9150, 66)
[[  35    9   37   75   11   35  192    9    7  150]
 [   9   14 1129   55   97   12 1129    0    0    0]
 [ 961  428   20   11   35 1129   17   35  420   17]
 [1129   16   17  256 1129  220   36   90   57    6]
 [ 256   44 1129  394   25   79  251    0    0    0]]
y_labels: (9150, 66)
[[ 35   9  37  75  11  35 192   9   7 150]
 [  9  14  18  55  97  12  83   0   0   0]
 [961 428  20  11  35  16  17  35 420  17]
 [ 33  16  17 256 178 220  36  90  57   6]
 [256  44 268 394  25  79 251   0   0   0]]
sample_weights: (9150, 66)
[[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 1. 0. 0. 0. 1. 0. 0. 0.]
 [0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]
 [1. 0. 0. 0. 1. 0. 0. 0. 0. 0.]
 [0. 0. 1. 0. 0. 0. 0. 0. 0. 0.]]<jupyter_text>**Create TF Datasets**<jupyter_code>BATCH_SIZE = 64
TRAIN_SHUFFLE_BUFFER_SIZE = encoded_texts.shape[0]

# Create TF Dataset
train_data = tf.data.Dataset.from_tensor_slices((encoded_texts_masked, y_labels, sample_weights))

#############
# Train data
#############
train_data = train_data.shuffle(buffer_size=TRAIN_SHUFFLE_BUFFER_SIZE)
train_data = train_data.batch(BATCH_SIZE)
train_data = train_data.prefetch(buffer_size=AUTOTUNE)

print("train_data",train_data)<jupyter_output>train_data <PrefetchDataset shapes: ((None, 66), (None, 66), (None, 66)), types: (tf.int64, tf.int64, tf.float64)><jupyter_text>**Mini BERT**  **Positional Encoding**<jupyter_code>def generate_positional_encoding(max_length, model_size):
    pos_enc = np.array(
        [
            [pos / np.power(10000, 2 * (j // 2) / model_size) for j in range(model_size)]
            if pos != 0
            else np.zeros(model_size)
            for pos in range(max_length)
        ]
    )
    pos_enc[1:, 0::2] = np.sin(pos_enc[1:, 0::2])  # dim 2i
    pos_enc[1:, 1::2] = np.cos(pos_enc[1:, 1::2])  # dim 2i+1
    return pos_enc<jupyter_output><empty_output><jupyter_text>**Transformer Encoder Block: Break Out Room  Activity🎊🎉** In this activity you will build out a Transfomer Encoder block from scratch. We will use this as the building block for our mini BERT model when you finish.<jupyter_code>def transfomer_encoder_block(query, key, value, embedding_dim, num_heads, ff_dim):

  # 1) Add a keras.layers.MultiHeadAttention Layer
  # Set num_heads
  # Set key_dim = embedding_dim // num_heads
  # Pass in the (query, key, value) to this layer to get the attention_output
  attention_output = ...

  # 2) Add Dropout(0.1) to the attention_output
  attention_output = ...

  # 3) Normalization + Residual Connection
  # Add a keras.layers.LayerNormalization with epsilon=1e-6
  # Pass in the query and attention_output to the LayerNormalization
  attention_output = ...

  # 4) Feedforward Layer
  # Add a Dense layer with size ff_dim and activation="relu"
  # Pass in attention_output to get the ffn_output
  # Add another Dense layer with size of embedding_dim and pass in the ffn_output
  ffn_output = ...

  # 5) Add Dropout(0.1) to the ffn_output
  ffn_output = ...

  # 6) Normalization + Residual Connection
  # Add a keras.layers.LayerNormalization with epsilon=1e-6
  # Pass in the attention_output and ffn_output to the LayerNormalization to get the final sequence_output
  sequence_output = ...


  return sequence_output<jupyter_output><empty_output><jupyter_text>Solution  Code ```def transfomer_encoder_block(query, key, value, embedding_dim, num_heads, ff_dim):     MultiHeadAttention Layer    attention_output = keras.layers.MultiHeadAttention(num_heads=num_heads,                                                       key_dim=embedding_dim // num_heads)(query, key, value)     Normalization + Residual Connection    attention_output = keras.layers.Dropout(0.1)(attention_output)    attention_output = keras.layers.LayerNormalization(epsilon=1e-6)(query + attention_output)     Feedforward Layer    ffn_output = keras.layers.Dense(units=ff_dim, activation="relu")(attention_output)    ffn_output = keras.layers.Dense(units=embedding_dim)(ffn_output)    ffn_output = keras.layers.Dropout(0.1)(ffn_output)     Normalization + Residual Connection     sequence_output = keras.layers.LayerNormalization(epsilon=1e-6)(attention_output + ffn_output)    return sequence_output```  **Masked Language Model**<jupyter_code># Loss Tracker
loss_tracker = tf.keras.metrics.Mean(name="loss")
loss_fn = keras.losses.SparseCategoricalCrossentropy(reduction=tf.keras.losses.Reduction.NONE)

class MaskedLanguageModel(keras.Model):

  def train_step(self, inputs):
      if len(inputs) == 3:
          features, labels, sample_weight = inputs
      else:
          features, labels = inputs
          sample_weight = None

      with tf.GradientTape() as tape:
          predictions = self(features, training=True)
          loss = loss_fn(labels, predictions[0], sample_weight=sample_weight)

      # Compute gradients
      trainable_vars = self.trainable_variables
      gradients = tape.gradient(loss, trainable_vars)

      # Update weights
      self.optimizer.apply_gradients(zip(gradients, trainable_vars))

      # Compute our own metrics
      loss_tracker.update_state(loss, sample_weight=sample_weight)

      # Return a dict mapping metric names to current value
      return {"loss": loss_tracker.result()}

  @property
  def metrics(self):
      return [loss_tracker]
def create_masked_language_bert_model(vocab_size,embedding_dim, sequence_length,num_layers, num_heads,ff_dim, enable_pe=True):
  # Model input
  model_input = keras.layers.Input(shape=(sequence_length), dtype=tf.int64)

  embeddings = keras.layers.Embedding(input_dim=vocab_size, output_dim=embedding_dim)(model_input)
  if enable_pe:
    embeddings = embeddings + generate_positional_encoding(sequence_length,embedding_dim)
  
  encoder_output = embeddings
  for i in range(num_layers):
      encoder_output = transfomer_encoder_block(encoder_output, encoder_output, encoder_output, embedding_dim,num_heads, ff_dim)

  # Output Layer
  output = keras.layers.Dense(units=vocab_size, activation="softmax")(encoder_output)

  # Create Model
  model = MaskedLanguageModel(inputs=[model_input], outputs=[output,encoder_output], name="masked_bert_model")

  return model<jupyter_output><empty_output><jupyter_text>**Train with Positional Encoding**<jupyter_code>############################
# Training Params
############################
epochs = 10
learning_rate = 0.001
embedding_dim = 256
sequence_length = max_len
num_layers = 2
num_heads = 8
ff_dim = 256

# Free up memory
K.clear_session()

# Build the model
model_w_pe = create_masked_language_bert_model(vocabulary_size,embedding_dim, sequence_length,num_layers,num_heads,ff_dim, enable_pe=True)

# Print the model architecture
print(model_w_pe.summary())

# Optimizer
optimizer = keras.optimizers.Adam(learning_rate=learning_rate)

# Compile
model_w_pe.compile(optimizer=optimizer)

# Train model
start_time = time.time()
training_results = model_w_pe.fit(
        train_data,
        epochs=epochs,
        verbose=1)
execution_time = (time.time() - start_time)/60.0
print("Training execution time (mins)",execution_time)<jupyter_output>Model: "masked_bert_model"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_1 (InputLayer)            [(None, 66)]         0                                            
__________________________________________________________________________________________________
embedding (Embedding)           (None, 66, 256)      289280      input_1[0][0]                    
__________________________________________________________________________________________________
tf.__operators__.add (TFOpLambd (None, 66, 256)      0           embedding[0][0]                  
__________________________________________________________________________________________________
multi_head_attention (MultiHead (None, 66, 256)      263168      tf.__operators__.[...]<jupyter_text>**Evaluate**Let us look at what exaclty is the language model predicting.<jupyter_code>sample_tokens = text_vectorizer(["convolutional [mask] networks have been applied to fields including computer vision"])
sample_tokens = sample_tokens.numpy()
print(sample_tokens)
# Make a prediction
predictions = model_w_pe.predict(sample_tokens)

# Get the first output from the model
mask_predictions = predictions[0]
encoder_output = predictions[1]
masked_index = np.where(sample_tokens == mask_token_id)
masked_index = masked_index[1]
mask_prediction = mask_predictions[0][masked_index]
print("mask_prediction:", mask_prediction)

top_indices = mask_prediction[0].argsort()[-5 :][::-1]
values = mask_prediction[0][top_indices]

def decode(tokens):
  return " ".join([index_word[t] for t in tokens if t != 0])

for i in range(len(top_indices)):
  p = top_indices[i]
  v = values[i]
  tokens = np.copy(sample_tokens[0])
  tokens[masked_index[0]] = p
  print("Prediction:",decode(tokens),"- Probability:",v, "- Mask token:",index_word[p])<jupyter_output>mask_prediction: [[4.02807254e-09 3.36504975e-08 5.23967714e-09 ... 2.05522133e-06
  3.68732032e-08 1.27696325e-08]]
Prediction: convolutional networks networks have been applied to fields including computer vision - Probability: 0.8604023 - Mask token: networks
Prediction: convolutional were networks have been applied to fields including computer vision - Probability: 0.074843526 - Mask token: were
Prediction: convolutional ai networks have been applied to fields including computer vision - Probability: 0.03147794 - Mask token: ai
Prediction: convolutional or networks have been applied to fields including computer vision - Probability: 0.003195785 - Mask token: or
Prediction: convolutional neural networks have been applied to fields including computer vision - Probability: 0.0031564995 - Mask token: neural<jupyter_text>**Train with NO Positional Encoding**<jupyter_code>############################
# Training Params
############################
epochs = 10
learning_rate = 0.001
embedding_dim = 256
sequence_length = max_len
num_layers = 2
num_heads = 8
ff_dim = 256

# Free up memory
K.clear_session()

# Build the model
model_no_pe = create_masked_language_bert_model(vocabulary_size,embedding_dim, sequence_length,num_layers,num_heads,ff_dim, enable_pe=False)

# Optimizer
optimizer = keras.optimizers.Adam(learning_rate=learning_rate)

# Compile
model_no_pe.compile(optimizer=optimizer)

# Train model
start_time = time.time()
training_results = model_no_pe.fit(
        train_data,
        epochs=epochs,
        verbose=1)
execution_time = (time.time() - start_time)/60.0
print("Training execution time (mins)",execution_time)<jupyter_output>Epoch 1/10
143/143 [==============================] - 128s 876ms/step - loss: 5.8659
Epoch 2/10
143/143 [==============================] - 125s 872ms/step - loss: 4.2099
Epoch 3/10
143/143 [==============================] - 124s 870ms/step - loss: 3.1755
Epoch 4/10
143/143 [==============================] - 125s 874ms/step - loss: 2.6924
Epoch 5/10
143/143 [==============================] - 125s 874ms/step - loss: 2.4065
Epoch 6/10
143/143 [==============================] - 125s 876ms/step - loss: 2.1758
Epoch 7/10
143/143 [==============================] - 125s 877ms/step - loss: 1.9950
Epoch 8/10
143/143 [==============================] - 125s 872ms/step - loss: 1.8559
Epoch 9/10
143/143 [==============================] - 125s 871ms/step - loss: 1.7486
Epoch 10/10
143/143 [==============================] - 125s 872ms/step - loss: 1.6759
Training execution time (mins) 20.861590536435447<jupyter_text>**Evaluate Postional Encoding**<jupyter_code>sample_tokens1 = text_vectorizer(["pavlos taught positional encoding to shivas"])
sample_tokens1 = sample_tokens1.numpy()
print(sample_tokens1)

sample_tokens2 = text_vectorizer(["shivas taught positional encoding to pavlos"])
sample_tokens2 = sample_tokens2.numpy()
print(sample_tokens2)
# Make a prediction on model with pe
predictions1 = model_w_pe.predict(sample_tokens1)

# Get the second output from the model
encoder_output1 = predictions1[1]
print("encoder_output shape:",encoder_output1.shape)
print(encoder_output1[:,:6,:10])
# Make a prediction on model with pe
predictions2 = model_w_pe.predict(sample_tokens2)

# Get the second output from the model
encoder_output2 = predictions2[1]
print("encoder_output shape:",encoder_output2.shape)
print(encoder_output2[:,:6,:10])
# Make a prediction on model without pe
predictions3 = model_no_pe.predict(sample_tokens1)

# Get the second output from the model
encoder_output3 = predictions3[1]
print("encoder_output shape:",encoder_output3.shape)
print(encoder_output3[:,:6,:10])
# Make a prediction on model without pe
predictions4 = model_no_pe.predict(sample_tokens2)

# Get the second output from the model
encoder_output4 = predictions4[1]
print("encoder_output shape:",encoder_output4.shape)
print(encoder_output4[:,:6,:10])<jupyter_output>encoder_output shape: (1, 66, 256)
[[[ 0.59668255  0.25934428 -1.1342949  -2.2415476   0.76485896
    0.50917    -1.1315589   0.01400714 -0.5282433  -0.3171306 ]
  [-0.11713661  0.42059946 -2.520322   -1.1835688   0.19510822
    0.24005441 -1.400293   -1.4046291  -0.23848428  0.38379204]
  [ 0.50030816  0.24928322 -0.8094312  -1.5947442   2.4495683
    0.6930185   0.9870634  -1.2392     -1.4726596   1.7183204 ]
  [-0.5274049  -2.1464505  -1.8480324  -1.4731193   0.11264288
   -1.5234902  -1.5618031   0.36495188  0.04311167  1.7619015 ]
  [-0.46989337  1.3999724  -2.9341397  -2.4048123   0.18449481
    0.8569854  -1.4570559  -0.6724977   0.9933045   0.57962257]
  [-0.03209894  0.6816822  -1.5825441  -2.5832229   0.4955349
    0.486806    0.16487104 -0.1900789   1.3275807  -0.3003537 ]]]<jupyter_text>Check if the word `pavlos` & `shivas` have the same represenation in the two sentences:* pavlos taught positional encoding to shivas* shivas taught positional encoding to pavlos<jupyter_code>print("Model with Postional Encoding:")
print("pavlos:",np.array_equal(encoder_output1[0,1], encoder_output2[0,6]))
print("shivas:",np.array_equal(encoder_output1[0,6], encoder_output2[0,1]))
print("Model without Postional Encoding:")
print("pavlos:",np.array_equal(encoder_output3[0,1], encoder_output4[0,6]))
print("shivas:",np.array_equal(encoder_output3[0,6], encoder_output4[0,1]))
# Generate postional encodings
positional_encodings = generate_positional_encoding(sequence_length, embedding_dim)
print(positional_encodings.shape)
print(positional_encodings)
# Lets compute the dot product of postion 1 with every other postions
dot_results = []
for idx in range(positional_encodings.shape[0]):
  dot_results.append(np.dot(positional_encodings[1], positional_encodings[idx]))

print(dot_results)
# Plot the dot product results
fig = plt.figure(figsize=(20,5))
axs = fig.add_subplot(1,3,1)
axs.plot(np.arange(0, positional_encodings.shape[0]), dot_results)
axs.set_xlabel('Positions')
axs.set_ylabel('Dot product of positions')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1258127393917915, 'fraction_numerical': 0.03621716669165656, 'mean_word_length': 4.666048810627124, 'pattern_counts': {'":': 20, '<': 216, '<?xml version=': 0, '>': 225, 'https://': 4, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 40, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '43', 'dir': 'jupyter-structured-clean-dedup', 'id': '227462', 'license': 'MIT', 'n_tokens_mistral': 26216, 'n_tokens_neox': 24004, 'n_words': 5830, 'path': 'content/sections/sec05/cs109b_sec5.ipynb', 'repo_name': 'Harvard-IACS/2021-CS10B', 'size': '73395'}