<jupyter_start><jupyter_text>Smart open-domain Chatbot using deep learning  Attention mechanism 


  Lesson Goals
This lesson is the continuation of big data natural language text processing where we took three dataset from Answers.Com, Yahoo Answers, Quora API. In this lesson, I will show you how to develop smart open-domain chatbot by using deep learning Attention mechanism.
 Prerequests

*   if your system has GPU you can run this example on your system but if not you can run it in GCP.

  Getting StartedImport modules:<jupyter_code>from sklearn.model_selection import train_test_split

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import math
import os
import time<jupyter_output><empty_output><jupyter_text>Data Preparation: Load up the data<jupyter_code>def load_dataset(path):
    """
    Reads the data from the csv file.
    Arguments:
        path: a string.
    Returns:
        dataset: a Pandas Dataframe with two columns, namely: `Question` and `Answer`.
        X: a NumPy array representing `Question` column.
        Y: a NumPy array representing `Answer` column.
    """
    
    # read the csv file into a pandas dataframe
    dataset = pd.read_csv(path, usecols=['Question', 'Answer'])
    # make sure all cell values are strings; because some of them 
    # only contain numbers, so they maybe mistaken with other types.
    dataset = dataset.applymap(str)
    # shuffle the rows of the dataframe and then reset the index
    dataset = dataset.sample(frac=1).reset_index(drop=True)
    
    X = np.asarray(dataset['Question'])
    Y = np.asarray(dataset['Answer'])
    
    X = np.apply_along_axis(lambda sen: '<start> '+ sen + ' <end>', 0, X)
    Y = np.apply_along_axis(lambda sen: '<start> '+ sen + ' <end>', 0, Y)
    
    return dataset, X, Y
os
os.getcwd
os.getcwd() 

!mkdir path
cd path/
from google.colab import files
uploaded = files.upload()
files_path = '/content/path/'
qa_dataframe, X, Y = load_dataset(files_path + 'dataset.csv')
qa_dataframe.head()
print(f"Number of question-answer pairs in the dataset: {len(qa_dataframe)}")<jupyter_output>Number of question-answer pairs in the dataset: 176980<jupyter_text>Cache all hyperparameters into this dictionary:<jupyter_code>hyperparameters = dict()<jupyter_output><empty_output><jupyter_text>**Choose the size of vocabulary**:This is a hyperparameter that you can play with.<jupyter_code>VOCAB_SIZE = 10000<jupyter_output><empty_output><jupyter_text>Now, let's tokenize the data, by converting each sentence (question or answer) to a sequence of integers which represent indices of their embeddings.<jupyter_code>def tokenize(sentences, vocab_size):
  
    """
    Using Tensorflow Tokenizer to turn each sentence into a sequence of integers 
    (each integer being the index of a token in a dictionary).
    Arguments:
      X: a list or a NumPy array of strings, where each element is a sentence.
    Returns:
      tensor: a NumPy ndarray, where each row represents the the sequence of
              integers that maps the words of the equivalent sentence in 
              the `sentences` list to a their indices (for embbedings). shape=(batch_size, )
      lang_tokenizer: a Tensorflow Tokenizer which have been fit on `sentences`.
    """ 

    lang_tokenizer = tf.keras.preprocessing.text.Tokenizer(num_words=vocab_size, 
                                                           filters='')
    lang_tokenizer.fit_on_texts(sentences)

    tensor = lang_tokenizer.texts_to_sequences(sentences)
    tensor = tf.keras.preprocessing.sequence.pad_sequences(tensor, padding='post')

    return tensor, lang_tokenizer
import tensorflow as tf

# all sentences (Questions & Answers)
# we need to fit the tokenizer on all sentences; to create word_index and 
# index_word mapper dictionaries for the most frequent VOCAB_SIZE= 10,000 words.
texts = np.concatenate((X, Y))
# tokenize the data
tensor, text_tokenizer = tokenize(texts, VOCAB_SIZE)

print(f"texts[0]= {texts[0]}")
print(f"tensor[0]= {tensor[0]}")<jupyter_output>texts[0]= <start> what do you call a car if you don ' t know its gender ? <end>
tensor[0]= [   1   10   16    8   38    5  173   51    8   60    4   21   58  197
 1466    7    2    0    0    0    0    0    0    0    0    0    0    0
    0    0    0    0]<jupyter_text>As we see, each sentence has been converted to an array with indices that will be used to map words to their embedding vectors.Now, let's extract back X (for questions) and Y (for answers) arrays from tensor:<jupyter_code># extract questions and answers back from tensor
X, Y = tensor[:len(X)], tensor[len(X):]

print(f"Shape of X (questions): {X.shape}")
print(f"Shape of Y (answers): {Y.shape}")<jupyter_output>Shape of X (questions): (176980, 32)
Shape of Y (answers): (176980, 32)<jupyter_text>Get the maximum sequence length for both input and target tensors:<jupyter_code>def max_seq_length(tensor):
    """
    Get maximum sequence length in the corpus. And, make sure that all rows in
    `tensor` has the same length.
    Arguments:
      tensor: a NumPy ndarray of shape (batch_size,) where each row represents
      indices mapping to words in equivalent sentences.
    Returns:
      max_len: an integer representing maximum sequence length.
    """
    batch_size = len(tensor)
    lengths = [len(sentence) for sentence in tensor]
    max_len = max(lengths)

    # check if all rows in `tensor` has the same length (equal to max_len)
    assert lengths == [max_len]*batch_size

    return max_len
# Get max_seq_length of input and target tensors
max_length_inp, max_length_targ = max_seq_length(X), max_seq_length(Y)
print(f"Maximum sequence length for input (questions) tensor: {max_length_inp}")
print(f"Maximum sequence length for target (answers) tensor: {max_length_targ}")<jupyter_output>Maximum sequence length for input (questions) tensor: 32
Maximum sequence length for target (answers) tensor: 32<jupyter_text>Save the arrays as npy files:<jupyter_code>data_arrays_path = os.path.join(files_path, 'data_arrays')
# create the folder if it does not exist
if not os.path.exists(data_arrays_path):
    os.makedirs(data_arrays_path)
    
np.save(os.path.join(data_arrays_path, 'X.npy'), X)
np.save(os.path.join(data_arrays_path, 'Y.npy'), Y)
def read_glove_vectors(glove_file):
    """
    This function reads GloVe vectors from .txt file and 
    returns a word to vector dictionary.
    Arguments:
      glove_file: a string path to GloVe word embeddings file.
    Returns:
      word_to_vec: a Python dictionary that maps words to their embeddings.
    """

    import numpy as np
    
    # open the file
    with open(glove_file, 'r', encoding="utf-8") as f:
        
        words = set()
        word_to_vec = {}

        # loop over the rows in the file
        for line in f:
            # read the line, strip it (remove leading and trailing spaces) and split it
            line = line.strip().split()
            # first item in the list 'line' is the word itself
            curr_word = line[0]
            # add the word to set of words
            words.add(curr_word)
            # add the words with its vector representation as a (key, value) pair to the dictionary
            word_to_vec[curr_word] = np.array(line[1:], dtype=np.float64)

    return word_to_vec<jupyter_output><empty_output><jupyter_text>Download GloVe Vectors<jupyter_code>!wget http://nlp.stanford.edu/data/glove.6B.zip
!unzip glove.6B.zip
!ls
!pwd
glove_file = "/content/path/glove.6B.200d.txt"
word_to_vec = read_glove_vectors(glove_file)<jupyter_output><empty_output><jupyter_text>The Model**Choosing Hyperparameters:**<jupyter_code>BUFFER_SIZE = X.shape[0]
BATCH_SIZE = 128
steps_per_epoch = BUFFER_SIZE//BATCH_SIZE 
embedding_dim = 200
units = 512 
vocab_size = VOCAB_SIZE + 1
print(f"Buffer size: {BUFFER_SIZE}, Batch size: {BATCH_SIZE}, Steps per epoch: {steps_per_epoch}")
print(f"Embedding size: {embedding_dim}, # of units: {units}")
print(f"Vocab size: {vocab_size}")
# cache these values into the hyperparameters dictionary
hyperparameters['buffer_size'] = BUFFER_SIZE
hyperparameters['batch_size'] = BATCH_SIZE
hyperparameters['steps_per_epoch'] = steps_per_epoch
hyperparameters['embedding_dim'] = embedding_dim
hyperparameters['units'] = units
hyperparameters['vocab_size'] = vocab_size<jupyter_output><empty_output><jupyter_text>Create the embedding matrix for words in the vocabulary:<jupyter_code>def create_embedding_matrix(words, word_to_vec, vocab_size, emb_dim):
  """
  Returns an embeddings matrix for the words in the vocabulary.
  Arguments:
      words: a list of words.
      word_to_vec: a dictionary that maps words to their embedding vectors.
      vocab_size: an integer which represents the size of the vocabulary.
      emb_dim: an integer which represents the dimension of word embeddings.
  Returns:
      embedding_matrix: a NumPy array with shape of (vocab_size, emb_dim).
  """

  # create embedding matrix
  embedding_matrix = np.zeros((vocab_size, emb_dim), dtype=np.float64)

  # loop over the words in our vocabulary
  for i, word in enumerate(words):
    if word in word_to_vec.keys():
      # if the current word is in glove vocab, get its glove vector.
      embedding_matrix[i, :] = word_to_vec[word]
    else:
      # if the current word does not exist in the vocabulary, set its vector to zeros.
      embedding_matrix[i, :] = np.zeros((emb_dim,), dtype=np.float64)

  return embedding_matrix
# since Tensorflow tokenizer word_index still have all words even when 
# vocab_size is passed while defining the Tokenizer. 
# So, we need to grab the first 10,000 words.
words = list(text_tokenizer.word_index.keys())[:vocab_size]
embedding_matrix = create_embedding_matrix(words, word_to_vec, vocab_size, 
                                           embedding_dim)
print(f"Embedding matrix shape: {embedding_matrix.shape}")
# save the embedding matrix
np.save(os.path.join(data_arrays_path, 'embedding_matrix.npy'), embedding_matrix)<jupyter_output><empty_output><jupyter_text>Create a tf.data dataset for X and Y with buffer size equal to BUFFER_SIZE and batch size equal to BATCH_SIZE, we will use this dataset to generate batches while training:<jupyter_code>dataset = tf.data.Dataset.from_tensor_slices((X, Y)).shuffle(BUFFER_SIZE)
dataset = dataset.batch(BATCH_SIZE)<jupyter_output><empty_output><jupyter_text>Example of an input batch that the model will receive while iterating over dataset batches:<jupyter_code>example_input_batch, example_target_batch = next(iter(dataset))
example_input_batch.shape, example_target_batch.shape<jupyter_output><empty_output><jupyter_text>Model ArchitectureI'm going to use an encoder-decoder model with Bahdanau's attention, which has been described in detail in this paper:[Effective Approaches to Attention-based Neural Machine Translation](https://)Some properties of the model:The encoder will be a bidirectional encoder with 512 hidden units for each direction, resulting in 1024 cells.Both encoder & decoder will use LSTM as the cell.The decoder will be unidirectional with 1024 hidden units.**The Encoder**Define the bidirectional encoder architecture which will consist of the following:   1. An Embedding layer that will map input sentences to their embeddings. The vocabulary size is equal to VOCAB_SIZE=10,000 and the dimension of embedding is 200 (Note: although a power of 2 embedding size would be more suitable to speed up training time by increasing cache utilization during data movement, thus reducing bottlenecks, but GloVe embeddings do not come with a power of 2 embedding size).   2. A forward & backward (bidirectional) LSTM layer with 512 units for each.The input to the encoder has a shape of (batch_size, ), which is the input array of sentences (questions).The hidden state arrays of the encoder are all of the shape (batch_size, n_units).The output of the encoder has a shape of (batch_size, max_sequence_length, n_units x2) ;the hidden size of the output is equal to double the size of units since the encoder is bidirectional.<jupyter_code>class Encoder(tf.keras.Model):

    def __init__(self, vocab_size, embedding_dim, embedding_matrix, enc_units, 
                 batch_sz):
        super(Encoder, self).__init__()
        self.batch_sz = batch_sz
        self.enc_units = enc_units
        self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim, 
                                                   weights=[embedding_matrix], 
                                                   trainable=True)
        self.bi_lstm = tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(self.enc_units,
                                       return_sequences=True,
                                       return_state=True,
                                       recurrent_initializer='glorot_uniform'))

    def call(self, x, hidden):
        """
        Propogates the input `x` through the bidirectional encoder and returns
        the outputs along with hidden states.
        Arguments:
            x: a tensor with shape (batch_size, max_seq_length)
            hidden: a tuple or a list, with four tensors representing hidden
                    and memory states for both the forward and backward LSTMs.
                    Each of them having a shape of (batch_size, max_seq_length)
        Returns:
            output: a tensor representing the output of the encoder with a shape
                    of (batch size, max_sequence length, units*2)
            state_h: a tensor, with forward and backward hidden states of 
                     the encoder with a shape of (batch size, units*2)
            state_c: a tensor, with forward and backward memory cell states of
                     the encoder with a shape of (batch size, units*2)
        """

        x = self.embedding(x)

        output, fstate_h, fstate_c, bstate_h, bstate_c = self.bi_lstm(x, initial_state = hidden)

        state_h = tf.keras.layers.Concatenate()([fstate_h, bstate_h])
        state_c = tf.keras.layers.Concatenate()([fstate_c, bstate_c])

        return output, state_h, state_c

    def initialize_hidden_state(self):
        # forward_state_h forward hidden state output, backward_state_h backward hidden state output
        # forward_state_c forward cell (memory) state output, backward_state_c backward cell (memory) state output
        return (tf.zeros((self.batch_sz, self.enc_units)), tf.zeros((self.batch_sz, self.enc_units)), 
                tf.zeros((self.batch_sz, self.enc_units)), tf.zeros((self.batch_sz, self.enc_units)))
import tensorflow as tf<jupyter_output><empty_output><jupyter_text>Define the encoder:<jupyter_code>encoder = Encoder(vocab_size, embedding_dim, embedding_matrix, units, 
                  BATCH_SIZE)

# usage example
sample_hidden = encoder.initialize_hidden_state()

sample_output, sample_h, sample_c = encoder(example_input_batch, sample_hidden)
print (f'Encoder output shape: (batch size, sequence length, units*2) {sample_output.shape}')
print (f'Encoder Hidden state shape: (batch size, units*2) {sample_h.shape}')
print (f'Encoder Memory state shape: (batch size, units*2) {sample_c.shape}')<jupyter_output>Encoder output shape: (batch size, sequence length, units*2) (128, 32, 1024)
Encoder Hidden state shape: (batch size, units*2) (128, 1024)
Encoder Memory state shape: (batch size, units*2) (128, 1024)<jupyter_text>Bahdanau Attention MechanismThe following diagram shows that each input words is assigned a weight by the attention mechanism which is then used by the decoder to predict the next word in the sentence. The below picture and formulas are an example of attention mechanism from Effective Approaches to Attention-based Neural Machine Translation.To learn more about Bahdanau's attention, you can refer to this paper: [Neural Machine Translation by Jointly Learning to Align and Translate](https://)<jupyter_code>class BahdanauAttention(tf.keras.layers.Layer):
  
    def __init__(self, units):
        super(BahdanauAttention, self).__init__()
        self.W1 = tf.keras.layers.Dense(units*2)
        self.W2 = tf.keras.layers.Dense(units*2)
        self.V = tf.keras.layers.Dense(1)

    def call(self, query, values):
        # query hidden state shape == (batch_size, hidden size)
        # query_with_time_axis shape == (batch_size, 1, hidden size)
        # values shape == (batch_size, max_len, hidden size)
        query_with_time_axis = tf.expand_dims(query, 1)
        
        # score shape == (batch_size, max_length, 1)
        # we get 1 at the last axis because we are applying score to self.V
        # the shape of the tensor before applying self.V is (batch_size, max_length, units)
        score = self.V(tf.nn.tanh(
            self.W1(query_with_time_axis) + self.W2(values)))

        # attention_weights shape == (batch_size, max_length, 1)
        attention_weights = tf.nn.softmax(score, axis=1)

        # context_vector shape after sum == (batch_size, hidden_size)
        context_vector = attention_weights * values
        context_vector = tf.reduce_sum(context_vector, axis=1)

        return context_vector, attention_weights<jupyter_output><empty_output><jupyter_text>Example of using Bahdanau's attention:<jupyter_code># usage example
attention_layer = BahdanauAttention(10)
attention_result, attention_weights = attention_layer(sample_h, sample_output)

print(f"Attention result shape: (batch size, units*2) {attention_result.shape}")
print(f"Attention weights shape: (batch_size, sequence_length, 1) {attention_weights.shape}")<jupyter_output>Attention result shape: (batch size, units*2) (128, 1024)
Attention weights shape: (batch_size, sequence_length, 1) (128, 32, 1)<jupyter_text>The DecoderDefine the decoder architecture which will consist of the following:  1. Bahdanau's attention which is applied to the output of the encoder to get the context which will be passed, along with the embeddings of the decoder input, to the LSTM layer of the decoder.  2. An Embedding layer with a vocabulary size of VOCAB_SIZE=10,000 words and an embedding dimension of 200. The embedding takes the decoder input which is of the shape (batch_size, ) and outputs a tensor of the shape (batch_size, 1, embedding_dim).   3. A forward (unidirectional) LSTM layer with 1024 units for each.The shape of the decoder's output is (batch_size, vocab_size) and the decoder hidden state size is of the shape (batch_size, units x2).<jupyter_code>class Decoder(tf.keras.Model):

    def __init__(self, vocab_size, embedding_dim, embedding_matrix, dec_units, 
                 batch_sz):
        super(Decoder, self).__init__()
        self.batch_sz = batch_sz
        self.dec_units = dec_units
        self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim,
                                                   weights=[embedding_matrix],
                                                   trainable=True)
        self.lstm = tf.keras.layers.LSTM(self.dec_units,
                                       return_sequences=True,
                                       return_state=True,
                                       recurrent_initializer='glorot_uniform')
        
        self.fc = tf.keras.layers.Dense(vocab_size)

        self.attention = BahdanauAttention(self.dec_units)

    def call(self, x, hidden, enc_output):
        """
        Takes decoder input, hidden state and output of the encoder and
        returns the decoder's predictions along with decoder hidden state and
        attention weights.
        Arguments:
            x: a tensor with shape (batch_size, 1) which is the decoder input
               at some timestep.
            hidden: a tensor representing hidden state of the encoder with a
                    shape of (batch size, units*2)
            enc_output: a tensor representing the encoder's output with a shape
                        of (batch size, sequence length, units*2)
        Returns:
            x: a tensor with shape (batch_size, vocab size) and it's decoder's
               output.
            state: a tensor that represents the hidden state of the decoder 
                   and has a shape of (batch_size, units*2)
            attention_weights: a tensor that represents attention weights and 
                               has a shape of (batch_size, sequence_length, 1)
        """


        # enc_output shape == (batch_size, max_length, hidden_size)
        context_vector, attention_weights = self.attention(hidden, enc_output)
        
        # x shape after passing through embedding == (batch_size, 1, embedding_dim)
        x = self.embedding(x)

        # x shape after concatenation == (batch_size, 1, embedding_dim + hidden_size)
        x = tf.concat([tf.expand_dims(context_vector, 1), x], axis=-1)
        
        # passing the concatenated vector to the LSTM
        output, state_h, state_c = self.lstm(x)

        state = state_h

        # output shape == (batch_size * 1, hidden_size)
        output = tf.reshape(output, (-1, output.shape[2]))

        # output shape == (batch_size, vocab)
        x = self.fc(output)

        return x, state, attention_weights<jupyter_output><empty_output><jupyter_text>Define the decoder:<jupyter_code>decoder = Decoder(vocab_size, embedding_dim, embedding_matrix, units*2, 
                  BATCH_SIZE)

# usage example
sample_decoder_output, dec_h, _ = decoder(tf.random.uniform((BATCH_SIZE, 1)),
                                      sample_h, sample_output)

print(f'Decoder output shape: (batch_size, vocab size) {sample_decoder_output.shape}')
print(f'Decoder hidden state shape: (batch_size, units*2) {dec_h.shape}')<jupyter_output>Decoder output shape: (batch_size, vocab size) (128, 10001)
Decoder hidden state shape: (batch_size, units*2) (128, 1024)<jupyter_text>Defining Optimizer, Loss Function and Metric Define the loss function:<jupyter_code>optimizer = tf.keras.optimizers.Adam()
loss_object = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True, 
                                                            reduction='none')

def compute_loss(real, pred):
    """
    This function returns the loss for model's predictions on a batch 
    of data in comparison with the real outputs at a timestep.
    Arguments:
        real: real output, a Tensorflow tensor with a shape 
              of: (batch_size, max_seq_length)
        pred: model's predictions at a certain timestep, a Tensorflow tensor 
              with a shape of: (batch_size, max_seq_length)
    Returns:
        A Tensorflow tensor with the loss.
    """

    mask = tf.math.logical_not(tf.math.equal(real, 0))
    loss_ = loss_object(real, pred)
    mask = tf.cast(mask, dtype=loss_.dtype)
    loss_ *= mask
    
    return tf.reduce_mean(loss_)<jupyter_output><empty_output><jupyter_text>Define the perplexity metric:**Note**: I was going to incorporate BLEU score as another metric, but to compute BLEU score on a batch of data, it turns out that we need access the data to count the number of n-grams. In order to do that, we can't use Tensorflow's AutoGraph feature tf.function. Anyways, I tried it and using tf.function was around 2.5 times faster than dropping this feature only to compute BLEU score. In case you want to use BLEU score, you can use nmt's open source implementation which you can find [here](https://), or you can use NLTK's implementation.<jupyter_code>def compute_perplexity(real, pred):
    """
    This function returns the perplexity for model's predictions on a batch 
    of data in comparison with the real outputs at a timestep.
    Arguments:
        real: real output, a Tensorflow tensor with a shape 
              of: (batch_size, max_seq_length)
        pred: model's predictions at a certain timestep, a Tensorflow tensor 
              with a shape of: (batch_size, max_seq_length)
    Returns:
        A Tensorflow tensor with the perplexity.
    """
    
    mask = tf.math.logical_not(tf.math.equal(real, 0))
    loss_ = loss_object(real, pred)
    mask = tf.cast(mask, dtype=loss_.dtype)
    loss_ *= mask
    
    return tf.cast(tf.pow(math.e, tf.keras.backend.mean(loss_, axis=-1)), 
                   dtype=tf.keras.backend.floatx())<jupyter_output><empty_output><jupyter_text>Checkpoints (Object-based saving)<jupyter_code>checkpoint_dir = '/content/path/training_checkpoints'

# create the folder if it does not exist
if not os.path.exists(checkpoint_dir):
    os.makedirs(checkpoint_dir)

checkpoint_prefix = os.path.join(checkpoint_dir, "ckpt")
checkpoint = tf.train.Checkpoint(optimizer=optimizer,
                                 encoder=encoder,
                                 decoder=decoder)<jupyter_output><empty_output><jupyter_text>TensorBoard<jupyter_code># Load the TensorBoard notebook extension
%load_ext tensorboard
log_file_name =f"/content/path/logs/metrics_{int(time.time())}"

# create the folder if it does not exist
if not os.path.exists(log_file_name):
    os.makedirs(log_file_name)
    
summary_writer = tf.summary.create_file_writer(log_file_name)<jupyter_output><empty_output><jupyter_text>Define training step:<jupyter_code>@tf.function
def train_step(inp, targ, enc_hidden):
    """
    This function performs a training step for the model on a batch of data.
    Arguments:
        inp: a tensor, the input to the encoder network which is a batch of 
              vectors of integers (indices of words) for sentences (questions) 
              with a shape of (batch_size, encoder_max_seq_len)
        targ: a tensor, the real output that the decoder will use to learn 
              using teacher forcing. It has a shape same as `inp`
              which is (batch_size, encoder_max_seq_len)
        enc_hidden: a tuple of four tensors, the initial hidden states for the
              encoder network, each of the shape (batch_size, n_units*2)
    Returns:
        batch_loss: loss for the given batch.
        batch_acc: accuracy for the given batch.
        batch_bleu: bleu score for the given batch.
        batch_ppl: perplexity for the given batch.
    """
    
    loss = 0
    ppl = 0

    with tf.GradientTape() as tape:
        # Run the input through the encoder and get back the encoder output
        # and the hidden states of the encoder.
        enc_output, enc_h, enc_c = encoder(inp, enc_hidden)
        
        # Set the hidden state of the encoder to be the initial hidden state
        # of the decoder.
        dec_hidden = enc_h

        # Define the decoder input which is basically the '<start>' token,
        # for every sentence in the batch.
        dec_input = tf.expand_dims([text_tokenizer.word_index['<start>']] 
                                   * BATCH_SIZE, 1)
        
        # Teacher forcing - feeding the target as the next input
        # looping over timesteps
        for t in range(1, targ.shape[1]):

            # Pass encoder output to the decoder along with the decoder
            # input and initial hidden state and get back the predictions
            # for this batch at the current timestep with the hidden state 
            # of the decoder
            predictions, dec_hidden, _ = decoder(dec_input, 
                                                 dec_hidden, 
                                                 enc_output)
            # predictions shape: (batch_size, decoder_vocab_size)
            # dec_hidden shape: (batch_size, units*2)
            # attention weights (3rd output that has been discarded) 
            # shape: (batch_size, decoder_max_seq_len, 1)

            # compute the loss
            loss += compute_loss(targ[:, t], predictions)

            # compute the perplexity
            ppl += compute_perplexity(targ[:, t], predictions)

            # using teacher forcing
            dec_input = tf.expand_dims(targ[:, t], 1)

    # compute the loss for the batch
    batch_loss = (loss / int(targ.shape[1]))
    
    # compute the perplexity for the batch
    batch_ppl = (ppl / int(targ.shape[1]))

    variables = encoder.trainable_variables + decoder.trainable_variables

    # get gradients
    gradients = tape.gradient(loss, variables)

    optimizer.apply_gradients(zip(gradients, variables))

    return batch_loss, batch_ppl
# Nicely formatted time string
def hms_string(sec_elapsed):
    h = int(sec_elapsed / (60 * 60))
    m = int((sec_elapsed % (60 * 60)) / 60)
    s = sec_elapsed % 60
    return f"{h}:{m}:{round(s,1)}"<jupyter_output><empty_output><jupyter_text>Training the model:<jupyter_code>print(hyperparameters)<jupyter_output>{'buffer_size': 176980, 'batch_size': 128, 'steps_per_epoch': 1382, 'embedding_dim': 200, 'units': 512, 'vocab_size': 10001}<jupyter_text>Let's train the model:<jupyter_code>EPOCHS = 10

start_time = time.time()

# to cache loss and perlexity over epochs
cache = dict({'train_loss': [], 'train_ppl':[]})

with tf.device('/device:GPU:0'):
    for epoch in range(EPOCHS):
        start_epoch = time.time()

        # Initialize encoder hidden state
        enc_hidden = encoder.initialize_hidden_state()

        total_loss = 0
        total_ppl = 0
        
        # Training the model using the training data
        for (batch, (inp, targ)) in enumerate(dataset.take(steps_per_epoch)):
            
            # Train the model on current batch
            (batch_loss, batch_ppl) = train_step(inp, targ, enc_hidden)

            total_loss += batch_loss
            total_ppl += batch_ppl  

            # print loss and perplexity for current batch
            if batch % 400 == 0:
              print(f"Epoch {epoch + 1}/{EPOCHS} - "
                      f"batch: {batch}/{steps_per_epoch} - "
                      f"loss: {batch_loss.numpy()} - ppl: {batch_ppl}")
        
        # compute batch loss and perplexity
        total_loss = total_loss / steps_per_epoch
        total_ppl = total_ppl / steps_per_epoch     

        # Log loss and perplexity to TensorBoard for current epoch
        with summary_writer.as_default():
          tf.summary.scalar('training_loss', total_loss, step=epoch)
          tf.summary.scalar('training_perplexity', total_ppl, step=epoch)

        # Save (checkpoint) the model every 15 epochs
        if ((epoch+1) > 1) and ((epoch+1) % 15 == 0):
          checkpoint.save(file_prefix = checkpoint_prefix)
          print(f'Saved checkpoint for epoch {epoch+1}/{EPOCHS} to: {checkpoint_prefix}')
        

        # cache the loss and perplexity for current epoch
        cache['train_loss'].append(total_loss)
        cache['train_ppl'].append(total_ppl)

        # print loss and perplexity for current epoch
        print(f"Epoch {epoch + 1}/{EPOCHS} - loss: {total_loss} - "
              f"ppl: {total_ppl}")
        
        print(f"Time taken for epoch ({epoch + 1}): "
              f"{time.time() - start_epoch} sec\n")

execution_time = (time.time() - start_time)
print(f'Elapsed time: {hms_string(execution_time)}')<jupyter_output>Epoch 1/10 - batch: 0/1382 - loss: 3.0469369888305664 - ppl: 967.1176147460938
Epoch 1/10 - batch: 400/1382 - loss: 1.3774396181106567 - ppl: 18.1285400390625
Epoch 1/10 - batch: 800/1382 - loss: 1.372939109802246 - ppl: 13.611625671386719
Epoch 1/10 - batch: 1200/1382 - loss: 1.372182011604309 - ppl: 11.62140941619873
Epoch 1/10 - loss: 1.4264665842056274 - ppl: 21.072736740112305
Time taken for epoch (1): 1093.9495713710785 sec

Epoch 2/10 - batch: 0/1382 - loss: 1.2829258441925049 - ppl: 9.259562492370605
Epoch 2/10 - batch: 400/1382 - loss: 1.1151574850082397 - ppl: 9.946002006530762
Epoch 2/10 - batch: 800/1382 - loss: 1.1067540645599365 - ppl: 8.797042846679688
Epoch 2/10 - batch: 1200/1382 - loss: 1.1032754182815552 - ppl: 6.854518890380859
Epoch 2/10 - loss: 1.1896915435791016 - ppl: 8.73717975616455
Time taken for epoch (2): 1063.7512044906616 sec

Epoch 3/10 - batch: 0/1382 - loss: 1.0923715829849243 - ppl: 5.646231651306152
Epoch 3/10 - batch: 400/1382 - loss: 1.037064433097[...]<jupyter_text>Plot metrics over time:<jupyter_code>def plot_metrics(cache):
    """
    Plots loss and perplexity over epochs:
    Arguemnts:
        cache: a dictionary that contains values over epochs
               for both loss and perplexity.
    """
    
    # since, items of cache['train_loss'] and cache['train_ppl']
    # are tensors, so let's extract values from these tensors.
    loss = [tensor.numpy() for tensor in cache['train_loss']]
    ppl = [tensor.numpy() for tensor in cache['train_ppl']]
    
    # Plot the loss
    plt.figure()
    plt.plot(loss, label='Training Loss')
    plt.title('Loss')
    plt.xlabel('Epochs')
    plt.show()

    # Plot the perplexity
    plt.figure()
    plt.plot(ppl, label='Training Accuracy')
    plt.title('Perplexity')
    plt.xlabel('Epochs')
    plt.show()
plot_metrics(cache)<jupyter_output><empty_output><jupyter_text>Inference ModeNow, it's time to try the model and ask him a few questins :)Helper function: to clean the text the user has entered.<jupyter_code>def clean_text(text):
    """
    This method takes a string, applies different text preprocessing 
    (characters replacement, removal of unwanted characters, removal of extra 
    whitespaces) operations and returns a string.
    Arguments:
      text: a string.
    Returns:
      a cleaned version of text.
    """
    
    import re
    
    text = str(text)
    
    # REPLACEMENT
    text = re.sub('\"', '\'', text)
    text = re.sub("“", '\'', text)
    text = re.sub("”", '\'', text)
    text = re.sub('’', '\'', text)
    text = re.sub('\[','(', text)
    text = re.sub('\]',')', text)
    text = re.sub('\{','(', text)
    text = re.sub('\}',')', text)
    text = re.sub("([?.!,:;'?!+\-*/=%$@&()])", r" \1 ", text)

    pattern = re.compile('[^a-zA-Z0-9_\.\,\:\;\'\?\!\+\-\*\/\=\%\$\@\&\(\)]')
    # remove unwanted characters
    text = re.sub(pattern, ' ', text)
    
    # lower case the characters in the string
    text = text.lower()
    
    # REMOVAL OF EXTRA WHITESPACES
    # remove duplicated spaces
    text = re.sub(' +', ' ', text)
    # remove leading and trailing spaces
    text = text.strip()
    
    return text<jupyter_output><empty_output><jupyter_text>The following fucntion passes a sentence that has been entered by the user to the model and returns its answer along with the input.<jupyter_code>def evaluate(sentence):
    """
    This function takes a sentence (question) and returns the model's output
    (answer) to it.
    Arguemnts:
      sentence: a string.
    Returns:
      result: a string, representing model's output to the input.
      sentence: a string, the input sentence.
    """

    # clean the input sentence (question) to prepare for the encoder
    sentence = clean_text(sentence)
    sentence = '<start> ' + sentence + ' <end>'

    # tokenize the input sentence and pad zeros if its length is less than
    # maximum sequence length.
    inputs = [text_tokenizer.word_index[i] for i in sentence.split(' ')]
    inputs = tf.keras.preprocessing.sequence.pad_sequences([inputs],
                                                         maxlen=max_length_inp,
                                                         padding='post')
  
    inputs = tf.convert_to_tensor(inputs)

    result = ''

    # initilize the hidden state of the encoder
    enc_hidden = (tf.zeros((1, units)), 
                  tf.zeros((1, units)), 
                  tf.zeros((1, units)), 
                  tf.zeros((1, units)))
    
    enc_output, enc_hidden, enc_c = encoder(inputs, enc_hidden)
    
    dec_hidden = enc_hidden

    dec_input = tf.expand_dims([text_tokenizer.word_index['<start>']], 0)
    
    # generate answer, where the maximum length for the answer is equal
    # to max_length_targ=32
    for t in range(max_length_targ):
      predictions, dec_hidden, attention_weights = decoder(dec_input, 
                                                          dec_hidden, 
                                                          enc_output)

      predicted_id = tf.argmax(predictions[0]).numpy()
      result += text_tokenizer.index_word[predicted_id] + ' '

      if text_tokenizer.index_word[predicted_id] == '<end>':
        return result, sentence

      # the predicted ID is fed back into the model
      dec_input = tf.expand_dims([predicted_id], 0)

    return result, sentence
def answer(sentence):
  """
  This function takes an input sentence by the user and prints the 
  model's answer along with the user's input sentence.
  Arguments:
      sentence: a string.
  """

  result, sentence = evaluate(sentence)

  print(f'INPUT: {sentence}')
  print(f'CHATBOT ANSWER: {result}')<jupyter_output><empty_output><jupyter_text>Let's try to talk to the chatbot:<jupyter_code>answer('tell me something funny')

answer('What is it that you want in life?')

answer("you're so fun")

answer("do you like annoying people?")

answer('are you my friend?')

answer("what is your favorite movie?")

answer("how do you sleep at night?")

answer("what will happen if you went inside a black hole?")

answer('If we count sheep to fall asleep, what do they count?')

answer("Why do people order double cheese burgers, large fries, and a diet coke?")

answer("What is always coming, but never arrives?")

answer("who is the most stupid person you know?")

answer("knock knock")

answer("tell me a joke")

answer("Do you exercise?")

answer("How do you keep a clear mind during hard times?")

answer("who is your best friend?")

answer("what do you have in mind?")

answer("who is you favorite singer?")
answer("Are you enjoying this conversation?")
answer("Do you know Google assistant?")

answer("what do you like to talk about the most?")

answer("Why do you like to talk about fire?")

answer("What was the best thing before sliced bread?")

answer("What is better than the FOUNTAIN OF YOUTH?")

answer("Are you lazy?")

answer("How do you know that you'll never quit smoking?")

answer("When will you quit smoking?")

answer("Who talks the most?")

answer("Who lies the most?")

answer("What do you know about computers?")

answer("Where do like to travel?")

answer("Which country do you want to visit?")

answer("Which type of music do you like?")

answer("what's your job?")

answer("Are you an artificial intelligence model?")

answer("How are you doing?")

answer("How are you?")

answer("How was your day?")

answer("Where do you vacation?")

answer("Thanks for your time.")

answer("What is your least favorite food?")

answer("What is the most hilarious childhood memory you can think of?")

answer("Are you funny?")

answer("Beethoven or Bach?")

answer("How do you like your coffee?")

answer("What is your favorite sport?")

answer("What is your favorite car?")<jupyter_output>INPUT: <start> what is your favorite car ? <end>
CHATBOT ANSWER: chihuahua : i ' m not sure , but i ' m not sure , but i ' m not sure , but i ' m not sure , but i '<jupyter_text>Start a conversation with the chatbot:The following function takes an input sentence from the user and prints the model's answer to it until the user enters exit() to finish the conversation:<jupyter_code>def chat():
    """
    This function takes input from the user and outputs the chatbot's answer
    until the user inputs `exit()`
    """

    print("Start your conversation with the chatbot!")
    print("If you want to end this conversation, enter: exit()\nHave fun!")  

    while True:
      # take user input
      user_input = str(input('>>'))
      
      if user_input == 'exit()':
        break

      try:
        # generate an answer for the user's question.
        chatbot_answer, _ = evaluate(user_input)
        print(chatbot_answer)
      except:
        # in case there's word out of chatbot vocabulary.
        print("Oops! can't help you there! Try different words or restructure your sentence.")
chat()<jupyter_output>Start your conversation with the chatbot!
If you want to end this conversation, enter: exit()
Have fun!
>>hi
a : and a mute are in a lightbulb . <end> 
>>what ' s the best anti diarrheal prescription ?
Oops! can't help you there! Try different words or restructure your sentence.
>>what do you call a russian tree ?
<end> 
>>how do you call it when an egg is on point
egg . <end> 
>>why are frogs so happy ?
because they eat whatever bugs them . <end> 
>>what is a pirate ' s worst nightmare ?
a sunken chest and no booty . <end> 
>>why does snoopdogg carry an umbrella ?
Oops! can't help you there! Try different words or restructure your sentence.
>>why can ' t obama poke fun at himself ?
because that would be racist . <end> 
>>why do turkeys always gobble ?
they ' re full . <end> 
>>why google is a she ?
she can ' t let you finish your mind . <end> 
>>why shouldn ' t you fart in church ?
because they don ' t have to sit in the oven . <end> 
>>what ' s green and has 4 wheels ?
grass , i lied[...]<jupyter_text>Conclusion
As we notice, not all the answers generated by the model are good or valid answers and there's a lot to improve here. For example, there's some answers with repeated blocks of words. Also, some of the answers are not valid responses to the questions and do not relate the topic being proposed in the question. This kind of performance is expected for a deep learning chatbot with this very small dataset but it's okay for a demo project which can be improved.  Next What can we do to improve performance:**More data**: if we want to have good performance with somewhat impressive responses, we need to train on a lot more data, maybe millions of pairs. However, training on a large scale dataset requires more computational power (maybe a more powerful GPU) and a more sophisticated model with more layers and units, let alone finding and acquiring such dataset.**DISCLAIMER**: the model has been trained on 175,671 question-answer pairs which have been gathered from different sources like reddit, and it turns out that some of the model's answers are impolite, just something to be aware of. Further Reading: - Effective Approaches to Attention-based Neural Machine Translation  - Neural machine translation with attention  - Neural Machine Translation by Jointly Learning to Align and Translate   - Neural Machine Translation (seq2seq) Tutorial<jupyter_code>!ls {checkpoint_dir}<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0893672383205204, 'fraction_numerical': 0.020555884092253104, 'mean_word_length': 3.1540729095018176, 'pattern_counts': {'":': 0, '<': 141, '<?xml version=': 0, '>': 168, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 40, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '32', 'dir': 'jupyter-structured-clean-dedup', 'id': '491771', 'license': 'MIT', 'n_tokens_mistral': 13295, 'n_tokens_neox': 12322, 'n_words': 4329, 'path': 'Smart_bot.ipynb', 'repo_name': 'gurmu/Smart-chatbot-using-deep-learning-Attention-mechanism', 'size': '42275'}