<jupyter_start><jupyter_text>Home 5: Build a seq2seq model for machine translation. Name: <NAME> Task: Translate English to Spanish  0. You will do the following:1. Read and run my code.2. Complete the code in Section 1.1 and Section 4.2.    * Translation **English** to **German** is not acceptable!!! Try another pair of languages.    3. **Make improvements.** Directly modify the code in Section 3. Do at least one of the two. By doing both correctly, you will get up to 1 bonus score to the total.    * Bi-LSTM instead of LSTM.            * Attention. (You are allowed to use existing code.)    4. Evaluate the translation using the BLEU score.     * Optional. Up to 1 bonus scores to the total.    5. Convert the notebook to .HTML file.     * The HTML file must contain the code and the output after execution.6. Put the .HTML file in your Google Drive, Dropbox, or Github repo.  (If you submit the file to Google Drive or Dropbox, you must make the file "open-access". The delay caused by "deny of access" may result in late penalty.)7. Submit the link to the HTML file to Canvas.      Hint: To implement ```Bi-LSTM```, you will need the following code to build the encoder. Do NOT use Bi-LSTM for the decoder.<jupyter_code># from keras.layers import Bidirectional, Concatenate, LSTM
#
# encoder_bilstm = Bidirectional(LSTM(latent_dim, return_state=True,
#                                   dropout=0.5, name='encoder_lstm'))
# _, forward_h, forward_c, backward_h, backward_c = encoder_bilstm(encoder_inputs)
#
# state_h = Concatenate()([forward_h, backward_h])
# state_c = Concatenate()([forward_c, backward_c])<jupyter_output><empty_output><jupyter_text>1. Data preparation1. Download data (e.g., "deu-eng.zip") from http://www.manythings.org/anki/2. Unzip the .ZIP file.3. Put the .TXT file (e.g., "deu.txt") in the directory "./Data/".  1.1. Load and clean text<jupyter_code>import re
import string

import numpy as np
from unicodedata import normalize
import numpy

# load doc into memory
def load_doc(filename):
    # open the file as read only
    file = open(filename, mode='rt', encoding='utf-8')
    # read all text
    text = file.read()
    # close the file
    file.close()
    return text


# split a loaded document into sentences
def to_pairs(doc):
    lines = doc.strip().split('\n')
    pairs = [line.split('\t') for line in  lines]
    return pairs

def clean_data(lines):
    cleaned = list()
    # prepare regex for char filtering
    re_print = re.compile('[^%s]' % re.escape(string.printable))
    # prepare translation table for removing punctuation
    table = str.maketrans('', '', string.punctuation)
    for pair in lines:
        clean_pair = list()
        for line in pair:
            # normalize unicode characters
            line = normalize('NFD', line).encode('ascii', 'ignore')
            line = line.decode('UTF-8')
            # tokenize on white space
            line = line.split()
            # convert to lowercase
            line = [word.lower() for word in line]
            # remove punctuation from each token
            line = [word.translate(table) for word in line]
            # remove non-printable chars form each token
            line = [re_print.sub('', w) for w in line]
            # remove tokens with numbers in them
            line = [word for word in line if word.isalpha()]
            # store as string
            clean_pair.append(' '.join(line))
        cleaned.append(clean_pair)
    return numpy.array(cleaned)<jupyter_output><empty_output><jupyter_text>Fill the following blanks:<jupyter_code># e.g., filename = 'Data/deu.txt'
filename = 'Data/spa.txt'

data_set_size = 50000
# load dataset
doc = load_doc(filename)

# split into Language1-Language2 pairs
pairs = to_pairs(doc)

# clean sentences
clean_pairs = clean_data(pairs)[0:data_set_size, :]
for i in range(3000, 3010):
    print('[' + clean_pairs[i, 0] + '] => [' + clean_pairs[i, 1] + ']')
from sklearn.model_selection import train_test_split
all_input_texts = clean_pairs[:, 0]
all_target_texts = ['\t' + text + '\n' for text in clean_pairs[:, 1]]

train_input, test_input, train_target, test_target = train_test_split(all_input_texts, all_target_texts, test_size=0.2)


print('Length of input_texts:  ' + str(train_input.shape))
print('Length of target_texts: ' + str(train_input.shape))
max_encoder_seq_length = max(len(line) for line in train_input)
max_decoder_seq_length = max(len(line) for line in train_target)

print('max length of input  sentences: %d' % (max_encoder_seq_length))
print('max length of target sentences: %d' % (max_decoder_seq_length))<jupyter_output>max length of input  sentences: 25
max length of target sentences: 76<jupyter_text>**Remark:** To this end, you have two lists of sentences: input_texts and target_texts  2. Text processing 2.1. Convert texts to sequences- Input: A list of $n$ sentences (with max length $t$).- It is represented by a $n\times t$ matrix after the tokenization and zero-padding.<jupyter_code>from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences

# encode and pad sequences
def text2sequences(max_len, lines):
    tokenizer = Tokenizer(char_level=True, filters='')
    tokenizer.fit_on_texts(lines)
    seqs = tokenizer.texts_to_sequences(lines)
    seqs_pad = pad_sequences(seqs, maxlen=max_len, padding='post')
    return seqs_pad, tokenizer.word_index


encoder_input_seq, input_token_index = text2sequences(max_encoder_seq_length, train_input)
decoder_input_seq, target_token_index = text2sequences(max_decoder_seq_length, train_target)

print('shape of encoder_input_seq: ' + str(encoder_input_seq.shape))
print('shape of input_token_index: ' + str(len(input_token_index)))
print('shape of decoder_input_seq: ' + str(decoder_input_seq.shape))
print('shape of target_token_index: ' + str(len(target_token_index)))
num_encoder_tokens = len(input_token_index) + 1
num_decoder_tokens = len(target_token_index) + 1

print('num_encoder_tokens: ' + str(num_encoder_tokens))
print('num_decoder_tokens: ' + str(num_decoder_tokens))<jupyter_output>num_encoder_tokens: 28
num_decoder_tokens: 30<jupyter_text>**Remark:** To this end, the input language and target language texts are converted to 2 matrices. - Their number of rows are both n_train.- Their number of columns are respective max_encoder_seq_length and max_decoder_seq_length. The followings print a sentence and its representation as a sequence.<jupyter_code>train_target[100]
decoder_input_seq[100, :]<jupyter_output><empty_output><jupyter_text>2.2. One-hot encode- Input: A list of $n$ sentences (with max length $t$).- It is represented by a $n\times t$ matrix after the tokenization and zero-padding.- It is represented by a $n\times t \times v$ tensor ($t$ is the number of unique chars) after the one-hot encoding.<jupyter_code>from keras.utils import to_categorical

# one hot encode target sequence
def onehot_encode(sequences, max_len, vocab_size):
    n = len(sequences)
    data = numpy.zeros((n, max_len, vocab_size))
    for i in range(n):
        data[i, :, :] = to_categorical(sequences[i], num_classes=vocab_size)
    return data

encoder_input_data = onehot_encode(encoder_input_seq, max_encoder_seq_length, num_encoder_tokens)
decoder_input_data = onehot_encode(decoder_input_seq, max_decoder_seq_length, num_decoder_tokens)

decoder_target_seq = numpy.zeros(decoder_input_seq.shape)
decoder_target_seq[:, 0:-1] = decoder_input_seq[:, 1:]
decoder_target_data = onehot_encode(decoder_target_seq, 
                                    max_decoder_seq_length, 
                                    num_decoder_tokens)

print(encoder_input_data.shape)
print(decoder_input_data.shape)<jupyter_output>(40000, 25, 28)
(40000, 76, 30)<jupyter_text>3. Build the networks (for training)- Build encoder, decoder, and connect the two modules to get "model". - Fit the model on the bilingual data to train the parameters in the encoder and decoder.  3.1. Encoder network- Input:  one-hot encode of the input language- Return:     -- output (all the hidden states   $h_1, \cdots , h_t$) are always discarded        -- the final hidden state  $h_t$        -- the final conveyor belt $c_t$<jupyter_code>from keras.layers import Input, Bidirectional, Concatenate, LSTM
from keras.models import Model

latent_dim = 256

# inputs of the encoder network
encoder_inputs = Input(shape=(None, num_encoder_tokens), 
                       name='encoder_inputs')

# Using BiLSTM
encoder_bilstm = Bidirectional(LSTM(latent_dim, return_state=True,
                                    dropout=0.3, name='encoder_bilstm'))
_, forward_h, forward_c, backward_h, backward_c = encoder_bilstm(encoder_inputs)

state_h = Concatenate()([forward_h, backward_h])
state_c = Concatenate()([forward_c, backward_c])

# build the encoder network model
encoder_model = Model(inputs=encoder_inputs, 
                      outputs=[state_h, state_c],
                      name='encoder')<jupyter_output><empty_output><jupyter_text>Print a summary and save the encoder network structure to "./encoder.pdf"<jupyter_code>from IPython.display import SVG
from keras.utils.vis_utils import model_to_dot, plot_model
# import pydot
# import keras
# from keras.utils.vis_utils import model_to_dot
# keras.utils.vis_utils.pydot = pydot

SVG(model_to_dot(encoder_model, show_shapes=False).create(prog='dot', format='svg'))

plot_model(
    model=encoder_model, show_shapes=False,
    to_file='encoder.pdf'
)

encoder_model.summary()<jupyter_output>Model: "encoder"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
encoder_inputs (InputLayer)     (None, None, 28)     0                                            
__________________________________________________________________________________________________
bidirectional_1 (Bidirectional) [(None, 512), (None, 583680      encoder_inputs[0][0]             
__________________________________________________________________________________________________
concatenate_1 (Concatenate)     (None, 512)          0           bidirectional_1[0][1]            
                                                                 bidirectional_1[0][3]            
____________________________________________________________________________________________[...]<jupyter_text>3.2. Decoder network- Inputs:      -- one-hot encode of the target language        -- The initial hidden state $h_t$         -- The initial conveyor belt $c_t$ - Return:     -- output (all the hidden states) $h_1, \cdots , h_t$    -- the final hidden state  $h_t$ (discarded in the training and used in the prediction)        -- the final conveyor belt $c_t$ (discarded in the training and used in the prediction)<jupyter_code>from keras.layers import Input, LSTM, Dense
from keras.models import Model

latent_dim = latent_dim * 2
# inputs of the decoder network
decoder_input_h = Input(shape=(latent_dim,), name='decoder_input_h')
decoder_input_c = Input(shape=(latent_dim,), name='decoder_input_c')
decoder_input_x = Input(shape=(None, num_decoder_tokens), name='decoder_input_x')

# set the LSTM layer
decoder_lstm = LSTM(latent_dim, return_sequences=True,
                    return_state=True, dropout=0.3, name='decoder_lstm')
decoder_lstm_outputs, state_h, state_c = decoder_lstm(decoder_input_x, 
                                                      initial_state=[decoder_input_h, decoder_input_c])

# set the dense layer
decoder_dense = Dense(num_decoder_tokens, activation='softmax', name='decoder_dense')
decoder_outputs = decoder_dense(decoder_lstm_outputs)

# build the decoder network model
decoder_model = Model(inputs=[decoder_input_x, decoder_input_h, decoder_input_c],
                      outputs=[decoder_outputs, state_h, state_c],
                      name='decoder')<jupyter_output><empty_output><jupyter_text>Print a summary and save the encoder network structure to "./decoder.pdf"<jupyter_code>from IPython.display import SVG
from keras.utils.vis_utils import model_to_dot, plot_model

SVG(model_to_dot(decoder_model, show_shapes=False).create(prog='dot', format='svg'))

plot_model(
    model=decoder_model, show_shapes=False,
    to_file='decoder.pdf'
)

decoder_model.summary()<jupyter_output>Model: "decoder"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
decoder_input_x (InputLayer)    (None, None, 30)     0                                            
__________________________________________________________________________________________________
decoder_input_h (InputLayer)    (None, 512)          0                                            
__________________________________________________________________________________________________
decoder_input_c (InputLayer)    (None, 512)          0                                            
__________________________________________________________________________________________________
decoder_lstm (LSTM)             [(None, None, 512),  1112064     decoder_input_x[0][0]      [...]<jupyter_text>3.3. Connect the encoder and decoder<jupyter_code># input layers
encoder_input_x = Input(shape=(None, num_encoder_tokens), name='encoder_input_x')
decoder_input_x = Input(shape=(None, num_decoder_tokens), name='decoder_input_x')

# connect encoder to decoder
encoder_final_states = encoder_model([encoder_input_x])
decoder_lstm_output, _, _ = decoder_lstm(decoder_input_x, initial_state=encoder_final_states)
decoder_pred = decoder_dense(decoder_lstm_output)

model = Model(inputs=[encoder_input_x, decoder_input_x], 
              outputs=decoder_pred, 
              name='model_training')
print(state_h)
print(decoder_input_h)
from IPython.display import SVG
from keras.utils.vis_utils import model_to_dot, plot_model

SVG(model_to_dot(model, show_shapes=False).create(prog='dot', format='svg'))

plot_model(
    model=model, show_shapes=False,
    to_file='model_training.pdf'
)

model.summary()<jupyter_output>Model: "model_training"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
encoder_input_x (InputLayer)    (None, None, 28)     0                                            
__________________________________________________________________________________________________
decoder_input_x (InputLayer)    (None, None, 30)     0                                            
__________________________________________________________________________________________________
encoder (Model)                 [(None, 512), (None, 583680      encoder_input_x[0][0]            
__________________________________________________________________________________________________
decoder_lstm (LSTM)             [(None, None, 512),  1112064     decoder_input_x[0][0[...]<jupyter_text>3.5. Fit the model on the bilingual dataset- encoder_input_data: one-hot encode of the input language- decoder_input_data: one-hot encode of the input language- decoder_target_data: labels (left shift of decoder_input_data)- tune the hyper-parameters- stop when the validation loss stop decreasing.<jupyter_code>print('shape of encoder_input_data' + str(encoder_input_data.shape))
print('shape of decoder_input_data' + str(decoder_input_data.shape))
print('shape of decoder_target_data' + str(decoder_target_data.shape))
model.compile(optimizer='rmsprop', loss='categorical_crossentropy')

model.fit([encoder_input_data, decoder_input_data],  # training data
          decoder_target_data,                       # labels (left shift of the target sequences)
          batch_size=64, epochs=25, validation_split=0.2, verbose=2)

model.save('seq2seq.h5')<jupyter_output>Train on 32000 samples, validate on 8000 samples
Epoch 1/25
 - 98s - loss: 0.7700 - val_loss: 0.5496
Epoch 2/25
 - 96s - loss: 0.5657 - val_loss: 0.4632
Epoch 3/25
 - 96s - loss: 0.5100 - val_loss: 0.4140
Epoch 4/25
 - 96s - loss: 0.4715 - val_loss: 0.3803
Epoch 5/25
 - 97s - loss: 0.4455 - val_loss: 0.3553
Epoch 6/25
 - 99s - loss: 0.4234 - val_loss: 0.3348
Epoch 7/25
 - 99s - loss: 0.4065 - val_loss: 0.3236
Epoch 8/25
 - 100s - loss: 0.3911 - val_loss: 0.3094
Epoch 9/25
 - 96s - loss: 0.3786 - val_loss: 0.3002
Epoch 10/25
 - 97s - loss: 0.3687 - val_loss: 0.2910
Epoch 11/25
 - 95s - loss: 0.3578 - val_loss: 0.2836
Epoch 12/25
 - 95s - loss: 0.3510 - val_loss: 0.2771
Epoch 13/25
 - 96s - loss: 0.3425 - val_loss: 0.2718
Epoch 14/25
 - 96s - loss: 0.3353 - val_loss: 0.2668
Epoch 15/25
 - 96s - loss: 0.3293 - val_loss: 0.2639
Epoch 16/25
 - 96s - loss: 0.3242 - val_loss: 0.2600
Epoch 17/25
 - 97s - loss: 0.3178 - val_loss: 0.2556
Epoch 18/25
 - 99s - loss: 0.3137 - val_loss: 0.2536
Epoch[...]<jupyter_text>4. Make predictions 4.1. Translate English to Spanish1. Encoder read a sentence (source language) and output its final states, $h_t$ and $c_t$.2. Take the [star] sign "\t" and the final state $h_t$ and $c_t$ as input and run the decoder.3. Get the new states and predicted probability distribution.4. sample a char from the predicted probability distribution5. take the sampled char and the new states as input and repeat the process (stop if reach the [stop] sign "\n").<jupyter_code># Reverse-lookup token index to decode sequences back to something readable.
reverse_input_char_index = dict((i, char) for char, i in input_token_index.items())
reverse_target_char_index = dict((i, char) for char, i in target_token_index.items())
def decode_sequence(input_seq):
    states_value = encoder_model.predict(input_seq)

    target_seq = numpy.zeros((1, 1, num_decoder_tokens))
    target_seq[0, 0, target_token_index['\t']] = 1.

    stop_condition = False
    decoded_sentence = ''
    while not stop_condition:
        output_tokens, h, c = decoder_model.predict([target_seq] + states_value)

        # this line of code is greedy selection
        # try to use multinomial sampling instead (with temperature)
        sampled_token_index = numpy.argmax(output_tokens[0, -1, :])

        if sampled_token_index in reverse_target_char_index:
            sampled_char = reverse_target_char_index[sampled_token_index]
        else:
            sampled_char = '?'
        decoded_sentence += sampled_char

        if (sampled_char == '\n' or
           len(decoded_sentence) > max_decoder_seq_length):
            stop_condition = True

        target_seq = numpy.zeros((1, 1, num_decoder_tokens))
        target_seq[0, 0, sampled_token_index] = 1.

        states_value = [h, c]

    return decoded_sentence


##%%

for seq_index in range(2100, 2120):
    # Take one sequence (part of the training set)
    # for trying out decoding.
    input_seq = encoder_input_data[seq_index: seq_index + 1]
    decoded_sentence = decode_sequence(input_seq)
    print('-')
    print('English:       ', train_input[seq_index])
    print('Spanish (true): ', train_target[seq_index][1:-1])
    print('Spanish (pred): ', decoded_sentence[0:-1])<jupyter_output>-
English:        that isnt to my liking
Spanish (true):  eso no es de mi gusto
Spanish (pred):  eso no es mi trabajo
-
English:        i killed them both
Spanish (true):  yo los mate a ambos
Spanish (pred):  yo los lame a la pelicula
-
English:        they ate some enchiladas
Spanish (true):  se comieron unas enchiladas
Spanish (pred):  ellos son les antestas de la cama
-
English:        you have to cook
Spanish (true):  tienes que cocinar
Spanish (pred):  tienes que estudiar
-
English:        go straight
Spanish (true):  ve recto
Spanish (pred):  vete a verme
-
English:        i love my son
Spanish (true):  amo a mi hijo
Spanish (pred):  amo a mi padre
-
English:        i remember tom
Spanish (true):  me acuerdo de tom
Spanish (pred):  recuerdo a tom
-
English:        he doesnt like to lose
Spanish (true):  no le gusta perder
Spanish (pred):  no le gusta mucho la carta
-
English:        tom said he would try
Spanish (true):  tom dijo que lo intentaria
Spanish (pred):  tom dije que te[...]<jupyter_text>4.2. Translate an English sentence to the target language1. Tokenization2. One-hot encode3. Translate<jupyter_code>def sentence_to_sequence(sentence, token_index, max_length):
    sentence = sentence.lower()
    sequence = numpy.zeros(max_length)
    for idx in range(len(sentence)):
        if idx >= len(sequence):
            break
        sequence[idx] = token_index[sentence[idx]]

    return sequence

def translate_sentence(sentence):
    input_sequence = sentence_to_sequence(sentence, input_token_index, max_encoder_seq_length)
    input_x = to_categorical(input_sequence, num_encoder_tokens)
    input_x = numpy.expand_dims(input_x, axis=0)
    return decode_sequence(input_x)

input_sentence = 'i love my son'
translated_sentence = translate_sentence(input_sentence)

print('source sentence is: ' + input_sentence)
print('translated sentence is: ' + translated_sentence)<jupyter_output>source sentence is: i love my son
translated sentence is: amo a mi padre<jupyter_text>5. Evaluate the translation using BLEU scoreReference: - https://machinelearningmastery.com/calculate-bleu-score-for-text-python/- https://en.wikipedia.org/wiki/BLEU**Hint:** - Randomly partition the dataset to training, validation, and test. - Evaluate the BLEU score using the test set. Report the average.- A reasonable BLEU score should be 0.1 ~ 0.5.<jupyter_code># Translate test sentences
translated_test_input = list(map(lambda sentence: translate_sentence(sentence), test_input))

from nltk.translate.bleu_score import sentence_bleu, corpus_bleu, SmoothingFunction

score = 0
for i in range(len(translated_test_input)):
    test_candidate = translated_test_input[i].replace('\n', '').replace('\t', '').split(' ')
    test_reference = test_target[i].replace('\n', '').replace('\t', '').split(' ')
    score += sentence_bleu([test_reference], test_candidate, smoothing_function=SmoothingFunction().method1)

score /= len(translated_test_input)

print(f'BLEU Score: {score}')<jupyter_output>BLEU Score: 0.15181906516451862
{'fraction_non_alphanumeric': 0.10666962109461556, 'fraction_numerical': 0.024551296255262575, 'mean_word_length': 3.8602196855481368, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 0, '>': 55, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 21, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '39817', 'license': 'MIT', 'n_tokens_mistral': 7250, 'n_tokens_neox': 6628, 'n_words': 1913, 'path': 'homework/HM5/HM5.ipynb', 'repo_name': 'Elkfrawy/CS583-2021S', 'size': '22573'}