<jupyter_start><jupyter_text>****Seq2Seq Model for generating the second mesras of ferdosi poems by Using Keras API*****   **In The First Part I will Implement Seq2Seq Encoder_Decoder model Without Attention***   **In The Second Part I will Implement Seq2Seq Encoder_Decoder model With Using Attention Based on "Effective Approaches to Attention-based Neural Machine Translation" Paper ***   https://arxiv.org/abs/1508.04025)   **Sequence to Sequence model Without Attention** **First Import The necessary Libraries**<jupyter_code>import numpy as np
from sklearn.model_selection import train_test_split
import keras
from keras.layers import LSTM, Input, Dense, Embedding, Activation, dot, concatenate
from keras.losses import categorical_crossentropy
from keras.preprocessing import text
from IPython.display import SVG
from keras.utils.vis_utils import model_to_dot
from keras.utils.np_utils import to_categorical
from tensorboardcolab import TensorBoardColab, TensorBoardColabCallback
from keras.models import Model, load_model<jupyter_output>Using TensorFlow backend.<jupyter_text>**Now Load The Dataset**<jupyter_code>sher = np.loadtxt('/content/drive/My Drive/Colab Notebooks/ferdosi.txt', dtype= 'str', delimiter= ',')<jupyter_output><empty_output><jupyter_text>***Let's See one of Dataset Samples ***<jupyter_code>sher[0]<jupyter_output><empty_output><jupyter_text>**Split The Sentences into characters and add 'bom', 'eom' to decoder inputs and outputs**<jupyter_code>sher_charac_first = []
sher_charac_second_input = []
sher_charac_second_output = []



for l in range(len(sher)):

    first_mes = list(sher[l][0]) # split the sentences of first mesra to characters
    sher_charac_first.append(first_mes)

    second_mes_in = list(sher[l][1])# split the sentences of second mesra to characters
    second_mes_in.insert(0, 'bom')
    sher_charac_second_input.append(second_mes_in)

    second_mes_out =  list(sher[l][1])# split the sentences of second mesra to characters
    second_mes_out.insert(len(sher[l][1]), 'eom')
    sher_charac_second_output.append(second_mes_out)<jupyter_output><empty_output><jupyter_text>**Here is the example of splited Sentences**<jupyter_code>sher_charac_first[0], sher_charac_second_input[0]<jupyter_output><empty_output><jupyter_text>**Compute the maximum length of mesras**<jupyter_code>first_max = 0
for t in range(len(sher_charac_first)):
    if len(sher_charac_first[t])> first_max:
        first_max = len(sher_charac_first[t])

second_max = 0
for y in range(len(sher_charac_second_input)):
    if len(sher_charac_second_input[y])> second_max:
        second_max = len(sher_charac_second_input[y])

max_total = max(first_max,second_max)
max_total<jupyter_output><empty_output><jupyter_text>**Add The 'pad' to mesras until their length be equal to max_total**<jupyter_code>for u in range(len(sher_charac_first)):
    while len(sher_charac_first[u]) < max_total:
          sher_charac_first[u].insert(0, 'pad')

for v in range(len(sher_charac_second_input)):
    while len(sher_charac_second_input[v]) < max_total:
          sher_charac_second_input[v].insert(len(sher_charac_second_input[v]), 'pad')

for w in range(len(sher_charac_second_output)):
    while len(sher_charac_second_output[w]) < max_total:
          sher_charac_second_output[w].insert(len(sher_charac_second_output[w])-1, 'pad')<jupyter_output><empty_output><jupyter_text>**Here is the final mesras after preprocessings**<jupyter_code>sher_charac_first[0], sher_charac_second_input[0]<jupyter_output><empty_output><jupyter_text>***Get the unique characters of mesras ***<jupyter_code>sher = sher_charac_first + sher_charac_second_input + sher_charac_second_output

combined_list=[]
for x in sher:
    combined_list.extend(x)

characters = list(np.unique(combined_list))
vocab_size = len(characters)<jupyter_output><empty_output><jupyter_text>**After That, vocab_size is the number of unique characters**<jupyter_code>(vocab_size)<jupyter_output><empty_output><jupyter_text>**Characters is the list of all unique characters of mesras**<jupyter_code>characters<jupyter_output><empty_output><jupyter_text>**Create The word to index and index to word dictionary**<jupyter_code>tokenizer = text.Tokenizer()
tokenizer.fit_on_texts(characters)

word2id = tokenizer.word_index
id2word = tokenizer.index_word<jupyter_output><empty_output><jupyter_text>**Let's see the word to index dictionary**<jupyter_code>word2id<jupyter_output><empty_output><jupyter_text>**Adding the space character to word to index and index to word dictionaries**<jupyter_code>word2id[' '] = 0
id2word[0] = ' '<jupyter_output><empty_output><jupyter_text>**See word to index and index to word after adding space character**<jupyter_code>word2id
id2word<jupyter_output><empty_output><jupyter_text>**check the shape of preprocessed mesras**<jupyter_code>np.shape(sher_charac_first)<jupyter_output><empty_output><jupyter_text>**Substitute the characters with  their index of word to index dictionary**<jupyter_code>sher_number_first = [[ word2id[w] for w in doc ] for doc in sher_charac_first]
sher_number_second_input = [[ word2id[w] for w in doc ] for doc in sher_charac_second_input]
sher_number_second_output = [[ list(to_categorical(word2id[w], vocab_size, dtype = int)) for w in doc ] for doc in sher_charac_second_output]<jupyter_output><empty_output><jupyter_text>**See the changes and check their shapes**<jupyter_code>sher_number_first[0][1:10], np.shape(sher_number_first), np.shape(sher_number_second_output)<jupyter_output><empty_output><jupyter_text>**Split the whole data to train and test parts**<jupyter_code>encoder_input_train,  encoder_input_test, decoder_input_train,decoder_input_test,outputs_train, outputs_test = train_test_split(sher_number_first,
                                                                                                               sher_number_second_input, sher_number_second_output
                                                                                                               , test_size= 0.1, shuffle= True )<jupyter_output><empty_output><jupyter_text>**Create the Network**<jupyter_code>latent_dim = 150
encoder_inputs = Input(shape= (None,))
encoder_embedding = Embedding(input_dim= vocab_size, output_dim= 30)
encoder_embedding_output = encoder_embedding(encoder_inputs)
encoder_lstm = LSTM(latent_dim, return_state= True, dropout= 0.3,  name= 'encoder_LSTM')
encoder_output, state_h, state_c = encoder_lstm(encoder_embedding_output)
encoder_states = [state_h, state_c]


decoder_inputs = Input(shape= (None, ))
decoder_embedding = Embedding(input_dim= vocab_size, output_dim= 30)
decoder_embedding_output = decoder_embedding(decoder_inputs)
decoder_lstm = LSTM(latent_dim, return_state= True, return_sequences= True, dropout= 0.3, name= 'decoder_LSTM')
decoder_outputs, decoder_state_h,  decoder_state_c = decoder_lstm(decoder_embedding_output, initial_state = encoder_states)
decoder_dense = Dense(vocab_size, activation= 'softmax')
output = decoder_dense(decoder_outputs)
model= keras.Model([encoder_inputs, decoder_inputs], output)
model.summary()<jupyter_output>WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:66: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:541: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4432: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:148: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3733: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a[...]<jupyter_text>**Let's Visualize the Network architecture to better understand****In the first part we define the encoder and use it's last hidden state as initial hidden state in decoder**<jupyter_code>SVG(model_to_dot(model, show_shapes=True, show_layer_names=False, 
                 rankdir='TB', dpi= 65).create(prog='dot', format='svg'))<jupyter_output><empty_output><jupyter_text>**Compile the model and prepare the colab Tensorboard**<jupyter_code>model.compile(optimizer= 'adam', loss= categorical_crossentropy)
tbc=TensorBoardColab()<jupyter_output>WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:793: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3576: The name tf.log is deprecated. Please use tf.math.log instead.

Wait for 8 seconds...
TensorBoard link:
https://0a4cb022.ngrok.io<jupyter_text>**Train the model with batch_size = 32 and number_of_epochs = 20**<jupyter_code>model.fit(x= [np.array(encoder_input_train), np.array(decoder_input_train)], y= np.array(outputs_train),
          batch_size= 32, epochs= 20, callbacks= [TensorBoardColabCallback(tbc)], shuffle= True)<jupyter_output>WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/math_grad.py:1424: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1033: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1020: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3005: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorboardcolab/core.py:49: The name tf.summary.FileWriter is deprecated. Please us[...]<jupyter_text>**See the loss during the time in training** **** ![alt text](https://drive.google.com/uc?id=12eYX5Py86V2aIzmyMTkWaEGFlGje4v5g) **After Training save the model and when necessary load the trained model**<jupyter_code># model.save('/content/drive/My Drive/Colab Notebooks/model.h5')
# model = load_model('/content/drive/My Drive/Colab Notebooks/model.h5')<jupyter_output><empty_output><jupyter_text>**Compute the loss in test part of dataset**<jupyter_code>model.evaluate(x= [np.array(encoder_input_test), np.array(decoder_input_test)], y= np.array(outputs_test))<jupyter_output>4961/4961 [==============================] - 3s 640us/step<jupyter_text>So the loss of model in test data is 0.99 and is lower than loss in tarin data.it's great and shows my model has't been overfitted. **Now it's the time for creating inference model**<jupyter_code>encoder_model = Model(encoder_inputs, encoder_states)
decoder_state_input_h = Input(shape=(latent_dim,))
decoder_state_input_c = Input(shape=(latent_dim,))

decoder_states_inputs = [decoder_state_input_h, decoder_state_input_c]
decoder_input_single = Input(shape= (1,))
decoder_embedding_single = decoder_embedding(decoder_input_single)
decoder_outputs, state_h, state_c = decoder_lstm(decoder_embedding_single, initial_state=decoder_states_inputs)
decoder_states = [state_h, state_c]
decoder_outputs = decoder_dense(decoder_outputs)

decoder_model = Model([decoder_input_single] + decoder_states_inputs, [decoder_outputs] + decoder_states)<jupyter_output><empty_output><jupyter_text>*** See the graphical view of inference model ***<jupyter_code>SVG(model_to_dot(encoder_model, show_shapes=True, show_layer_names=False, 
                 rankdir='TB', dpi= 65).create(prog='dot', format='svg'))
SVG(model_to_dot(decoder_model, show_shapes=True, show_layer_names=False, 
                 rankdir='TB', dpi= 65).create(prog='dot', format='svg'))<jupyter_output><empty_output><jupyter_text>**In the below cell we loaded the poems again because after preprocessing the loaded dataset has been changed.**<jupyter_code>sher_for_inference = np.loadtxt('/content/drive/My Drive/Colab Notebooks/ferdosi.txt', dtype= 'str', delimiter= ',')<jupyter_output><empty_output><jupyter_text>**Define the mesra generation function given the first mesra**<jupyter_code>def decode_sequence(input_first_mes):

    first_mes = list(input_first_mes) # split the sentences of first mesra to characters
    while len(first_mes) < max_total:
          first_mes.insert(0, 'pad')#add 'pad'to the input first mesra
    first_mes_number = [word2id[w] for w in first_mes]
    first_mes_number = np.reshape(first_mes_number, [1, -1])

    # Encode the input as state vectors
    states_value = encoder_model.predict(first_mes_number)

    # Populate the first character of target sequence with the start character
    target_seq = word2id['bom']
    target_seq = np.reshape(target_seq,[1, 1])  

    stop_condition = False
    decoded_sentence = []

    while not stop_condition:

        output_tokens, h, c = decoder_model.predict(
            [target_seq] + states_value)

        # Sample a token that has maximum probability
        sampled_token_index = np.argmax(output_tokens[0, 0, :])
        sampled_char = id2word[sampled_token_index]
        decoded_sentence.append(sampled_char)

        # Exit condition: either hit max length
        # or find stop character.
        if (sampled_char == 'eom' or
           len(decoded_sentence) > max_total):
            stop_condition = True

        # Update the target sequence (of length 1).
        target_seq = sampled_token_index
        target_seq = np.reshape(target_seq,[1, 1]) 

        # Update states
        states_value = [h, c]

    return decoded_sentence<jupyter_output><empty_output><jupyter_text>**Give the 10 first mesra of ferdosi poems and generate the second mesras**<jupyter_code>sequences_generated = []
sequences_true = []


for i in range(10):
  gen_seq = decode_sequence(sher_for_inference[i+300][0])
  gen_seq_refi = [x for x in gen_seq if x != 'eom' and x != 'pad']
  gen_seq_refi =''.join(gen_seq_refi)
  sequences_generated.append(gen_seq_refi)
  sequences_true.append(sher_for_inference[i+300][0])<jupyter_output><empty_output><jupyter_text>**The Generated and True Mesras**<jupyter_code>sequences_generated
sequences_true<jupyter_output><empty_output><jupyter_text>**I also put the picture of generated and true mesras below for the sake of caution** ![alt text](https://drive.google.com/uc?id=1YfuZO0z0LiQqsTafse9o3iSiei1HRzme)  **Sequence to Sequence Model with Attention** **Now Let's try the attention based network. In the below cell the attention mechanism based on  "Effective Approaches to Attention-based Neural Machine Translation" paper is implemented ***<jupyter_code>latent_dim = 150


encoder_inputs = Input(shape= (None, ))
encoder_embedding = Embedding(input_dim= vocab_size, output_dim= 20)
encoder_embedding_output = encoder_embedding(encoder_inputs)
encoder_lstm = LSTM(latent_dim, return_state= True,return_sequences= True, dropout= 0.3,  name= 'encoder_LSTM')
encoder_outputs, state_h, state_c = encoder_lstm(encoder_embedding_output)
encoder_states = [state_h, state_c]


decoder_inputs = Input(shape= (None, ))
decoder_embedding = Embedding(input_dim= vocab_size, output_dim= 20)
decoder_embedding_output = decoder_embedding(decoder_inputs)
decoder_lstm = LSTM(latent_dim, return_state= True, return_sequences= True, dropout= 0.3, name= 'decoder_LSTM')
decoder_outputs, decoder_state_h,  decoder_state_c = decoder_lstm(decoder_embedding_output, initial_state = encoder_states)

attention = dot([decoder_outputs,encoder_outputs], axes= [2,2])
attention_output = Activation('softmax')(attention)
context = dot([attention_output, encoder_outputs], axes= [2,1])
decoder_combined_context = concatenate([context, decoder_outputs])
decoder_dense_context = Dense(64, activation= 'tanh')
decoder_dense = decoder_dense_context(decoder_combined_context)

decoder_dense_output = Dense(vocab_size, activation= 'softmax')
output = decoder_dense_output(decoder_dense)
model_attention = keras.Model([encoder_inputs, decoder_inputs], output)
model_attention.summary()<jupyter_output>Model: "model_2"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_3 (InputLayer)            (None, None)         0                                            
__________________________________________________________________________________________________
input_4 (InputLayer)            (None, None)         0                                            
__________________________________________________________________________________________________
embedding_3 (Embedding)         (None, None, 20)     800         input_3[0][0]                    
__________________________________________________________________________________________________
embedding_4 (Embedding)         (None, None, 20)     800         input_4[0][0]              [...]<jupyter_text>**Let's Visualize the Attention Network architecture to better understand*****In the first part we define the encoder and use it's last hidden state as initial hidden state in decoder and also compute context tensor based on all encoder_outputs and alignment model. Finally we Concatenate the context tensor and decoder outputs  as input to fully connected with "softmax" activation and vocabulary size dimension ***<jupyter_code>SVG(model_to_dot(model_attention, show_shapes=True, show_layer_names=False, 
                 rankdir='TB', dpi= 65).create(prog='dot', format='svg'))<jupyter_output><empty_output><jupyter_text>**Compile the attention model and prepare the colab Tensorboard**<jupyter_code>model_attention.compile(optimizer= 'adam', loss= categorical_crossentropy)
tbc=TensorBoardColab()<jupyter_output>WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:793: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3576: The name tf.log is deprecated. Please use tf.math.log instead.

Wait for 8 seconds...
TensorBoard link:
https://3334020f.ngrok.io<jupyter_text>**Train the attention model with batch_size = 32 and number_of_epochs = 20**<jupyter_code>model_attention.fit(x= [np.array(encoder_input_train), np.array(decoder_input_train)], y= np.array(outputs_train),
          batch_size= 32, epochs= 20, callbacks= [TensorBoardColabCallback(tbc)], shuffle= True)<jupyter_output>WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/math_grad.py:1424: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1033: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1020: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3005: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorboardcolab/core.py:49: The name tf.summary.FileWriter is deprecated. Please us[...]<jupyter_text>**See the loss during the time in training** ![alt text](https://drive.google.com/uc?id=1i0x9I3EPW-A1sP4JNnkuY_vjBRmzBmA6) We See that the loss of attention model is lower that model without attention **After Training save the attention model and when necessary load the trained attention model**<jupyter_code># model_attention.save('/content/drive/My Drive/Colab Notebooks/model_attention.h5')
# model_attention = load_model('/content/drive/My Drive/Colab Notebooks/model_attention.h5')<jupyter_output><empty_output><jupyter_text>**get the attention model loss in test part of dataset**<jupyter_code>model_attention.evaluate(x= [np.array(encoder_input_test), np.array(decoder_input_test)], y= np.array(outputs_test))<jupyter_output>4961/4961 [==============================] - 3s 701us/step<jupyter_text>**So the loss of attention model in test data is 0.94 and is lower than loss in tarin data.** **it's great and shows my attention model has't been overfitted.** **Now it's the time for creating inference model**<jupyter_code>encoder_model_attention = Model(encoder_inputs, [encoder_outputs] + encoder_states)
decoder_state_input_h = Input(shape=(latent_dim,))
decoder_state_input_c = Input(shape=(latent_dim,))
encoder_outputs_inference = Input(shape= (None,latent_dim,))

decoder_states_inputs = [decoder_state_input_h, decoder_state_input_c]
decoder_input_single = Input(shape= (1,))
decoder_embedding_single = decoder_embedding(decoder_input_single)
decoder_outputs, state_h, state_c = decoder_lstm(decoder_embedding_single, initial_state=decoder_states_inputs)
decoder_states = [state_h, state_c]

attention_inference = dot([decoder_outputs,encoder_outputs_inference], axes= [2,2])
attention_output_inference = Activation('softmax')(attention_inference)
context_inference = dot([attention_output_inference, encoder_outputs_inference], axes= [2,1])
decoder_combined_context_inference = concatenate([context_inference, decoder_outputs])
decoder_outputs_context = decoder_dense_context(decoder_combined_context_inference)
decoder_outputs_fianl = decoder_dense_output(decoder_outputs_context)

decoder_model_attention = Model([decoder_input_single] + [encoder_outputs_inference] + decoder_states_inputs , [decoder_outputs_fianl] + decoder_states)<jupyter_output><empty_output><jupyter_text>***See the graphical view of encoder and decoder part of inference attention model  ***<jupyter_code>SVG(model_to_dot(encoder_model_attention, show_shapes=True, show_layer_names=False, 
                 rankdir='TB', dpi= 65).create(prog='dot', format='svg'))
SVG(model_to_dot(decoder_model_attention, show_shapes=True, show_layer_names=False, 
                 rankdir='TB', dpi= 65).create(prog='dot', format='svg'))<jupyter_output><empty_output><jupyter_text>**Define the mesra generation function given the first mesrabased on attention sequence to sequence model**<jupyter_code>def decode_sequence_attention(input_first_mes):

    first_mes = list(input_first_mes) # split the sentences of first mesra to characters
    while len(first_mes) < max_total:
          first_mes.insert(0, 'pad')#add 'pad'to the input first mesra
    first_mes_number = [word2id[w] for w in first_mes]
    first_mes_number = np.reshape(first_mes_number, [1, -1])

    # Encode the input as state vectors and outputs
    outputs_encoder, states_value = encoder_model_attention.predict(first_mes_number)[0], encoder_model_attention.predict(first_mes_number)[1:]

    # Populate the first character of target sequence with the start character
    target_seq = word2id['bom']
    target_seq = np.reshape(target_seq,[1, 1])  

    stop_condition = False
    decoded_sentence = []

    while not stop_condition:

        output_tokens, h, c = decoder_model_attention.predict(
            [target_seq] + [outputs_encoder]+ states_value)

        # Sample a token with maximum probability
        sampled_token_index = np.argmax(output_tokens[0, 0, :])
        sampled_char = id2word[sampled_token_index]
        decoded_sentence.append(sampled_char)

        # Exit condition: either hit max length
        # or find stop character.
        if (sampled_char == 'eom' or
           len(decoded_sentence) > max_total):
            stop_condition = True

        # Update the target sequence (of length 1).
        target_seq = sampled_token_index
        target_seq = np.reshape(target_seq,[1, 1]) 

        # Update states
        states_value = [h, c]

    return decoded_sentence<jupyter_output><empty_output><jupyter_text>**chose the 10 first mesra of ferdosi poems and generate the second mesras**<jupyter_code>sequences_generated_attention = []
sequences_true_attention = []

for i in range(10):
  gen_seq = decode_sequence_attention(sher_for_inference[i+300][0])
  gen_seq_refi = [x for x in gen_seq if x != 'eom' and x != 'pad']
  gen_seq_refi =''.join(gen_seq_refi)
  sequences_generated_attention.append(gen_seq_refi)
  sequences_true_attention.append(sher_for_inference[i+300][0])<jupyter_output><empty_output><jupyter_text>**The generated and true mesras**<jupyter_code>sequences_generated_attention
sequences_true_attention<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10579881656804734, 'fraction_numerical': 0.01455621301775148, 'mean_word_length': 5.907629427792916, 'pattern_counts': {'":': 0, '<': 165, '<?xml version=': 0, '>': 164, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '42', 'dir': 'jupyter-structured-clean-dedup', 'id': '546489', 'license': 'MIT', 'n_tokens_mistral': 8528, 'n_tokens_neox': 7996, 'n_words': 1733, 'path': 'Seq2Seq.ipynb', 'repo_name': 'hassancpu/Deep-Learning-Course-Sharif-University-of-Technology-', 'size': '25350'}