<jupyter_start><jupyter_text>Source - https://nextjournal.com/gkoehler/machine-translation-seq2seq-cpuData Source - http://www.manythings.org/anki/  Data Read<jupyter_code>with open('Data/deu-eng/deu.txt', 'r', encoding='utf-8') as f:
    lines = f.read().split('\n')
lines[155]
len(lines)
input_texts = []
target_texts = []
input_characters = set()
target_characters = set()<jupyter_output><empty_output><jupyter_text>Generate Input and TargetsCreates a list of uninque characters in the corpur - for input and target separately<jupyter_code>num_samples = 10000
'''TEST HOW THIS CODE WORKS'''

for line in lines[:min(num_samples, len(lines) - 1)]:
    input_text, target_text, extra = line.split('\t')
    print(input_text, target_text, extra)
    target_text = '\t' + target_text + '\n'
    input_texts.append(input_text)
    target_texts.append(target_text)
    
    # making set of unique characters - input
    for char in input_text:
        if char not in input_characters:
            input_characters.add(char)
    # making set of unique characters - target
    for char in target_text:
        if char not in target_characters:
            target_characters.add(char)
    break
input_texts
target_texts
input_characters
target_characters<jupyter_output><empty_output><jupyter_text>We don't need input and target characters.<jupyter_code>''' ACTUAL EXECUTION '''

input_texts = []
target_texts = []
input_characters = set()
target_characters = set()

num_samples = 10000

for line in lines[:min(num_samples, len(lines) - 1)]:
    input_text, target_text, extra = line.split('\t')
    #     print(input_text, target_text, extra)
    target_text = '\t' + target_text + '\n'
    input_texts.append(input_text)
    target_texts.append(target_text)
    for char in input_text:
        if char not in input_characters:
            input_characters.add(char)
    for char in target_text:
        if char not in target_characters:
            target_characters.add(char)
print(input_characters)
len(input_characters)
print(target_characters)
len(target_characters)<jupyter_output><empty_output><jupyter_text>Get Configs for this CorpusHere, we obtain1. number of input texts (should be equal to number of samples)2. Total number of input characters for encoder model3. Total number of output characters for decoder model4. Maximum sequence length for input and output for binarised one-hot encoding of te texts<jupyter_code>input_characters = sorted(list(input_characters))
target_characters = sorted(list(target_characters))
num_encoder_tokens = len(input_characters)
num_decoder_tokens = len(target_characters)
max_encoder_seq_length = max([len(txt) for txt in input_texts
                              ])  # length includes spaces
max_decoder_seq_length = max([len(txt) for txt in target_texts
                              ])  # length includes spaces

print('Number of samples:', len(input_texts))  # we set this earlier
print('Number of unique input tokens (characters):', num_encoder_tokens)
print('Number of unique output tokens (characters):', num_decoder_tokens)
print('Max sequence length for inputs:', max_encoder_seq_length)
print('Max sequence length for outputs:', max_decoder_seq_length)
# # TEST - Length includes spaces
# for txt in input_texts[15:17]:
#     print(txt)
#     print(len(txt))
# one-hot encoding of the characters - basically creating word embeddings

input_token_index = dict([(char, i)
                          for i, char in enumerate(input_characters)])
target_token_index = dict([(char, i)
                           for i, char in enumerate(target_characters)])
import numpy as np

encoder_input_data = np.zeros(
    (len(input_texts), max_encoder_seq_length, num_encoder_tokens),
    dtype='float32')
decoder_input_data = np.zeros(
    (len(input_texts), max_decoder_seq_length, num_decoder_tokens),
    dtype='float32')
decoder_target_data = np.zeros(
    (len(input_texts), max_decoder_seq_length, num_decoder_tokens),
    dtype='float32')
for i, (input_text, target_text) in enumerate(zip(input_texts, target_texts)):
    for t, char in enumerate(input_text):
        encoder_input_data[i, t, input_token_index[char]] = 1.
    for t, char in enumerate(target_text):
        # decoder_target_data is ahead of decoder_input_data by one timestep
        decoder_input_data[i, t, target_token_index[char]] = 1.
        if t > 0:
            # decoder_target_data will be ahead by one timestep
            # and will not include the start character.
            decoder_target_data[i, t - 1, target_token_index[char]] = 1.<jupyter_output><empty_output><jupyter_text>Example - Input Characters<jupyter_code>input_texts[1]
print(input_characters)
index_h = input_characters.index('H')
index_h
encoder_input_data[1][0]
encoder_input_data[1][0][index_h]
encoder_input_data[1].shape<jupyter_output><empty_output><jupyter_text>Example - Output Characters<jupyter_code>target_texts[1]
for each_char in target_texts[1]:
    print("{}: {}".format(each_char, np.where(np.array(target_characters) == each_char)))
print(target_characters)
for array_ in decoder_input_data[1]:
    idx = np.where(array_ == 1)
    try:
        if idx:
            print(idx[0][0])
    except:
        pass
    
print("Number of arrays: ", len(decoder_input_data[1])) # Should be equal to max_decoder_seq_length
# Expected the array to start from 29, since it is ahead by one step

for array_ in decoder_target_data[1]:
    idx = np.where(array_ == 1)
    try:
        if idx:
            print(idx[0][0])
    except:
        pass
    
print("Number of arrays: ", len(decoder_target_data[1])) # Should be equal to max_decoder_seq_length<jupyter_output>29
47
58
58
61
3
1
Number of arrays:  45<jupyter_text>Building the Model<jupyter_code>import keras, tensorflow
from keras.models import Model
from keras.layers import Input, LSTM, Dense
import numpy as np
batch_size = 64  # batch size for training
epochs = 100  # number of epochs to train for
lstm_units = 256  # latent dimensionality of the encoding space
encoder_inputs = Input(shape=(None, num_encoder_tokens))  # input texts
encoder = LSTM(lstm_units, return_state=True)
encoder_outputs, state_h, state_c = encoder(
    encoder_inputs
)  # Gets back the hidden and cell states to feed in to the decoder
encoder_states = [
    state_h, state_c
]  # obtained the encoder vector states cz only interested in the state vectors
decoder_inputs = Input(shape=(None, num_decoder_tokens))  # the target texts
decoder_lstm = LSTM(lstm_units, return_sequences=True, return_state=True)
decoder_outputs, _, _ = decoder_lstm(decoder_inputs,
                                     initial_state=encoder_states)
decoder_dense = Dense(num_decoder_tokens, activation='softmax')
decoder_outputs = decoder_dense(decoder_outputs)
model = Model(inputs=[encoder_inputs, decoder_inputs], outputs=decoder_outputs)
model.compile(optimizer='rmsprop', loss='categorical_crossentropy')
model.summary()
model.fit([encoder_input_data, decoder_input_data], decoder_target_data,
          batch_size=batch_size,
          epochs=epochs,
          validation_split=0.2)
model.save('/Models/seq2seq_eng-ger.h5')<jupyter_output><empty_output><jupyter_text>Inference  Load from Saved Model<jupyter_code>## LOAD FROM SAVED MODEL

# model = Model([encoder_inputs, decoder_inputs], decoder_outputs)
# model.compile(optimizer='rmsprop', loss='categorical_crossentropy')
# model.load_weights()
encoder_model = Model(encoder_inputs, encoder_states)
decoder_state_input_h = Input(shape=(lstm_units, ))
decoder_state_input_c = Input(shape=(lstm_units, ))
decoder_states_inputs = [decoder_state_input_h, decoder_state_input_c]
decoder_outputs, state_h, state_c = decoder_lstm(
    decoder_inputs, initial_state=decoder_states_inputs)
decoder_states = [state_h, state_c]
decoder_outputs = decoder_dense(decoder_outputs)

decoder_model = Model([decoder_inputs] + decoder_states_inputs,
                      [decoder_outputs] + decoder_states)
# reverse-lookup token index to turn sequences back to characters

reverse_input_char_index = dict(
    (i, char) for char, i in input_token_index.items())
reverse_target_char_index = dict(
    (i, char) for char, i in target_token_index.items())
input_sentence = "How are you?"
test_sentence_tokenized = np.zeros(
    (1, max_encoder_seq_length, num_encoder_tokens), dtype='float32')
print(test_sentence_tokenized.shape) 

for t, char in enumerate(input_sentence):
    test_sentence_tokenized[0, t, input_token_index[char]] = 1.
print(test_sentence_tokenized) # the binarised one-hot encoded form
states_value = encoder_model.predict(test_sentence_tokenized) # getting the states values
# states_value is a list with two elements

states_value[0].shape # recall lstm layer units
# generate empty target sequence
target_seq = np.zeros((1, 1, num_decoder_tokens))
target_seq
# Add 1 to indicate [START] or \t
target_seq[0, 0, target_token_index['\t']] = 1.
target_seq
target_seq.shape
output_tokens, h, c = decoder_model.predict([target_seq] + states_value)
output_tokens
output_tokens.shape
output_tokens[0, -1, :] == output_tokens[0][0]
sampled_token_index = np.argmax(output_tokens[0, -1, :])
sampled_token_index
# # To explain argmax
# np.where(output_tokens[0][0] == np.max(output_tokens[0, -1, :]))
sampled_char = reverse_target_char_index[sampled_token_index] # Get the character from the dictionary
sampled_char
decoded_sentence = ''
decoded_sentence += sampled_char # add the character
# This will be in a while loop later to keep adding until the [END] is reached
# STOP CONDITION  - if the character is \n or the lenth of the sentence if more than the max target inputs we used for training
if (sampled_char == '\n' or len(decoded_sentence) > max_decoder_seq_length):
    stop_condition = True
# update the target sequence (length 1).
target_seq = np.zeros((1, 1, num_decoder_tokens))
target_seq[0, 0, sampled_token_index] = 1.
target_seq
states_value = [h, c]
output_tokens, h, c = decoder_model.predict([target_seq] + states_value)
    
# sample a token and add the corresponding character to the 
# decoded sequence
sampled_token_index = np.argmax(output_tokens[0, -1, :])
sampled_char = reverse_target_char_index[sampled_token_index]
decoded_sentence += sampled_char
decoded_sentence
''' Putting all of that together '''

def decode_sequence(input_seq):
    # encode the input sequence to get the internal state vectors.
    states_value = encoder_model.predict(input_seq)

    # generate empty target sequence of length 1 with only the start character
    target_seq = np.zeros((1, 1, num_decoder_tokens))
    target_seq[0, 0, target_token_index['\t']] = 1

    # output sequence loop
    stop_condition = False
    decoded_sentence = ''
    while not stop_condition:
        output_tokens, h, c = decoder_model.predict([target_seq] +
                                                    states_value)

        # sample a token and add the corresponding character to the
        # decoded sequence
        sampled_token_index = np.argmax(output_tokens[0, -1, :])
        sampled_char = reverse_target_char_index[sampled_token_index]
        decoded_sentence += sampled_char
        # check for the exit condition: either hitting max length
        # or predicting the 'stop' character
        if (sampled_char == '\n' or len(decoded_sentence) > max_decoder_seq_length):
            stop_condition = True

        # update the target sequence (length 1).
        target_seq = np.zeros((1, 1, num_decoder_tokens))
        target_seq[0, 0, sampled_token_index] = 1.

        # update states
        states_value = [h, c]

    return decoded_sentence
input_sentence = "How are you?"
test_sentence_tokenized = np.zeros(
    (1, max_encoder_seq_length, num_encoder_tokens), dtype='float32')
for t, char in enumerate(input_sentence):
    test_sentence_tokenized[0, t, input_token_index[char]] = 1.
print(input_sentence)
print(decode_sequence(test_sentence_tokenized))
for seq_index in range(6, 10):
    input_seq = encoder_input_data[seq_index:seq_index + 1]
    decoded_sentence = decode_sequence(input_seq)
    print('-')
    print('Input sentence:', input_texts[seq_index])
    print('Decoded sentence:', decoded_sentence)
    print('Target sentence:', target_texts[seq_index].strip())<jupyter_output>-
Input sentence: Wow!
Decoded sentence: Wonnerten?

Target sentence: Donnerwetter!
-
Input sentence: Fire!
Decoded sentence: Gehen.

Target sentence: Feuer!
-
Input sentence: Help!
Decoded sentence: Zie in der auf.

Target sentence: Hilfe!
-
Input sentence: Help!
Decoded sentence: Zie in der auf.

Target sentence: Zu Hülf!<jupyter_text>Model Validation<jupyter_code>val_input_texts = []
val_target_texts = []
line_ix = 12000 # Stating after 10000
for line in lines[line_ix:line_ix + 10]: # Getting 10 lines for validation
    input_text, target_text, extra = line.split('\t')
    val_input_texts.append(input_text)
    val_target_texts.append(target_text)

# Prep the encoder
val_encoder_input_data = np.zeros(
    (len(val_input_texts), max([len(txt) for txt in val_input_texts
                                ]), num_encoder_tokens),
    dtype='float32')

for i, input_text in enumerate(val_input_texts):
    for t, char in enumerate(input_text):
        val_encoder_input_data[i, t, input_token_index[char]] = 1.
val_input_texts
for seq_index in range(10):
    input_seq = val_encoder_input_data[seq_index:seq_index + 1]
    decoded_sentence = decode_sequence(input_seq)
    print('-')
    print('Input sentence:', val_input_texts[seq_index])
    print('Decoded sentence:', decoded_sentence[:-1])
    print('Ground Truth sentence:', val_target_texts[seq_index])<jupyter_output>-
Input sentence: I saw him again.
Decoded sentence: Ich habe einen Siegenerten.
Ground Truth sentence: Ich habe ihn wieder gesehen.
-
Input sentence: I saw him first.
Decoded sentence: Ich habe eine Schleren.
Ground Truth sentence: Ich habe ihn zuerst gesehen.
-
Input sentence: I saw it coming.
Decoded sentence: Ich habe einen Siegerehen.
Ground Truth sentence: Ich habe es kommen sehen.
-
Input sentence: I saw it coming.
Decoded sentence: Ich habe einen Siegerehen.
Ground Truth sentence: Ich habe es geahnt.
-
Input sentence: I saw it coming.
Decoded sentence: Ich habe einen Siegerehen.
Ground Truth sentence: Ich ahnte es.
-
Input sentence: I saw one today.
Decoded sentence: Ich habe eine Sie gefunden.
Ground Truth sentence: Ich habe heute einen gesehen.
-
Input sentence: I saw something.
Decoded sentence: Ich habe einen Schlesen gehen.
Ground Truth sentence: Ich habe etwas gesehen.
-
Input sentence: I saw that, too.
Decoded sentence: Ich habe eine Sie gesehen.
Ground Truth sentence: D[...]
{'fraction_non_alphanumeric': 0.09366898621912949, 'fraction_numerical': 0.010642652476463364, 'mean_word_length': 4.5296114673708034, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 37, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '313313', 'license': 'MIT', 'n_tokens_mistral': 4856, 'n_tokens_neox': 4614, 'n_words': 1261, 'path': 'NLP/Translation_using_LSTM_Eng_German.ipynb', 'repo_name': 'royn5618/Deep-Learning-Things', 'size': '14658'}