title characterlevel recurrent sequencetosequence model fchollethttps twitter comfchollet date created 20170929 last modified 20231122 description characterlevel recurrent sequencetosequence model accelerator gpu introduction this example demonstrates how to implement a basic characterlevel recurrent sequencetosequence model we apply it to translating short english sentences into short french sentences characterbycharacter note that it is fairly unusual to do characterlevel machine translation as wordlevel models are more common in this domain summary of the algorithm we start with input sequences from a domain e g english sentences and corresponding target sequences from another domain e g french sentences an encoder lstm turns input sequences to 2 state vectors we keep the last lstm state and discard the outputs a decoder lstm is trained to turn the target sequences into the same sequence but offset by one timestep in the future a training process called teacher forcing in this context it uses as initial state the state vectors from the encoder effectively the decoder learns to generate targetst1 given targets t conditioned on the input sequence in inference mode when we want to decode unknown input sequences we encode the input sequence into state vectors start with a target sequence of size 1 just the startofsequence character feed the state vectors and 1char target sequence to the decoder to produce predictions for the next character sample the next character using these predictions we simply use argmax append the sampled character to the target sequence repeat until we generate the endofsequence character or we hit the character limit setup download the data configuration path to the data txt file on disk prepare the data vectorize the data we use tab as the start sequence character for the targets and n as end sequence character decodertargetdata is ahead of decoderinputdata by one timestep decodertargetdata will be ahead by one timestep and will not include the start character build the model define an input sequence and process it we discard encoderoutputs and only keep the states set up the decoder using encoderstates as initial state we set up our decoder to return full output sequences and to return internal states as well we don t use the return states in the training model but we will use them in inference define the model that will turn encoderinputdata decoderinputdata into decodertargetdata train the model save model run inference sampling 1 encode input and retrieve initial decoder state 2 run one step of decoder with this initial state and a start of sequence token as target output will be the next target token 3 repeat with the current target token and current states define sampling models restore the model and construct the encoder and decoder reverselookup token index to decode sequences back to something readable encode the input as state vectors generate empty target sequence of length 1 populate the first character of target sequence with the start character sampling loop for a batch of sequences to simplify here we assume a batch of size 1 sample a token exit condition either hit max length or find stop character update the target sequence of length 1 update states you can now generate decoded sentences as such take one sequence part of the training set for trying out decoding introduction this example demonstrates how to implement a basic character level recurrent sequence to sequence model we apply it to translating short english sentences into short french sentences character by character note that it is fairly unusual to do character level machine translation as word level models are more common in this domain summary of the algorithm we start with input sequences from a domain e g english sentences and corresponding target sequences from another domain e g french sentences an encoder lstm turns input sequences to 2 state vectors we keep the last lstm state and discard the outputs a decoder lstm is trained to turn the target sequences into the same sequence but offset by one timestep in the future a training process called teacher forcing in this context it uses as initial state the state vectors from the encoder effectively the decoder learns to generate targets t 1 given targets t conditioned on the input sequence in inference mode when we want to decode unknown input sequences we encode the input sequence into state vectors start with a target sequence of size 1 just the start of sequence character feed the state vectors and 1 char target sequence to the decoder to produce predictions for the next character sample the next character using these predictions we simply use argmax append the sampled character to the target sequence repeat until we generate the end of sequence character or we hit the character limit setup download the data configuration batch size for training number of epochs to train for latent dimensionality of the encoding space number of samples to train on path to the data txt file on disk prepare the data vectorize the data we use tab as the start sequence character for the targets and n as end sequence character decoder_target_data is ahead of decoder_input_data by one timestep decoder_target_data will be ahead by one timestep and will not include the start character build the model define an input sequence and process it we discard encoder_outputs and only keep the states set up the decoder using encoder_states as initial state we set up our decoder to return full output sequences and to return internal states as well we don t use the return states in the training model but we will use them in inference define the model that will turn encoder_input_data decoder_input_data into decoder_target_data train the model save model run inference sampling 1 encode input and retrieve initial decoder state 2 run one step of decoder with this initial state and a start of sequence token as target output will be the next target token 3 repeat with the current target token and current states define sampling models restore the model and construct the encoder and decoder input_1 lstm_1 input_2 reverse lookup token index to decode sequences back to something readable encode the input as state vectors generate empty target sequence of length 1 populate the first character of target sequence with the start character sampling loop for a batch of sequences to simplify here we assume a batch of size 1 sample a token exit condition either hit max length or find stop character update the target sequence of length 1 update states you can now generate decoded sentences as such take one sequence part of the training set for trying out decoding
import numpy as np
import keras
import os
from pathlib import Path



fpath = keras.utils.get_file(
    origin="http://www.manythings.org/anki/fra-eng.zip"
)
dirpath = Path(fpath).parent.absolute()
os.system(f"unzip -q {fpath} -d {dirpath}")



batch_size = 64  
epochs = 100  
latent_dim = 256  
num_samples = 10000  

data_path = os.path.join(dirpath, "fra.txt")




input_texts = []
target_texts = []
input_characters = set()
target_characters = set()
with open(data_path, "r", encoding="utf-8") as f:
    lines = f.read().split("\n")
for line in lines[: min(num_samples, len(lines) - 1)]:
    input_text, target_text, _ = line.split("\t")
    
    
    target_text = "\t" + target_text + "\n"
    input_texts.append(input_text)
    target_texts.append(target_text)
    for char in input_text:
        if char not in input_characters:
            input_characters.add(char)
    for char in target_text:
        if char not in target_characters:
            target_characters.add(char)

input_characters = sorted(list(input_characters))
target_characters = sorted(list(target_characters))
num_encoder_tokens = len(input_characters)
num_decoder_tokens = len(target_characters)
max_encoder_seq_length = max([len(txt) for txt in input_texts])
max_decoder_seq_length = max([len(txt) for txt in target_texts])

print("Number of samples:", len(input_texts))
print("Number of unique input tokens:", num_encoder_tokens)
print("Number of unique output tokens:", num_decoder_tokens)
print("Max sequence length for inputs:", max_encoder_seq_length)
print("Max sequence length for outputs:", max_decoder_seq_length)

input_token_index = dict([(char, i) for i, char in enumerate(input_characters)])
target_token_index = dict(
    [(char, i) for i, char in enumerate(target_characters)]
)

encoder_input_data = np.zeros(
    (len(input_texts), max_encoder_seq_length, num_encoder_tokens),
    dtype="float32",
)
decoder_input_data = np.zeros(
    (len(input_texts), max_decoder_seq_length, num_decoder_tokens),
    dtype="float32",
)
decoder_target_data = np.zeros(
    (len(input_texts), max_decoder_seq_length, num_decoder_tokens),
    dtype="float32",
)

for i, (input_text, target_text) in enumerate(zip(input_texts, target_texts)):
    for t, char in enumerate(input_text):
        encoder_input_data[i, t, input_token_index[char]] = 1.0
    encoder_input_data[i, t + 1 :, input_token_index[" "]] = 1.0
    for t, char in enumerate(target_text):
        
        decoder_input_data[i, t, target_token_index[char]] = 1.0
        if t > 0:
            
            
            decoder_target_data[i, t - 1, target_token_index[char]] = 1.0
    decoder_input_data[i, t + 1 :, target_token_index[" "]] = 1.0
    decoder_target_data[i, t:, target_token_index[" "]] = 1.0




encoder_inputs = keras.Input(shape=(None, num_encoder_tokens))
encoder = keras.layers.LSTM(latent_dim, return_state=True)
encoder_outputs, state_h, state_c = encoder(encoder_inputs)


encoder_states = [state_h, state_c]


decoder_inputs = keras.Input(shape=(None, num_decoder_tokens))




decoder_lstm = keras.layers.LSTM(
    latent_dim, return_sequences=True, return_state=True
)
decoder_outputs, _, _ = decoder_lstm(
    decoder_inputs, initial_state=encoder_states
)
decoder_dense = keras.layers.Dense(num_decoder_tokens, activation="softmax")
decoder_outputs = decoder_dense(decoder_outputs)



model = keras.Model([encoder_inputs, decoder_inputs], decoder_outputs)



model.compile(
    optimizer="rmsprop", loss="categorical_crossentropy", metrics=["accuracy"]
)
model.fit(
    [encoder_input_data, decoder_input_data],
    decoder_target_data,
    batch_size=batch_size,
    epochs=epochs,
    validation_split=0.2,
)

model.save("s2s_model.keras")





model = keras.models.load_model("s2s_model.keras")

encoder_inputs = model.input[0]  
encoder_outputs, state_h_enc, state_c_enc = model.layers[2].output  
encoder_states = [state_h_enc, state_c_enc]
encoder_model = keras.Model(encoder_inputs, encoder_states)

decoder_inputs = model.input[1]  
decoder_state_input_h = keras.Input(shape=(latent_dim,))
decoder_state_input_c = keras.Input(shape=(latent_dim,))
decoder_states_inputs = [decoder_state_input_h, decoder_state_input_c]
decoder_lstm = model.layers[3]
decoder_outputs, state_h_dec, state_c_dec = decoder_lstm(
    decoder_inputs, initial_state=decoder_states_inputs
)
decoder_states = [state_h_dec, state_c_dec]
decoder_dense = model.layers[4]
decoder_outputs = decoder_dense(decoder_outputs)
decoder_model = keras.Model(
    [decoder_inputs] + decoder_states_inputs, [decoder_outputs] + decoder_states
)



reverse_input_char_index = dict(
    (i, char) for char, i in input_token_index.items()
)
reverse_target_char_index = dict(
    (i, char) for char, i in target_token_index.items()
)


def decode_sequence(input_seq):
    
    states_value = encoder_model.predict(input_seq, verbose=0)

    
    target_seq = np.zeros((1, 1, num_decoder_tokens))
    
    target_seq[0, 0, target_token_index["\t"]] = 1.0

    
    
    stop_condition = False
    decoded_sentence = ""
    while not stop_condition:
        output_tokens, h, c = decoder_model.predict(
            [target_seq] + states_value, verbose=0
        )

        
        sampled_token_index = np.argmax(output_tokens[0, -1, :])
        sampled_char = reverse_target_char_index[sampled_token_index]
        decoded_sentence += sampled_char

        
        
        if (
            sampled_char == "\n"
            or len(decoded_sentence) > max_decoder_seq_length
        ):
            stop_condition = True

        
        target_seq = np.zeros((1, 1, num_decoder_tokens))
        target_seq[0, 0, sampled_token_index] = 1.0

        
        states_value = [h, c]
    return decoded_sentence




for seq_index in range(20):
    
    
    input_seq = encoder_input_data[seq_index : seq_index + 1]
    decoded_sentence = decode_sequence(input_seq)
    print("-")
    print("Input sentence:", input_texts[seq_index])
    print("Decoded sentence:", decoded_sentence)