title sequence to sequence learning for performing number addition smerityhttps twitter comsmerity and others date created 20150817 last modified 20200417 description a model that learns to add strings of numbers e g 53561 596 accelerator gpu introduction in this example we train a model to learn to add two numbers provided as strings example input 53561 output 596 input may optionally be reversed which was shown to increase performance in many tasks in learning to executehttp arxiv orgabs1410 4615 and sequence to sequence learning with neural networkshttp papers nips ccpaper5346sequencetosequencelearningwithneuralnetworks pdf theoretically sequence order inversion introduces shorter term dependencies between source and target for this problem results for two digits reversed one layer lstm 128 hn 5k training examples 99 traintest accuracy in 55 epochs three digits reversed one layer lstm 128 hn 50k training examples 99 traintest accuracy in 100 epochs four digits reversed one layer lstm 128 hn 400k training examples 99 traintest accuracy in 20 epochs five digits reversed one layer lstm 128 hn 550k training examples 99 traintest accuracy in 30 epochs setup parameters for the model and dataset maximum length of input is int int e g 345678 maximum length of int is digits generate the data given a set of characters encode them to a onehot integer representation decode the onehot or integer representation to their character output decode a vector of probabilities to their character output initialize character table arguments chars characters that can appear in the input onehot encode given string c arguments c string to be encoded numrows number of rows in the returned onehot encoding this is used to keep the of rows for each data the same decode the given vector or 2d array to their character output arguments x a vector or a 2d array of probabilities or onehot representations or a vector of character indices used with calcargmaxfalse calcargmax whether to find the character index with maximum probability defaults to true all the numbers plus sign and space for padding skip any addition questions we ve already seen also skip any such that xy yx hence the sorting pad the data with spaces such that it is always maxlen answers can be of maximum size digits 1 reverse the query e g 12345 becomes 54321 note the space used for padding vectorize the data shuffle x y in unison as the later parts of x will almost all be larger digits explicitly set apart 10 for validation data that we never train over build the model encode the input sequence using a lstm producing an output of size 128 note in a situation where your input sequences have a variable length use inputshapenone numfeature as the decoder rnn s input repeatedly provide with the last output of rnn for each time step repeat digits 1 times as that s the maximum length of output e g when digits3 max output is 9999991998 the decoder rnn could be multiple layers stacked or a single layer by setting returnsequences to true return not only the last output but all the outputs so far in the form of numsamples timesteps outputdim this is necessary as timedistributed in the below expects the first dimension to be the timesteps apply a dense layer to the every temporal slice of an input for each of step of the output sequence decide which character should be chosen train the model train the model each generation and show predictions against the validation dataset select 10 samples from the validation set at random so we can visualize errors you ll get to 99 validation accuracy after 30 epochs example available on huggingface trained model demo generic badgehttps img shields iobadge20modeladdition20lstmblack svghttps huggingface cokerasioadditionlstm generic badgehttps img shields iobadge20spacesaddition20lstmblack svghttps huggingface cospaceskerasioadditionlstm introduction in this example we train a model to learn to add two numbers provided as strings example input 535 61 output 596 input may optionally be reversed which was shown to increase performance in many tasks in learning to execute http arxiv org abs 1410 4615 and sequence to sequence learning with neural networks http papers nips cc paper 5346 sequence to sequence learning with neural networks pdf theoretically sequence order inversion introduces shorter term dependencies between source and target for this problem results for two digits reversed one layer lstm 128 hn 5k training examples 99 train test accuracy in 55 epochs three digits reversed one layer lstm 128 hn 50k training examples 99 train test accuracy in 100 epochs four digits reversed one layer lstm 128 hn 400k training examples 99 train test accuracy in 20 epochs five digits reversed one layer lstm 128 hn 550k training examples 99 train test accuracy in 30 epochs setup parameters for the model and dataset maximum length of input is int int e g 345 678 maximum length of int is digits generate the data given a set of characters encode them to a one hot integer representation decode the one hot or integer representation to their character output decode a vector of probabilities to their character output initialize character table arguments chars characters that can appear in the input one hot encode given string c arguments c string to be encoded num_rows number of rows in the returned one hot encoding this is used to keep the of rows for each data the same decode the given vector or 2d array to their character output arguments x a vector or a 2d array of probabilities or one hot representations or a vector of character indices used with calc_argmax false calc_argmax whether to find the character index with maximum probability defaults to true all the numbers plus sign and space for padding skip any addition questions we ve already seen also skip any such that x y y x hence the sorting pad the data with spaces such that it is always maxlen answers can be of maximum size digits 1 reverse the query e g 12 345 becomes 543 21 note the space used for padding vectorize the data shuffle x y in unison as the later parts of x will almost all be larger digits explicitly set apart 10 for validation data that we never train over build the model try to add more lstm layers encode the input sequence using a lstm producing an output of size 128 note in a situation where your input sequences have a variable length use input_shape none num_feature as the decoder rnn s input repeatedly provide with the last output of rnn for each time step repeat digits 1 times as that s the maximum length of output e g when digits 3 max output is 999 999 1998 the decoder rnn could be multiple layers stacked or a single layer by setting return_sequences to true return not only the last output but all the outputs so far in the form of num_samples timesteps output_dim this is necessary as timedistributed in the below expects the first dimension to be the timesteps apply a dense layer to the every temporal slice of an input for each of step of the output sequence decide which character should be chosen train the model train the model each generation and show predictions against the validation dataset select 10 samples from the validation set at random so we can visualize errors you ll get to 99 validation accuracy after 30 epochs example available on huggingface trained model demo generic badge https img shields io badge 20model addition 20lstm black svg https huggingface co keras io addition lstm generic badge https img shields io badge 20spaces addition 20lstm black svg https huggingface co spaces keras io addition lstm
import keras
from keras import layers
import numpy as np


TRAINING_SIZE = 50000
DIGITS = 3
REVERSE = True



MAXLEN = DIGITS + 1 + DIGITS




class CharacterTable:
    

    def __init__(self, chars):
        
        self.chars = sorted(set(chars))
        self.char_indices = dict((c, i) for i, c in enumerate(self.chars))
        self.indices_char = dict((i, c) for i, c in enumerate(self.chars))

    def encode(self, C, num_rows):
        
        x = np.zeros((num_rows, len(self.chars)))
        for i, c in enumerate(C):
            x[i, self.char_indices[c]] = 1
        return x

    def decode(self, x, calc_argmax=True):
        
        if calc_argmax:
            x = x.argmax(axis=-1)
        return "".join(self.indices_char[x] for x in x)



chars = "0123456789+ "
ctable = CharacterTable(chars)

questions = []
expected = []
seen = set()
print("Generating data...")
while len(questions) < TRAINING_SIZE:
    f = lambda: int(
        "".join(
            np.random.choice(list("0123456789"))
            for i in range(np.random.randint(1, DIGITS + 1))
        )
    )
    a, b = f(), f()
    
    
    key = tuple(sorted((a, b)))
    if key in seen:
        continue
    seen.add(key)
    
    q = "{}+{}".format(a, b)
    query = q + " " * (MAXLEN - len(q))
    ans = str(a + b)
    
    ans += " " * (DIGITS + 1 - len(ans))
    if REVERSE:
        
        
        query = query[::-1]
    questions.append(query)
    expected.append(ans)
print("Total questions:", len(questions))



print("Vectorization...")
x = np.zeros((len(questions), MAXLEN, len(chars)), dtype=bool)
y = np.zeros((len(questions), DIGITS + 1, len(chars)), dtype=bool)
for i, sentence in enumerate(questions):
    x[i] = ctable.encode(sentence, MAXLEN)
for i, sentence in enumerate(expected):
    y[i] = ctable.encode(sentence, DIGITS + 1)



indices = np.arange(len(y))
np.random.shuffle(indices)
x = x[indices]
y = y[indices]


split_at = len(x) - len(x) // 10
(x_train, x_val) = x[:split_at], x[split_at:]
(y_train, y_val) = y[:split_at], y[split_at:]

print("Training Data:")
print(x_train.shape)
print(y_train.shape)

print("Validation Data:")
print(x_val.shape)
print(y_val.shape)



print("Build model...")
num_layers = 1  

model = keras.Sequential()



model.add(layers.Input((MAXLEN, len(chars))))
model.add(layers.LSTM(128))



model.add(layers.RepeatVector(DIGITS + 1))

for _ in range(num_layers):
    
    
    
    
    model.add(layers.LSTM(128, return_sequences=True))



model.add(layers.Dense(len(chars), activation="softmax"))
model.compile(
    loss="categorical_crossentropy", optimizer="adam", metrics=["accuracy"]
)
model.summary()



epochs = 30
batch_size = 32




for epoch in range(1, epochs):
    print()
    print("Iteration", epoch)
    model.fit(
        x_train,
        y_train,
        batch_size=batch_size,
        epochs=1,
        validation_data=(x_val, y_val),
    )
    
    
    for i in range(10):
        ind = np.random.randint(0, len(x_val))
        rowx, rowy = x_val[np.array([ind])], y_val[np.array([ind])]
        preds = np.argmax(model.predict(rowx), axis=-1)
        q = ctable.decode(rowx[0])
        correct = ctable.decode(rowy[0])
        guess = ctable.decode(preds[0], calc_argmax=False)
        print("Q", q[::-1] if REVERSE else q, end=" ")
        print("T", correct, end=" ")
        if correct == guess:
            print("☑ " + guess)
        else:
            print("☒ " + guess)