<jupyter_start><jupyter_text>Sequence Modeling: Train a model to do basic math@<EMAIL>@<EMAIL>code will be available after session @ https://github.com/sunilmallya/dl-twitch-series  DatasetWe know how to generate basic math sequences, our input looks like this:5+4+9 = 181+4+2 = 7 3 * 2 * 5 = 309 * 2 * 2 = 36  Recurrent Neural Networks (RNN)Certain problems require you to model information dependency. They can be external or could be based on previous context, which we'll explore in this notebook. Recurrent neural networks (RNN) are designed to address this issue. They are networks with loops in them, thus allowing information to persist, making them ideal to model sequences. Speech recognition, image captioning, video analysis, language modeling and many other use case can be addressed using RNN's. Check out <NAME>'s [blog](http://karpathy.github.io/2015/05/21/rnn-effectiveness/) for more details. Lets visualize a few examples of sequence in input or oputput: ![rnn](http://karpathy.github.io/assets/rnn/diags.jpeg) img src: http://karpathy.github.ioA RNN can be imagined as a repeated copy of the same network, each passing a message to the next step, in this case it may be to the same layer. But RNN's have problems with long range dependencies - interactions between sequences that are many steps apart, this is explained in detail in this [blog](http://colah.github.io/posts/2015-08-Understanding-LSTMs/). Also RNN's tend to be deep and hence more vulnerable to the [vanishing gradient problem](https://en.wikipedia.org/wiki/Vanishing_gradient_problem) To address these issues [Hochreiter & Schmidhuber (1997)](http://www.bioinf.jku.at/publications/older/2604.pdf) introduced LSTMs, which now are widely used to model the problems described above.   Sequence modeling basics What is sequence modeling?![](https://indico.io/blog/wp-content/uploads/2016/04/seq-nathan-fig3a.jpg) Encoder - Decoder![](https://indico.io/blog/wp-content/uploads/2016/04/seq-nathan-figure3_b.jpg)Encoder transforms the input in to a hidden state. This can now be translated or converted in to any desirable form. The decoder tries topredict the next word in the outpute (decoder) sequence, given the current word in the decoder sequence and the context from the encoder sequence. What problems can we solve with this- Language translation- image caption- sequence conversionimg src: https://indico.io/blog/sequence-modeling-neuralnets-part1/   why one hot ?In general most ML agorithms don't understand the label data directly. They like input/output variables to be numbers.For categorical data where there is no ordering, its not desirable to let the model assume any kind of ordering. One-hot encoding can help establish this and is a more desirable format for the neural network to work on.https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/  ReferencesIts inspired by serveral blogs, but mainly from this paper - "Sequence to Sequence Learning with Neural Networks"http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf<jupyter_code>import numpy as np
import mxnet as mx

import random
random.seed(10)

n_samples = 10000
n_numbers = 3 # numbers to operate on
largest = 10

character_set = ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '+', '*', ' ']

input_sequence_length =  8 # (10 + 10 + 10)
output_sequence_length = 4 



def generate_data(n_samples):
    inputs = []
    labels = []
    
    char_to_int = dict((c,i)  for i,c in enumerate(character_set))
    
    for i in range(n_samples):
        lhs = [random.randint(1, largest) for _ in range(n_numbers)]
        op = random.choice(['+', '*'])
        if op == '+':
            rhs = sum(lhs)
        elif op == '*':
            rhs = 1
            for l in lhs:
                rhs *= l
        
        lhs = [str(l) for l in lhs]
        strng = op.join(lhs)
        padded_strng = "%*s" % (input_sequence_length, strng)
        enc_input = [char_to_int[ch] for ch in padded_strng]
        
        #RHS
        padded_strng = "%*s" % (output_sequence_length, rhs)
        enc_lbl = [char_to_int[ch] for ch in padded_strng]
    
        inputs.append(enc_input)
        labels.append(enc_lbl)
        
    return np.array(inputs), np.array(labels)


dataX, dataY = generate_data(n_samples)

print dataX.shape, dataY.shape
# Iterators

batch_size = 32
data_dim = len(character_set)

train_iter = mx.io.NDArrayIter(data=dataX, label=dataY,
                               data_name='data', label_name='target',
                               batch_size=batch_size, shuffle=True)

train_iter.provide_data, train_iter.provide_label
# Lets build the model!

data = mx.sym.var('data')
target = mx.sym.var('target')

# Encoder - Decoder 

lstm1 = mx.rnn.FusedRNNCell(num_hidden=32, prefix="lstm1_", get_next_state=True)
lstm2 = mx.rnn.FusedRNNCell(num_hidden=32, prefix="lstm2_", get_next_state=False)

# convert to one-hot encoding

data_one_hot = mx.sym.one_hot(data, depth=len(character_set))
data_one_hot = mx.sym.transpose(data_one_hot, axes=(1,0,2))

# unroll the loop/lstm

# Note that when unrolling, if 'merge_outputs' is set to True, the 'outputs' is merged into a single symbol
# In the layout, 'N' represents batch size, 'T' represents sequence length, and 'C' represents the
# number of dimensions in hidden states.

l_out, encode_state = lstm1.unroll(length=input_sequence_length, inputs=data_one_hot, layout="TNC")
encode_state_h = mx.sym.broadcast_to(encode_state[0], shape=(output_sequence_length, 0, 0))

# Decoder

decode_out, l2 = lstm2.unroll(length=output_sequence_length, inputs=encode_state_h, layout="TNC")
decode_out = mx.sym.reshape(decode_out, shape=(-1,batch_size))

out = mx.sym.FullyConnected(decode_out, num_hidden=data_dim)
out = mx.sym.reshape(out, shape=(output_sequence_length, -1, data_dim))
out = mx.sym.transpose(out, axes=(1,0,2))

loss = mx.sym.mean(-mx.sym.pick(mx.sym.log_softmax(out), target, axis=-1))
loss  = mx.sym.make_loss(loss)

shape = {"data": (batch_size, dataX[0].shape[0])}
#mx.viz.plot_network(out, shape=shape)
#["cats", "dogs"] ==> [0, 1] ==> [[1, 0], [0, 1]]
# Module


net = mx.mod.Module(symbol=loss,
                   data_names=['data'], label_names=['target'],
                    context=mx.gpu(7)
                   )

net.bind(data_shapes=train_iter.provide_data, label_shapes=train_iter.provide_label)
net.init_params(initializer=mx.init.Xavier())
net.init_optimizer(optimizer='adam',
                  optimizer_params={'learning_rate': 1E-3},
                   kvstore=None
                  )
#Train

epochs = 100

total_batches = len(dataX) // batch_size

for epoch in range(epochs):
    avg_loss =0
    train_iter.reset()
    
    for i, data_batch in enumerate(train_iter):
        net.forward_backward(data_batch=data_batch)
        loss = net.get_outputs()[0].asscalar()
        avg_loss += loss
        net.update()
    avg_loss /= total_batches

    print epoch, "%.7f" % avg_loss
# test module
test_net = mx.mod.Module(symbol=out,
                         data_names=['data'],
                         label_names=None,
                         context=mx.gpu(7)) # FusedRNNCell works only with GPU

# data descriptor
data_desc = train_iter.provide_data[0]

# set shared_module = model used for training so as to share same parameters and memory
test_net.bind(data_shapes=[data_desc],
              label_shapes=None,
              for_training=False,
              grad_req='null',
              shared_module=net)

n_test = 100
testX, testY = generate_data(n_test)

testX = np.array(testX, dtype=np.int)

test_net.reshape(data_shapes=[mx.io.DataDesc('data', (1, input_sequence_length))])
predictions = test_net.predict(mx.io.NDArrayIter(testX, batch_size=1)).asnumpy()

print "expression", "predicted", "actual"

correct = 0
for i, prediction in enumerate(predictions):
    x_str = [character_set[j] for j in testX[i]]
    index = np.argmax(prediction, axis=1)
    result = [character_set[j] for j in index]
    label = [character_set[j] for j in testY[i]]
    #print result, label
    if result == label:
        correct +=1
    print "".join(x_str), "".join(result), "    ", "".join(label)
    
print correct, correct/(n_test*1.0)<jupyter_output>expression predicted actual
   4+3+5   12        12
   5*2*8   80        80
   9*7*1   63        63
   7+5+9   21        21
   5+6+9   20        20
   6*7*9  378       378
   6+2+8   16        16
   1*9*8   72        72
   4+9+4   17        17
   8+8+3   19        19
   2*3*7   42        42
   1+9+5   15        15
   2+4+1    7         7
   2*3*4   24        24
   3+5+8   16        16
   1*8*3   24        24
   6*8*6  288       288
 10+10+5   25        25
  10+1+7   18        18
  8+4+10   22        22
   7+4+4   15        15
   6*3*5   90        90
 10*10*7  700       700
  10+8+8   26        26
   7*8*7  392       392
   1*3*8   24        24
   2*5*9   90        90
  4*10*1   40        40
   9*4*8  288       288
   6+7+2   15        15
  8*10*1   80        80
   9+5+8   22        22
   5+4+3   12        12
   7*9*9  567       567
  8*10*7  560       560
   7+3+1   11        11
   9*9*5  405       405
   4*2*7   56        56
   6+1+2    9         9
  1+10+6   17        17
   8*8*9  57[...]
{'fraction_non_alphanumeric': 0.11136314538777274, 'fraction_numerical': 0.04882263987902355, 'mean_word_length': 2.9067510548523208, 'pattern_counts': {'":': 1, '<': 7, '<?xml version=': 0, '>': 9, 'https://': 6, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '561597', 'license': 'Apache-2.0', 'n_tokens_mistral': 3330, 'n_tokens_neox': 3094, 'n_words': 880, 'path': 'E2_sequence_modeling_basic_math.ipynb', 'repo_name': 'nandus12/tensorflow1', 'size': '9270'}