<jupyter_start><jupyter_text>Day 6: Sequence Models in Deep Learning  Exercise 6.1 Convince yourself that a RNN is just an FF unfolded in time. Complete the `backpropagation()` method in the `NumpyRNN` class in     lxmls/deep_learning/numpy_models/rnn.py     and compare it with    lxmls/deep_learning/numpy_models.mlp.py.  WSJ DataTo work with RNNs we will use the Part-of-speech data-set seen in the sequence models day.<jupyter_code># Load Part-of-Speech data 
from lxmls.readers.pos_corpus import PostagCorpusData
data = PostagCorpusData()<jupyter_output><empty_output><jupyter_text>Load and configure the NumpyRNN. Remember to use reload if you want to modify the code inside the rnns module<jupyter_code>from lxmls.deep_learning.numpy_models.rnn import NumpyRNN
model = NumpyRNN(
    input_size=data.input_size,
    embedding_size=50,
    hidden_size=20,
    output_size=data.output_size,
    learning_rate=0.1
)<jupyter_output><empty_output><jupyter_text>Milestone 1:As in the case of the feed-forward networks you can use the following setup to test step by step the implementation of the gradients. First compute the cost variation for the variation of a single weight<jupyter_code>print([x.shape for x in model.parameters])
from lxmls.deep_learning.rnn import get_rnn_parameter_handlers, get_rnn_loss_range

# Get functions to get and set values of a particular weight of the model
get_parameter, set_parameter = get_rnn_parameter_handlers(
    layer_index=-1,
    row=0, 
    column=0
)

# Get batch of data
batch = data.batches('train', batch_size=1)[0]

# Get loss and weight value
current_loss = model.cross_entropy_loss(batch['input'], batch['output'])
current_weight = get_parameter(model.parameters)

# Get range of values of the weight and loss around current parameters values
weight_range, loss_range = get_rnn_loss_range(model, get_parameter, set_parameter, batch)<jupyter_output><empty_output><jupyter_text>Then conmpute the desired gradient from your implementation<jupyter_code># Get the gradient value for that weight
gradients = model.backpropagation(batch['input'], batch['output'])
current_gradient = get_parameter(gradients)<jupyter_output><empty_output><jupyter_text>And finally call matlplotlib to plot the loss variation versus the gradient<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt
# Plot empirical
plt.plot(weight_range, loss_range)
plt.plot(current_weight, current_loss, 'xr')
plt.ylabel('loss value')
plt.xlabel('weight value')
# Plot real
h = plt.plot(
    weight_range,
    current_gradient*(weight_range - current_weight) + current_loss, 
    'r--'
)
plt.show()<jupyter_output><empty_output><jupyter_text>Milestone 2:After you have completed the gradients you can run the model in the POS task<jupyter_code># Hyper-parameters
num_epochs = 20

import numpy as np

# Get batch iterators for train and test
train_batches = data.batches('train', batch_size=1)
dev_set = data.batches('dev', batch_size=1)
test_set = data.batches('test', batch_size=1)

# Epoch loop
import time
start = time.time()
for epoch in range(num_epochs):

    # Batch loop
    for batch in train_batches:
        model.update(input=batch['input'], output=batch['output'])

    # Evaluation dev
    is_hit = []
    for batch in dev_set:
        is_hit.extend(model.predict(input=batch['input']) == batch['output'])
    accuracy = 100*np.mean(is_hit)
    print("Epoch %d: dev accuracy %2.2f %%" % (epoch+1, accuracy))

print("Training took %2.2f seconds per epoch" % ((time.time() - start)/num_epochs))    
    
# Evaluation test
is_hit = []
for batch in test_set:
    is_hit.extend(model.predict(input=batch['input']) == batch['output'])
accuracy = 100*np.mean(is_hit)

# Inform user
print("Test accuracy %2.2f %%" % accuracy)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09696169088507266, 'fraction_numerical': 0.008982826948480845, 'mean_word_length': 5.460750853242321, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 5}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '256997', 'license': 'MIT', 'n_tokens_mistral': 1267, 'n_tokens_neox': 1226, 'n_words': 348, 'path': 'labs/notebooks/non_linear_sequence_classifiers/exercise_1.ipynb', 'repo_name': 'jazcarretao/lxmls2018_juan', 'size': '3785'}