<jupyter_start><jupyter_text>Lab4: Sentiment with GRUs In this assignment, you'll convert the RNN sentiment classifier from last time into a **GRU** RNN sentiment classifier. While the small dataset and tiny vocabulary that we're using here (for speed) will limit the performance of the model, it should still do substantially better than the plain RNN.![](http://vignette1.wikia.nocookie.net/despicableme/images/b/ba/Gru.jpg/revision/latest/scale-to-width-down/250?cb=20130711023954)  Setup First, let's load the data as before.<jupyter_code>sst_home = '../trees'

import re
import random

# Let's do 2-way positive/negative classification instead of 5-way
easy_label_map = {0:0, 1:0, 2:None, 3:1, 4:1}

def load_sst_data(path):
    data = []
    with open(path) as f:
        for i, line in enumerate(f): 
            example = {}
            example['label'] = easy_label_map[int(line[1])]
            if example['label'] is None:
                continue
            
            # Strip out the parse information and the phrase labels---we don't need those here
            text = re.sub(r'\s*(\(\d)|(\))\s*', '', line)
            example['text'] = text[1:]
            data.append(example)

    random.seed(1)
    random.shuffle(data)
    return data
     
training_set = load_sst_data(sst_home + '/train.txt')
dev_set = load_sst_data(sst_home + '/dev.txt')
test_set = load_sst_data(sst_home + '/test.txt')

# Note: Unlike with k-nearest neighbors, evaluation here should be fast, and we don't need to
# trim down the dev and test sets.<jupyter_output><empty_output><jupyter_text>Next, we'll convert the data to index vectors.To simplify your implementation, we'll use a fixed unrolling length of 20. In the conversion process, we'll cut off excess words (towards the left/start end of the sentence), pad short sentences (to the left) with a special word symbol ``, and mark out-of-vocabulary words with ``, for unknown. As in the previous assignment, we'll use a very small vocabulary for this assignment, so you'll see `` often.<jupyter_code>import collections
import numpy as np

def sentence_to_padded_index_sequence(datasets):
    '''Annotates datasets with feature vectors.'''
    

    PADDING = "<PAD>"
    UNKNOWN = "<UNK>"
    SEQ_LEN = 20
    
    # Extract vocabulary
    def tokenize(string):
        return string.lower().split()
    
    word_counter = collections.Counter()
    for example in datasets[0]:
        word_counter.update(tokenize(example['text']))
    
    vocabulary = set([word for word in word_counter if word_counter[word] > 10])
    vocabulary = list(vocabulary)
    vocabulary = [PADDING, UNKNOWN] + vocabulary
        
    word_indices = dict(zip(vocabulary, range(len(vocabulary))))
    indices_to_words = {v: k for k, v in word_indices.items()}
        
    for i, dataset in enumerate(datasets):
        for example in dataset:
            example['index_sequence'] = np.zeros((SEQ_LEN), dtype=np.int32)
            
            token_sequence = tokenize(example['text'])
            padding = SEQ_LEN - len(token_sequence)
            
            for i in range(SEQ_LEN):
                if i >= padding:
                    if token_sequence[i - padding] in word_indices:
                        index = word_indices[token_sequence[i - padding]]
                    else:
                        index = word_indices[UNKNOWN]
                else:
                    index = word_indices[PADDING]
                example['index_sequence'][i] = index
    return indices_to_words, word_indices
    
indices_to_words, word_indices = sentence_to_padded_index_sequence([training_set, dev_set, test_set])
print(training_set[18])
print(len(word_indices))
def evaluate_classifier(classifier, eval_set):
    correct = 0
    hypotheses = classifier(eval_set)
    for i, example in enumerate(eval_set):
        hypothesis = hypotheses[i]
        if hypothesis == example['label']:
            correct += 1        
    return correct / float(len(eval_set))<jupyter_output><empty_output><jupyter_text>Assignments: Building the RNN The class below is a solved version of last week's RNN exercise. The only change I've made is to increase the learning rate, since GRUs are less likely to do crazy things during optimization. Your job is to convert it into a GRU model. You should have to:- **TODO1**: Add additional trained parameters.- **TODO2**: Modify the `step()` function.- **TODO3**: Modify L2 regularization to incorporate the new parameters.You shouldn't have to edit anything outside of `__init__()`.<jupyter_code>import tensorflow as tf
class RNNSentimentClassifier:
    def __init__(self, vocab_size, sequence_length):
        # Define the hyperparameters
        self.learning_rate = 1.0  # Should be about right
        self.training_epochs = 500  # How long to train for - chosen to fit within class time
        self.display_epoch_freq = 5  # How often to test and print out statistics
        self.dim = 12  # The dimension of the hidden state of the RNN
        self.embedding_dim = 8  # The dimension of the learned word embeddings
        self.batch_size = 256  # Somewhat arbitrary - can be tuned, but often tune for speed, not accuracy
        self.vocab_size = vocab_size  # Defined by the file reader above
        self.sequence_length = sequence_length  # Defined by the file reader above
        self.l2_lambda = 0.001
        
        # Define the parameters
        self.E = tf.Variable(tf.random_normal([self.vocab_size, self.embedding_dim], stddev=0.1))
        
        # now this is update gate
        self.W_rnn = tf.Variable(tf.random_normal([self.embedding_dim + self.dim, self.dim], stddev=0.1))
        self.b_rnn = tf.Variable(tf.random_normal([self.dim], stddev=0.1))                
        
        # TODO1: Add additional GRU parameters
        
        #forget gate
        self.W_rnn_z = tf.Variable(tf.random_normal([self.embedding_dim + self.dim, self.dim], stddev=0.1))
        self.b_rnn_z = tf.Variable(tf.random_normal([self.dim], stddev=0.1))
        
        #reset gate 
        self.W_rnn_t = tf.Variable(tf.random_normal([self.embedding_dim + self.dim, self.dim], stddev=0.1))
        self.b_rnn_t = tf.Variable(tf.random_normal([self.dim], stddev=0.1))

        
        self.W_cl = tf.Variable(tf.random_normal([self.dim, 2], stddev=0.1))
        self.b_cl = tf.Variable(tf.random_normal([2], stddev=0.1))
        
        # Define the placeholders
        self.x = tf.placeholder(tf.int32, [None, self.sequence_length])
        self.y = tf.placeholder(tf.int32, [None])
        
        # Define one step of the RNN
        # TODO2: Modify the step() function to compute GRU step
        def step(x, h_prev):
            emb = tf.nn.embedding_lookup(self.E, x)
            emb_h_prev = tf.concat([emb, h_prev], 1)
            #forget gate
            zt=tf.nn.sigmoid(tf.matmul(emb_h_prev, self.W_rnn_z)  + self.b_rnn_z)
            #reset gate
            rt=tf.nn.sigmoid(tf.matmul(emb_h_prev, self.W_rnn_t)  + self.b_rnn_t)
                        
            ht = tf.nn.tanh(tf.matmul(tf.concat([rt * h_prev, emb],1),self.W_rnn)  + self.b_rnn)
            
            h = (1 - zt) * h_prev + zt * ht
            return h
        
        # Split up the inputs into individual tensors
        self.x_slices = tf.split(self.x, self.sequence_length, 1)
        
        self.h_zero = tf.zeros([self.batch_size, self.dim])
        
        h_prev = self.h_zero
        
        # Unroll the RNN
        for t in range(self.sequence_length):
            x_t = tf.reshape(self.x_slices[t], [-1])
            h_prev = step(x_t, h_prev)
        
        # Compute the logits using one last linear layer
        self.logits = tf.matmul(h_prev, self.W_cl) + self.b_cl

        # Define the L2 cost
        # TODO3: Modify L2 regularization to incorporate the new parameters.
        self.l2_cost = self.l2_lambda * (tf.reduce_sum(tf.square(self.W_rnn)) +
                                         tf.reduce_sum(tf.square(self.W_rnn_z)) +
                                         tf.reduce_sum(tf.square(self.W_rnn_t)) +
                                         tf.reduce_sum(tf.square(self.W_cl)) +
                                         tf.reduce_sum(tf.square(self.b_rnn)) +
                                         tf.reduce_sum(tf.square(self.b_rnn_z)) +
                                         tf.reduce_sum(tf.square(self.b_rnn_t)) +
                                         tf.reduce_sum(tf.square(self.b_cl)))
        
        # Define the cost function (here, the softmax exp and sum are built in)
        self.total_cost = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(logits=self.logits, labels=self.y)
                                         + self.l2_cost)
        
        # This  performs the main SGD update equation with gradient clipping
        optimizer_obj = tf.train.GradientDescentOptimizer(learning_rate=self.learning_rate)
        gvs = optimizer_obj.compute_gradients(self.total_cost)
        capped_gvs = [(tf.clip_by_norm(grad, 5.0), var) for grad, var in gvs if grad is not None]
        self.optimizer = optimizer_obj.apply_gradients(capped_gvs)
        
        # Create an operation to fill zero values in for W and b
        self.init = tf.global_variables_initializer()
        
        # Create a placeholder for the session that will be shared between training and evaluation
        self.sess = None
        
    def train(self, training_data, dev_set):
        def get_minibatch(dataset, start_index, end_index):
            indices = range(start_index, end_index)
            vectors = np.vstack([dataset[i]['index_sequence'] for i in indices])
            labels = [dataset[i]['label'] for i in indices]
            return vectors, labels
        
        self.sess = tf.Session()
        
        self.sess.run(self.init)
        print ('Training.')

        # Training cycle
        for epoch in range(self.training_epochs):
            random.shuffle(training_set)
            avg_cost = 0.
            total_batch = int(len(training_set) / self.batch_size)
            
            # Loop over all batches in epoch
            for i in range(total_batch):
                # Assemble a minibatch of the next B examples
                minibatch_vectors, minibatch_labels = get_minibatch(training_set, 
                                                                    self.batch_size * i, 
                                                                    self.batch_size * (i + 1))

                # Run the optimizer to take a gradient step, and also fetch the value of the 
                # cost function for logging
                _, c = self.sess.run([self.optimizer, self.total_cost], 
                                     feed_dict={self.x: minibatch_vectors,
                                                self.y: minibatch_labels})
                                                                    
                # Compute average loss
                avg_cost += c / total_batch
                
            # Display some statistics about the step
            # Evaluating only one batch worth of data -- simplifies implementation slightly
            if (epoch+1) % self.display_epoch_freq == 0:
                print("Epoch:", (epoch+1), "Cost:", avg_cost, \
                    "Dev acc:", evaluate_classifier(self.classify, dev_set[0:256]), \
                    "Train acc:", evaluate_classifier(self.classify, training_set[0:256]))  
    
    def classify(self, examples):
        # This classifies a list of examples
        vectors = np.vstack([example['index_sequence'] for example in examples])
        logits = self.sess.run(self.logits, feed_dict={self.x: vectors})
        return np.argmax(logits, axis=1)<jupyter_output><empty_output><jupyter_text>Now let's train it. If the GRU is doing what it should, you should reach 74% accuracy within your first 200 epochs—a substantial improvement over the 70% figure we saw last week.<jupyter_code>classifier = RNNSentimentClassifier(len(word_indices), 20)
classifier.train(training_set, dev_set)<jupyter_output>Training.
Epoch: 5 Cost: 0.697890829157 Dev acc: 0.5546875 Train acc: 0.515625
Epoch: 10 Cost: 0.695686269689 Dev acc: 0.5546875 Train acc: 0.60546875
Epoch: 15 Cost: 0.694159534242 Dev acc: 0.5546875 Train acc: 0.51953125
Epoch: 20 Cost: 0.693850585708 Dev acc: 0.5546875 Train acc: 0.5546875
Epoch: 25 Cost: 0.693378640546 Dev acc: 0.5546875 Train acc: 0.5390625
Epoch: 30 Cost: 0.692986903367 Dev acc: 0.5546875 Train acc: 0.5390625
Epoch: 35 Cost: 0.692858073446 Dev acc: 0.5546875 Train acc: 0.5390625
Epoch: 40 Cost: 0.692991918988 Dev acc: 0.5546875 Train acc: 0.54296875
Epoch: 45 Cost: 0.69298329839 Dev acc: 0.5546875 Train acc: 0.5390625
Epoch: 50 Cost: 0.693066314415 Dev acc: 0.5859375 Train acc: 0.55859375
Epoch: 55 Cost: 0.692937431512 Dev acc: 0.4453125 Train acc: 0.48828125
Epoch: 60 Cost: 0.692489065506 Dev acc: 0.5625 Train acc: 0.55859375
Epoch: 65 Cost: 0.691304162697 Dev acc: 0.56640625 Train acc: 0.57421875
Epoch: 70 Cost: 0.690803922989 Dev acc: 0.5625 Train acc: 0.51171[...]
{'fraction_non_alphanumeric': 0.0949265992241576, 'fraction_numerical': 0.042595268882634825, 'mean_word_length': 2.274719800747198, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 25, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '593246', 'license': 'Apache-2.0', 'n_tokens_mistral': 4373, 'n_tokens_neox': 3943, 'n_words': 1104, 'path': 'labs/lab4/GRU based classifier.ipynb', 'repo_name': 'isanvicente/dl4nlp-ehu', 'size': '13147'}