<jupyter_start><jupyter_text>Assignment 4: A GRU-pair model for SNLI with attention In this assignment we'll build and train an attention-based model for SNLI. The model will incorporate the basic word-by-word attention+matching layer design from [<NAME> Jiang](https://arxiv.org/pdf/1512.08849v2.pdf), but it will use GRUs instead of LSTMs, and it will use [Luong et al.](http://www.aclweb.org/anthology/D15-1166)'s 'general' bilinear attention scoring function.Do not start this assignment at the last minute, since the model you build may take several hours to train.  Setup You'll need to download and unzip SNLI, which you can find [here](http://nlp.stanford.edu/projects/snli/). Set `snli_home` below to point to it. The following block of code loads it.<jupyter_code>snli_home = '../snli_1.0'

import re
import random
import json

LABEL_MAP = {
    "entailment": 0,
    "neutral": 1,
    "contradiction": 2
}

def load_snli_data(path):
    data = []
    with open(path) as f:
        for line in f:
            loaded_example = json.loads(line)
            if loaded_example["gold_label"] not in LABEL_MAP:
                continue
            loaded_example["label"] = LABEL_MAP[loaded_example["gold_label"]]
            data.append(loaded_example)
        random.seed(1)
        random.shuffle(data)
    return data
     
training_set = load_snli_data(snli_home + '/snli_1.0_train.jsonl')
dev_set = load_snli_data(snli_home + '/snli_1.0_dev.jsonl')
test_set = load_snli_data(snli_home + '/snli_1.0_test.jsonl')<jupyter_output><empty_output><jupyter_text>Next, we'll convert the data to index vectors in the same way that we've done for in-class exercises with RNN-based sentiment models. A few notes:- We use a sequence length of only 10, which is short enough that we're truncating a large fraction of sentences.- Tokenization is easy here because we're relying on the output of a parser (which does tokenization as part of parsing), just as with the SST corpus that we've been using until now. Note that we use the 'sentence1_binary_parse' field of each example rather than the human-readable 'sentence1'.- We're using a moderately large vocabulary of about 36k words.<jupyter_code>SEQ_LEN = 10

import collections
import numpy as np

def sentences_to_padded_index_sequences(datasets):
    '''Annotates datasets with feature vectors.'''
    
    PADDING = "<PAD>"
    UNKNOWN = "<UNK>"
    
    # Extract vocabulary
    def tokenize(string):
        string = re.sub(r'\(|\)', '', string)
        return string.lower().split()
    
    word_counter = collections.Counter()
    for example in datasets[0]:
        word_counter.update(tokenize(example['sentence1_binary_parse']))
        word_counter.update(tokenize(example['sentence2_binary_parse']))
        
    vocabulary = set([word for word in word_counter])
    vocabulary = list(vocabulary)
    vocabulary = [PADDING, UNKNOWN] + vocabulary
        
    word_indices = dict(zip(vocabulary, range(len(vocabulary))))
    indices_to_words = {v: k for k, v in word_indices.items()}
        
    for i, dataset in enumerate(datasets):
        for example in dataset:
            for sentence in ['sentence1_binary_parse', 'sentence2_binary_parse']:
                example[sentence + '_index_sequence'] = np.zeros((SEQ_LEN), dtype=np.int32)

                token_sequence = tokenize(example[sentence])
                padding = SEQ_LEN - len(token_sequence)

                for i in range(SEQ_LEN):
                    if i >= padding:
                        if token_sequence[i - padding] in word_indices:
                            index = word_indices[token_sequence[i - padding]]
                        else:
                            index = word_indices[UNKNOWN]
                    else:
                        index = word_indices[PADDING]
                    example[sentence + '_index_sequence'][i] = index
    return indices_to_words, word_indices
    
indices_to_words, word_indices = sentences_to_padded_index_sequences([training_set, dev_set, test_set])
print training_set[6]
print len(word_indices)<jupyter_output><empty_output><jupyter_text>Now we load GloVe. You'll need the same file that you used for the in-class exercise on word embeddings.<jupyter_code>glove_home = '../'
words_to_load = 50000

with open(glove_home + 'glove.6B.50d.txt') as f:
    loaded_embeddings = np.zeros((len(word_indices), 50), dtype='float32')
    for i, line in enumerate(f):
        if i >= words_to_load: 
            break
        
        s = line.split()
        if s[0] in word_indices:
            loaded_embeddings[word_indices[s[0]], :] = np.asarray(s[1:])<jupyter_output><empty_output><jupyter_text>Now we set up an evaluation function as before.<jupyter_code>def evaluate_classifier(classifier, eval_set):
    correct = 0
    hypotheses = classifier(eval_set)
    for i, example in enumerate(eval_set):
        hypothesis = hypotheses[i]
        if hypothesis == example['label']:
            correct += 1        
    return correct / float(len(eval_set))<jupyter_output><empty_output><jupyter_text>Assignment Part Expand the below starter code to build a working attention-based NLI model. The model should feature the following:- 50D word embeddings initialized with GloVe and trained. (Using self.E should provide this.)- Three GRUs (with two different sets of parameters): a premise encoder, a hypothesis encoder (sharing the same parameters), and matching GRU, taking the place of [Wang and Jiang](https://arxiv.org/pdf/1512.08849v2.pdf)'s mLSTM, with its own parameters. There shouldn't be a connection between the premise and hypothesis GRUs: the starting hidden state for each of the three GRUs should be zeros.- Word-by-word attention using [Luong et al.](http://www.aclweb.org/anthology/D15-1166)'s 'general' (bilinear) scoring function.- A three-way softmax classifier whose input is the final hidden state of the matching RNN.You are welcome to use code from solutions to past exercises. As ever, do not use any other pre-written code or TF functions that are specific to RNNs or attention. As in class, the below line may give you a warning. It should be safe to ignore.<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt
import tensorflow as tf
class RNNEntailmentClassifier:
    def __init__(self, vocab_size, sequence_length):
        # Define the hyperparameters
        self.training_epochs = 30  # How long to train for - chosen to fit within class time
        self.display_epoch_freq = 1  # How often to print cost (in epochs)
        self.display_step_freq = 250  # How often to test (in steps)
        self.dim = 24  # The dimension of the hidden state of each RNN
        self.embedding_dim = 50  # The dimension of the learned word embeddings
        self.batch_size = 64  # Somewhat arbitrary - can be tuned, but often tune for speed, not accuracy
        self.vocab_size = vocab_size  # Defined by the file reader above
        self.sequence_length = sequence_length  # Defined by the file reader above
        
        # Define the parameters
        self.E = tf.Variable(loaded_embeddings, trainable=False)
        
        # TODO: Define the rest of the parameters
        #  - Note that we need to use three GRUs, but two of them share parameters
        #  - You can re-use part of you code in lab5
        
        # Define the placeholders
        self.premise_x = tf.placeholder(tf.int32, [None, self.sequence_length])
        self.hypothesis_x = tf.placeholder(tf.int32, [None, self.sequence_length])
        self.y = tf.placeholder(tf.int32, [None])
        
        # TODO: Define the model
        ## - Define GRU function. You can re-use the code for lab5 
        ## - Define one step of the premise encoder RNN
        ## - Define one step of the hypothesis encoder RNN. Note that attention mechanism is
        
        
        
        # TODO: Define the logits here
        #  - Split up the inputs into individual tensors
        #  - Unroll the first RNN (Premise)
        #  - Unroll the second RNN (Hypothesis). You can compute the attention inside the step, 
        #   this means that you need to use the two RNN layers defined in Luong et al. 
        
        self.logits = None
        
        # Define the cost function (here, the softmax exp and sum are built in)
        self.total_cost = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(self.logits, self.y))
        
        # This  performs the main SGD update equation with gradient clipping
        optimizer_obj = tf.train.AdamOptimizer()
        gvs = optimizer_obj.compute_gradients(self.total_cost)
        capped_gvs = [(tf.clip_by_norm(grad, 5.0), var) for grad, var in gvs if grad is not None]
        self.optimizer = optimizer_obj.apply_gradients(capped_gvs)
        
        # Create an operation to fill zero values in for W and b
        self.init = tf.initialize_all_variables()

        # Create a placeholder for the session that will be shared between training and evaluation
        self.sess = tf.Session()
        
        # Initialize variables
        self.sess.run(self.init)
        self.step = 1
        self.epoch = 0
        
    def train(self, training_data, dev_data):
        def get_minibatch(dataset, start_index, end_index):
            indices = range(start_index, end_index)
            premise_vectors = np.vstack([dataset[i]['sentence1_binary_parse_index_sequence'] for i in indices])
            hypothesis_vectors = np.vstack([dataset[i]['sentence2_binary_parse_index_sequence'] for i in indices])
            labels = [dataset[i]['label'] for i in indices]
            return premise_vectors, hypothesis_vectors, labels

        print('Training.')

        # Training cycle
        for _ in range(self.training_epochs):
            random.shuffle(training_data)
            avg_cost = 0.
            total_batch = int(len(training_data) / self.batch_size)
            
            # Loop over all batches in epoch
            for i in range(total_batch):
                # Assemble a minibatch of the next B examples
                minibatch_premise_vectors, minibatch_hypothesis_vectors, minibatch_labels = get_minibatch(
                    training_data, self.batch_size * i, self.batch_size * (i + 1))

                # Run the optimizer to take a gradient step, and also fetch the value of the 
                # cost function for logging
                _, c = self.sess.run(
                     [self.optimizer, self.total_cost], 
                     feed_dict={self.premise_x: minibatch_premise_vectors,
                                self.hypothesis_x: minibatch_hypothesis_vectors,
                                self.y: minibatch_labels})

                if self.step % self.display_step_freq == 0:
                    print("Step:", self.step, "Dev acc:", evaluate_classifier(self.classify, dev_data[0:1000]), \
                        "Train acc:", evaluate_classifier(self.classify, training_data[0:1000])) 
                                  
                self.step += 1
                avg_cost += c / (total_batch * self.batch_size)
                                
            # Display some statistics about the step
            # Evaluating only one batch worth of data -- simplifies implementation slightly
            if self.epoch % self.display_epoch_freq == 0:
                print("Epoch:", (self.epoch + 1), "Cost:", avg_cost) 
            self.epoch += 1
    
    def classify(self, examples):
        # This classifies a list of examples
        premise_vectors = np.vstack([example['sentence1_binary_parse_index_sequence'] for example in examples])
        hypothesis_vectors = np.vstack([example['sentence2_binary_parse_index_sequence'] for example in examples])
        logits = self.sess.run(self.logits,
                               feed_dict={self.premise_x: premise_vectors,
                                          self.hypothesis_x: hypothesis_vectors})

        return np.argmax(logits, axis=1)

    def get_attn(self, examples):
        premise_vectors = np.vstack([example['sentence1_binary_parse_index_sequence'] for example in examples])
        hypothesis_vectors = np.vstack([example['sentence2_binary_parse_index_sequence'] for example in examples])
        attn_weights = self.sess.run(self.complete_attn_weights, 
                                     feed_dict={self.premise_x: premise_vectors,
                                                self.hypothesis_x: hypothesis_vectors})
        return np.reshape(attn_weights, [len(examples), 10, 10])
        
    def plot_attn(self, examples):
        attn_weights = self.get_attn(examples)

        for i in range(len(examples)):    
            fig = plt.figure()
            ax = fig.add_subplot(111)
            ax.matshow(attn_weights[i,:,:], vmin=0., vmax=1., cmap=plt.cm.inferno)

            premise_tokens = [indices_to_words[index] for index in examples[i]['sentence1_binary_parse_index_sequence']]
            hypothesis_tokens = [indices_to_words[index] for index in examples[i]['sentence2_binary_parse_index_sequence']]

            ax.set_yticklabels(premise_tokens)
            ax.set_xticklabels(hypothesis_tokens, rotation=45)

            plt.xticks(np.arange(0, 10, 1.0))
            plt.yticks(np.arange(0, 10, 1.0))<jupyter_output><empty_output><jupyter_text>This creates a classifier and initializes it. Note that starting and stopping training will not re-initialize the model.<jupyter_code>classifier = RNNEntailmentClassifier(len(word_indices), SEQ_LEN)<jupyter_output><empty_output><jupyter_text>Running this line tests that you've successfully avoided a common error in implementing attention: mixing up dimensions in a way that allows information about one example in a batch to impact the results for other examples. It should do nothing if the model behaves correctly, and will throw an error otherwise.<jupyter_code>assert (classifier.get_attn(training_set[0:2])[0, :, :] == \
        classifier.get_attn(training_set[0:3])[0, :, :]).all(), \
       'Warning: There is cross-example information flow.'<jupyter_output><empty_output><jupyter_text>Run the model below. Your goal is dev set performance above 74% at at least a few steps within the first 30 epochs. This may require running your model overnight, though it won't necessarily require running for all 30 epochs. Since a single epoch can take a long time with this model, you'll see accuracy results every 250 steps rather than every epoch.<jupyter_code>classifier.train(training_set, dev_set)<jupyter_output><empty_output><jupyter_text>This will print some visualizations for the first ten training examples. It should be useful in the questions below.<jupyter_code>classifier.plot_attn(training_set[0:10])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08300662430715154, 'fraction_numerical': 0.011423550087873463, 'mean_word_length': 2.786792935756335, 'pattern_counts': {'":': 3, '<': 40, '<?xml version=': 0, '>': 42, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '134825', 'license': 'Apache-2.0', 'n_tokens_mistral': 4409, 'n_tokens_neox': 4152, 'n_words': 1318, 'path': 'assignments/assignment4/word-by-word_attention.ipynb', 'repo_name': 'isanvicente/dl4nlp-ehu', 'size': '14796'}