<jupyter_start><jupyter_text>Sentiment Analysis with an RNNIn this notebook, you'll implement a recurrent neural network that performs sentiment analysis. >Using an RNN rather than a strictly feedforward network is more accurate since we can include information about the *sequence* of words. Here we'll use a dataset of movie reviews, accompanied by sentiment labels: positive or negative. Network ArchitectureThe architecture for this network is shown below.>**First, we'll pass in words to an embedding layer.** We need an embedding layer because we have tens of thousands of words, so we'll need a more efficient representation for our input data than one-hot encoded vectors. You should have seen this before from the Word2Vec lesson. You can actually train an embedding with the Skip-gram Word2Vec model and use those embeddings as input, here. However, it's good enough to just have an embedding layer and let the network learn a different embedding table on its own. *In this case, the embedding layer is for dimensionality reduction, rather than for learning semantic representations.*>**After input words are passed to an embedding layer, the new embeddings will be passed to LSTM cells.** The LSTM cells will add *recurrent* connections to the network and give us the ability to include information about the *sequence* of words in the movie review data. >**Finally, the LSTM outputs will go to a sigmoid output layer.** We're using a sigmoid function because positive and negative = 1 and 0, respectively, and a sigmoid will output predicted, sentiment values between 0-1. We don't care about the sigmoid outputs except for the **very last one**; we can ignore the rest. We'll calculate the loss by comparing the output at the last time step and the training label (pos or neg). --------------I based my solution on the one at:    https://www.kaggle.com/nilanml/imdb-review-deep-model-94-89-accuracy<jupyter_code># Prettily debug any object passed

import inspect
import re

def describe(arg):
    frame = inspect.currentframe()
    callerframeinfo = inspect.getframeinfo(frame.f_back)
    try:
        context = inspect.getframeinfo(frame.f_back).code_context
        caller_lines = ''.join([line.strip() for line in context])
        m = re.search(r'describe\s*\((.+?)\)$', caller_lines)
        if m:
            caller_lines = m.group(1)
            position = str(callerframeinfo.filename) + "@" + str(callerframeinfo.lineno)

            # Add additional info such as shape, length, datatype
            additional = []

            if hasattr(arg, "dtype"):
                additional.append("{}".format(arg.dtype))

            additional.append(type(arg).__qualname__)
            
            if hasattr(arg, "shape"):
                additional.append("shape={}".format(tuple(arg.shape)))
            elif hasattr(arg, "__len__"):  # shape includes length information
                additional.append("[len={}]".format(len(arg)))

                    
            # Use str() representation if it is printable
            str_arg = str(arg)
            str_arg = str_arg if str_arg.isprintable() else repr(arg)

            print(position, "describe(" + caller_lines + ") = ", end='')
            print(" ".join(additional) + " ", end='')
            if "\n" in str_arg: print()  # It's multi-line
            print(str_arg)
        else:
            print("Describe: couldn't find caller context")

    finally:
        del frame
        del callerframeinfo<jupyter_output><empty_output><jupyter_text>--- Load in and visualize the data<jupyter_code>import numpy as np

# read data from text files
with open('data/reviews.txt', 'r') as f:
    reviews = f.read()
with open('data/labels.txt', 'r') as f:
    labels = f.read()
print(reviews[:2000])
print()
print(labels[:20])<jupyter_output>bromwell high is a cartoon comedy . it ran at the same time as some other programs about school life  such as  teachers  . my   years in the teaching profession lead me to believe that bromwell high  s satire is much closer to reality than is  teachers  . the scramble to survive financially  the insightful students who can see right through their pathetic teachers  pomp  the pettiness of the whole situation  all remind me of the schools i knew and their students . when i saw the episode in which a student repeatedly tried to burn down the school  i immediately recalled . . . . . . . . . at . . . . . . . . . . high . a classic line inspector i  m here to sack one of your teachers . student welcome to bromwell high . i expect that many adults of my age think that bromwell high is far fetched . what a pity that it isn  t   
story of a man who has unnatural feelings for a pig . starts out with a opening scene that is a terrific example of absurd comedy . a formal orchestra audience is turn[...]<jupyter_text>Data pre-processingThe first step when building a neural network model is getting your data into the proper form to feed into the network. Since we're using embedding layers, we'll need to encode each word with an integer. We'll also want to clean it up a bit.You can see an example of the reviews data above. Here are the processing steps, we'll want to take:>* We'll want to get rid of periods and extraneous punctuation.* Also, you might notice that the reviews are delimited with newline characters `\n`. To deal with those, I'm going to split the text into each review using `\n` as the delimiter. * Then I can combined all the reviews back together into one big string.First, let's remove all punctuation. Then get all the text without the newlines and split it into individual words.<jupyter_code>from string import punctuation

print(f'Punctuation chars: {punctuation}')

# Return list of reviews, one line per element
def clean_data(reviews):    
    # Remove punctuation
    reviews = ''.join([c for c in reviews if c not in punctuation])
    
    # Split into lines and lowercase
    lines = [line.strip().lower() for line in reviews.split('\n') if line]
    return lines

reviews_split = clean_data(reviews);
# split by new lines and spaces
# reviews_split = [line.strip().lower() for line in reviews.split('\n') if line]
all_text = ' '.join(reviews_split)

# create a list of words
words = all_text.split()

# Show first items of split words
print(', '.join(map(repr, words[:30])))<jupyter_output>'bromwell', 'high', 'is', 'a', 'cartoon', 'comedy', 'it', 'ran', 'at', 'the', 'same', 'time', 'as', 'some', 'other', 'programs', 'about', 'school', 'life', 'such', 'as', 'teachers', 'my', 'years', 'in', 'the', 'teaching', 'profession', 'lead', 'me'<jupyter_text>Encoding the wordsThe embedding lookup requires that we pass in integers to our network. The easiest way to do this is to create dictionaries that map the words in the vocabulary to integers. Then we can convert each of our reviews into integers so they can be passed into the network.> **Exercise:** Now you're going to encode the words with integers. Build a dictionary that maps words to integers. Later we're going to pad our input vectors with zeros, so make sure the integers **start at 1, not 0**.> Also, convert the reviews to integers and store the reviews in a new list called `reviews_ints`.<jupyter_code>## Build a dictionary that maps words to integers

from collections import Counter
word_freq = Counter(words).most_common()  # sorted by frequency
# Can use set() if sorting isn't important

vocab_to_int = {word: index for index, (word, _) in enumerate(word_freq, start=1)}

# print(vocab_to_int['the'])<jupyter_output><empty_output><jupyter_text>Use vocab_to_int to tokenize each review in reviews_split<jupyter_code># reviews_ints = [[vocab_to_int[word] for word in line.split()] for line in reviews_split]

def reviews_to_ints(reviews):
    '''Convert review tokens to integers
    param reviews: list of reviews'''
    return [[vocab_to_int[word] for word in line.split()] for line in reviews]

reviews_ints = reviews_to_ints(reviews_split)<jupyter_output><empty_output><jupyter_text>**Test your code**As a text that you've implemented the dictionary correctly, print out the number of unique words in your vocabulary and the contents of the first, tokenized review.<jupyter_code># stats about vocabulary
print('Unique words: ', len((vocab_to_int)))  # should ~ 74000+
print()

print(reviews_split[0])

# print tokens in first review
print('\nTokenized review: \n', reviews_ints[:1])<jupyter_output>Unique words:  74072

bromwell high is a cartoon comedy  it ran at the same time as some other programs about school life  such as  teachers   my   years in the teaching profession lead me to believe that bromwell high  s satire is much closer to reality than is  teachers   the scramble to survive financially  the insightful students who can see right through their pathetic teachers  pomp  the pettiness of the whole situation  all remind me of the schools i knew and their students  when i saw the episode in which a student repeatedly tried to burn down the school  i immediately recalled          at           high  a classic line inspector i  m here to sack one of your teachers  student welcome to bromwell high  i expect that many adults of my age think that bromwell high is far fetched  what a pity that it isn  t

Tokenized review: 
 [[21025, 308, 6, 3, 1050, 207, 8, 2138, 32, 1, 171, 57, 15, 49, 81, 5785, 44, 382, 110, 140, 15, 5194, 60, 154, 9, 1, 4975, 5852, 475, 71, 5, 260, 12, 210[...]<jupyter_text>Encoding the labelsOur labels are "positive" or "negative". To use these labels in our network, we need to convert them to 0 and 1.> **Exercise:** Convert labels from `positive` and `negative` to 1 and 0, respectively, and place those in a new list, `encoded_labels`.<jupyter_code># Remove dups and capitals
cleaned_labels = [line.strip().lower() for line in labels.split('\n') if line]

# 1=positive, 0=negative label conversion
label_to_int = {'negative': 0, 'positive': 1}

encoded_labels = [label_to_int[label] for label in cleaned_labels]

assert len(encoded_labels) == len(reviews_ints), "Mismatch of number of labels and reviews"<jupyter_output><empty_output><jupyter_text>Removing OutliersAs an additional pre-processing step, we want to make sure that our reviews are in good shape for standard processing. That is, our network will expect a standard input text size, and so, we'll want to shape our reviews into a specific length. We'll approach this task in two main steps:1. Getting rid of extremely long or short reviews; the outliers2. Padding/truncating the remaining data so that we have reviews of the same length.Before we pad our review text, we should check for reviews of extremely short or long lengths; outliers that may mess with our training.<jupyter_code># outlier review stats
review_lens = Counter([len(x) for x in reviews_ints])
print("Zero-length reviews: {}".format(review_lens[0]))
print("Maximum review length: {}".format(max(review_lens)))<jupyter_output>Zero-length reviews: 0
Maximum review length: 2514<jupyter_text>Okay, a couple issues here. We seem to have one review with zero length. And, the maximum review length is way too many steps for our RNN. We'll have to remove any super short reviews and truncate super long reviews. This removes outliers and should allow our model to train more efficiently.> **Exercise:** First, remove *any* reviews with zero length from the `reviews_ints` list and their corresponding label in `encoded_labels`.<jupyter_code>print('Number of reviews before removing outliers: ', len(reviews_ints))

## remove any reviews/labels with zero length from the reviews_ints list.

# Equivalent to:
# reviews_ints = [review for review in reviews_ints if review]
reviews_ints = list(filter(None, reviews_ints)) 

# encoded_labels has already been cleaned above.

print('Number of reviews after removing outliers: ', len(reviews_ints))<jupyter_output>Number of reviews before removing outliers:  25000
Number of reviews after removing outliers:  25000<jupyter_text>--- Padding sequencesTo deal with both short and very long reviews, we'll pad or truncate all our reviews to a specific length. For reviews shorter than some `seq_length`, we'll pad with 0s. For reviews longer than `seq_length`, we can truncate them to the first `seq_length` words. A good `seq_length`, in this case, is 200.> **Exercise:** Define a function that returns an array `features` that contains the padded data, of a standard size, that we'll pass to the network. * The data should come from `review_ints`, since we want to feed integers to the network. * Each row should be `seq_length` elements long. * For reviews shorter than `seq_length` words, **left pad** with 0s. That is, if the review is `['best', 'movie', 'ever']`, `[117, 18, 128]` as integers, the row will look like `[0, 0, 0, ..., 0, 117, 18, 128]`. * For reviews longer than `seq_length`, use only the first `seq_length` words as the feature vector.As a small example, if the `seq_length=10` and an input review is: ```[117, 18, 128]```The resultant, padded sequence should be: ```[0, 0, 0, 0, 0, 0, 0, 117, 18, 128]```**Your final `features` array should be a 2D array, with as many rows as there are reviews, and as many columns as the specified `seq_length`.**This isn't trivial and there are a bunch of ways to do this. But, if you're going to be building your own deep learning networks, you're going to have to get used to preparing your data.<jupyter_code>def pad_features(reviews_ints, seq_length):
    ''' Return features of review_ints, where each review is padded with 0's 
        or truncated to the input seq_length.
    '''
    padding = lambda ints: [0] * (seq_length - len(ints))
    
    features = [padding(review) + review[:seq_length] for review in reviews_ints]
    
    return features

test = [ [1, 2, 3], [1, 2], [1] ]
assert pad_features(test, 2) ==  [ [1, 2], [1, 2], [0, 1] ], '0s are not padded correctly'
# Test your implementation!

seq_length = 200

features = pad_features(reviews_ints, seq_length=seq_length)

# Further tests
assert len(features)==len(reviews_ints), "Your features should have as many rows as reviews."

for i, r in enumerate(features):
    assert len(r) == 200, f'Row {i} should contain seq_length values.'

features = np.array(features)
print(features[:20,:12])<jupyter_output>[[    0     0     0     0     0     0     0     0     0     0     0     0]
 [    0     0     0     0     0     0     0     0     0     0     0     0]
 [22382    42 46418    15   706 17139  3389    47    77    35  1819    16]
 [ 4505   505    15     3  3342   162  8312  1652     6  4819    56    17]
 [    0     0     0     0     0     0     0     0     0     0     0     0]
 [    0     0     0     0     0     0     0     0     0     0     0     0]
 [    0     0     0     0     0     0     0     0     0     0     0     0]
 [    0     0     0     0     0     0     0     0     0     0     0     0]
 [    0     0     0     0     0     0     0     0     0     0     0     0]
 [   54    10    14   116    60   798   552    71   364     5     1   730]
 [    0     0     0     0     0     0     0     0     0     0     0     0]
 [    0     0     0     0     0     0     0     0     0     0     0     0]
 [    0     0     0     0     0     0     0     0     0     0     0     0]
 [    1   330   578    34[...]<jupyter_text>Training, Validation, TestWith our data in nice shape, we'll split it into training, validation, and test sets.> **Exercise:** Create the training, validation, and test sets. * You'll need to create sets for the features and the labels, `train_x` and `train_y`, for example. * Define a split fraction, `split_frac` as the fraction of data to **keep** in the training set. Usually this is set to 0.8 or 0.9. * Whatever data is left will be split in half to create the validation and *testing* data.<jupyter_code># "Import" torch.utils.data.random_split if if using torch <= v0.4.0
import torch.utils.data

if torch.utils.data.__dict__.get('random_split', None):
    random_split = torch.utils.data.random_split
else:
    class Subset(torch.utils.data.Dataset):
        """
        Subset of a dataset at specified indices.

        Arguments:
            dataset (Dataset): The whole Dataset
            indices (sequence): Indices in the whole set selected for subset
        """
        def __init__(self, dataset, indices):
            self.dataset = dataset
            self.indices = indices

        def __getitem__(self, idx):
            return self.dataset[self.indices[idx]]

        def __len__(self):
            return len(self.indices)

    def random_split(dataset, lengths):
        """
        Randomly split a dataset into non-overlapping new datasets of given lengths.

        Arguments:
            dataset (Dataset): Dataset to be split
            lengths (sequence): lengths of splits to be produced
        """
        if sum(lengths) != len(dataset):
            raise ValueError("Sum of input lengths does not equal the length of the input dataset!")

        indices = torch.randperm(sum(lengths))
        return [Subset(dataset, indices[offset - length:offset]) for offset, length in \
                zip(itertools.accumulate(lengths), lengths)]
import torch, itertools
from torch.utils.data import TensorDataset

# TODO: Allow dataset to be auto-converted from a list of lists
def dataset_split(dataset, train_frac):
    '''
    param dataset:    Dataset object to be split
    param train_frac: Ratio of train set to whole dataset
    
    Randomly split dataset into a dictionary with keys, based on these ratios:
        'train': train_frac
        'valid': (1-train_frac) / 2
        'test':  (1-train_frac) / 2
    '''
    assert train_frac >= 0 and train_frac <= 1, "Invalid training set fraction"
    
    length = len(dataset)
    
    # Use int to get the floor to favour allocation to the smaller valid and test sets    
    train_length = int(length * train_frac)
    valid_length = int((length - train_length) / 2)
    test_length  = length - train_length - valid_length
    
    dataset_list = random_split(dataset, (train_length, valid_length, test_length))
    dataset = {name: dset for name, dset in zip(('train', 'valid', 'test'), dataset_list)}
    return dataset
## Split data into training, validation, and test datasets

dataset = TensorDataset(torch.tensor(features, dtype=torch.int),
                        torch.tensor(encoded_labels, dtype=torch.uint8))

# Test
# dataset = TensorDataset(torch.Tensor([1, 2, 3]), torch.Tensor([11, 22, 33]))

split_frac = 0.8
dataset = dataset_split(dataset, split_frac) # Get dict of Subset objects
      
# Print out the lengths of the resultant datasets
print(f'len(dataset["train"] == {len(dataset["train"])}')
print(f'len(dataset["valid"] == {len(dataset["valid"])}')
print(f'len(dataset["test" ] == {len(dataset["test" ])}')
      
# describe(datasets["train"].dataset[0][0].shape) # == torch.Size([200]) # shape of first x value<jupyter_output>len(dataset["train"] == 20000
len(dataset["valid"] == 2500
len(dataset["test" ] == 2500<jupyter_text>**Check your work**With train, validation, and test fractions equal to 0.8, 0.1, 0.1, respectively, the final, feature data shapes should look like:```                    Feature Shapes:Train set: 		 (20000, 200) Validation set: 	(2500, 200) Test set: 		  (2500, 200)``` --- DataLoaders and BatchingAfter creating training, test, and validation data, we can create DataLoaders for this data by following two steps:1. Create a known format for accessing our data, using [TensorDataset](https://pytorch.org/docs/stable/data.html) which takes in an input set of data and a target set of data with the same first dimension, and creates a dataset.2. Create DataLoaders and batch our training, validation, and test Tensor datasets.```train_data = TensorDataset(torch.from_numpy(train_x), torch.from_numpy(train_y))train_loader = DataLoader(train_data, batch_size=batch_size)```This is an alternative to creating a generator function for batching our data into full batches.<jupyter_code>###
### This code was provided by the course.
### Its flaw is assuming that the complete set is randomly distributed
###

# from torch.utils.data import TensorDataset, DataLoader

# # create Tensor datasets
# train_data = TensorDataset(torch.from_numpy(train_x), torch.from_numpy(train_y))
# valid_data = TensorDataset(torch.from_numpy(val_x), torch.from_numpy(val_y))
# test_data = TensorDataset(torch.from_numpy(test_x), torch.from_numpy(test_y))
# Create dataloaders
from torch.utils.data import DataLoader

batch_size = 96

# Shuffle again to ensure that epochs provide different batches
train_loader = DataLoader(dataset['train'], shuffle=True, batch_size=batch_size)
valid_loader = DataLoader(dataset['train'], shuffle=True, batch_size=batch_size)
test_loader =  DataLoader(dataset['test' ], shuffle=True, batch_size=batch_size)
# obtain one batch of training data
dataiter = iter(train_loader)
sample_x, sample_y = dataiter.next()

print('Sample input size: ', sample_x.size()) # batch_size, seq_length
print('Sample input: \n', sample_x)
print()
print('Sample label size: ', sample_y.size()) # batch_size
print('Sample label: \n', sample_y)<jupyter_output>Sample input size:  torch.Size([96, 200])
Sample input: 
 tensor([[     0,      0,      0,  ...,    789,      8,     45],
        [    11,      6,   4037,  ...,    692,    200,      8],
        [  2018,     10,     89,  ...,     26,    179,     50],
        ...,
        [    11,      6,      3,  ...,     12,     25,    552],
        [     0,      0,      0,  ...,      8,     55,     74],
        [  8139,     22,   1193,  ...,    144,      5,   3476]], dtype=torch.int32)

Sample label size:  torch.Size([96])
Sample label: 
 tensor([ 1,  1,  1,  1,  0,  1,  1,  1,  0,  0,  0,  0,  0,  0,
         0,  1,  1,  1,  1,  0,  1,  1,  0,  0,  1,  1,  0,  1,
         0,  0,  1,  0,  1,  0,  1,  0,  0,  1,  0,  1,  1,  1,
         0,  1,  1,  0,  0,  0,  0,  1,  1,  1,  1,  1,  1,  1,
         0,  1,  0,  0,  0,  1,  1,  1,  0,  1,  1,  1,  0,  0,
         0,  1,  0,  1,  1,  1,  0,  0,  0,  0,  0,  0,  0,  1,
         1,  1,  0,  0,  1,  0,  0,  0,  1,  0,  1,  1], dtype=torch.uint8)<jupyter_text>--- Sentiment Network with PyTorchBelow is where you'll define the network.The layers are as follows:1. An [embedding layer](https://pytorch.org/docs/stable/nn.htmlembedding) that converts our word tokens (integers) into embeddings of a specific size.2. An [LSTM layer](https://pytorch.org/docs/stable/nn.htmllstm) defined by a hidden_state size and number of layers3. A fully-connected output layer that maps the LSTM layer outputs to a desired output_size4. A sigmoid activation layer which turns all outputs into a value 0-1; return **only the last sigmoid output** as the output of this network. The Embedding LayerWe need to add an [embedding layer](https://pytorch.org/docs/stable/nn.htmlembedding) because there are 74000+ words in our vocabulary. It is massively inefficient to one-hot encode that many classes. So, instead of one-hot encoding, we can have an embedding layer and use that layer as a lookup table. You could train an embedding layer using Word2Vec, then load it here. But, it's fine to just make a new layer, using it for only dimensionality reduction, and let the network learn the weights. The LSTM Layer(s)We'll create an [LSTM](https://pytorch.org/docs/stable/nn.htmllstm) to use in our recurrent network, which takes in an input_size, a hidden_dim, a number of layers, a dropout probability (for dropout between multiple layers), and a batch_first parameter.Most of the time, you're network will have better performance with more layers; between 2-3. Adding more layers allows the network to learn really complex relationships. > **Exercise:** Complete the `__init__`, `forward`, and `init_hidden` functions for the SentimentRNN model class.Note: `init_hidden` should initialize the hidden and cell state of an lstm layer to all zeros, and move those state to GPU, if available.<jupyter_code># First checking if GPU is available
train_on_gpu=torch.cuda.is_available()

if(train_on_gpu):
    print('Training on GPU.')
else:
    print('No GPU available, training on CPU.')
    
device = 'cuda' if torch.cuda.is_available() else 'cpu'
import torch.nn as nn
import torch.nn.functional as F

class SentimentRNN(nn.Module):
    """
    The RNN model that will be used to perform Sentiment analysis.
    """

    def __init__(self, vocab_size, output_size, embedding_dim, hidden_dim, n_layers, \
                 rnn_drop_prob=0.5, fc_drop_prob=0.1):
        """
        Initialize the model by setting up the layers.
        """
        super(SentimentRNN, self).__init__()

        self.output_size = output_size
        self.n_layers = n_layers
        self.hidden_dim = hidden_dim
        
        # https://pytorch.org/docs/stable/nn.html#embedding
        self.embedding = nn.Embedding(vocab_size, embedding_dim)
        
        # https://pytorch.org/docs/stable/nn.html#torch.nn.GRU
        # batch_first=True gives input and output of shape (batch, seq, feature)
        
        # https://pytorch.org/docs/stable/nn.html#lstm
        self.lstm = nn.LSTM(input_size=embedding_dim, hidden_size=hidden_dim, num_layers=n_layers,
                          dropout=0 if n_layers == 1 else rnn_drop_prob, batch_first=True)
        
        self.fc_dropout = nn.Dropout(p=fc_drop_prob)
        self.activation = nn.LeakyReLU()
        self.conv1d = nn.MaxPool1d(2)
#         self.fc = nn.Linear(hidden_dim, output_size)
        
        self.fc_hidden = 20  # Number of neurons in layer between conv1d and output
        self.fc1 = nn.Linear(hidden_dim//2, self.fc_hidden)
        self.fc2 = nn.Linear(self.fc_hidden, output_size)


    def forward(self, x, hidden):
        """
        Perform a forward pass of our model on some input and hidden state.
        """
        
        seq_len = x.shape[1]
        
#         describe(x.shape)
        e = self.embedding(x)
#         describe(e.shape)
        
        lstm_out, hidden = self.lstm(e, hidden)

#         describe(lstm_out.shape)
        
        # Select only the final outputs from dimension 1 (sequences/columns)
        # https://pytorch.org/docs/stable/tensors.html?highlight=select#torch.Tensor.select
        # Slices the self tensor along the selected dimension at the given index.
        # This function returns a tensor with the given dimension removed.
        seq_finals = lstm_out.select(1, seq_len-1).contiguous()

        # Hmm, I'm not too sure about dropout 
        x = self.fc_dropout(seq_finals)

#         describe(dropped.shape)
        x.unsqueeze_(0)
#         describe(self.conv1d(dropped).shape)
    
        
#         describe(dropped.shape)

        x = self.conv1d(x)
        x = self.fc1(x.squeeze(0))
        x = self.fc_dropout(x)
        x = self.activation(x)
        
        x = self.fc2(x)
        
        # No sigmoid here for better numerical stability
#         return F.sigmoid(fc_out), hidden
        return x, hidden

    def init_hidden(self, batch_size):
        ''' Initializes hidden state '''
        # Create two new tensors with sizes n_layers x batch_size x hidden_dim,
        # initialized to zero, for hidden state and cell state of LSTM

#         describe(batch_size)
#         describe(self.hidden_dim)
        return [torch.zeros(self.n_layers, batch_size, self.hidden_dim).to(device) for _ in range(2)]<jupyter_output><empty_output><jupyter_text>Instantiate the networkHere, we'll instantiate the network. First up, defining the hyperparameters.* `vocab_size`: Size of our vocabulary or the range of values for our input, word tokens.* `output_size`: Size of our desired output; the number of class scores we want to output (pos/neg).* `embedding_dim`: Number of columns in the embedding lookup table; size of our embeddings.* `hidden_dim`: Number of units in the hidden layers of our LSTM cells. Usually larger is better performance wise. Common values are 128, 256, 512, etc.* `n_layers`: Number of LSTM layers in the network. Typically between 1-3> **Exercise:** Define the model  hyperparameters.<jupyter_code># Using hyperparams based on:
# https://www.kaggle.com/nilanml/imdb-review-deep-model-94-89-accuracy

# Instantiate the model w/ hyperparams
vocab_size = len(vocab_to_int) + 1  # Also include padding
output_size = 1
embedding_dim = 128
hidden_dim = 32
n_layers = 1
rnn_drop_prob = 0
fc_drop_prob = 0.05

net = SentimentRNN(vocab_size, output_size, embedding_dim, hidden_dim, n_layers, \
                   rnn_drop_prob, fc_drop_prob).to(device)
print(net)

# Tests
test_input = torch.randint(low=0, high=3+1, size=(batch_size, seq_length), dtype=torch.long, device=device) 
hidden = net.init_hidden(batch_size)
assert net(test_input, hidden)[0].shape == (batch_size, output_size), "Network output shape incorrect. " \
    f"Expected: {(batch_size, output_size)}, got: {net(test_input, hidden)[0].shape}"<jupyter_output>SentimentRNN(
  (embedding): Embedding(74073, 128)
  (lstm): LSTM(128, 32, batch_first=True)
  (fc_dropout): Dropout(p=0.05)
  (activation): LeakyReLU(negative_slope=0.01)
  (conv1d): MaxPool1d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (fc1): Linear(in_features=16, out_features=20, bias=True)
  (fc2): Linear(in_features=20, out_features=1, bias=True)
)<jupyter_text>--- TrainingBelow is the typical training code. If you want to do this yourself, feel free to delete all this code and implement it yourself. You can also add code to save a model by name.>We'll also be using a new kind of cross entropy loss, which is designed to work with a single Sigmoid output. [BCELoss](https://pytorch.org/docs/stable/nn.htmlbceloss), or **Binary Cross Entropy Loss**, applies cross entropy loss to a single value between 0 and 1.We also have some data and training hyparameters:* `lr`: Learning rate for our optimizer.* `epochs`: Number of times to iterate through the training dataset.* `clip`: The maximum gradient value to clip at (to prevent exploding gradients).<jupyter_code># loss and optimization functions
lr=0.001

# criterion = nn.BCELoss()  # when output has sigmoid applied
criterion = nn.BCEWithLogitsLoss()
optimizer = torch.optim.Adam(net.parameters(), lr=lr)
model_save_path = 'sentiment_rnn.pth'
optim_save_path = 'optimizer.pth'
# Whether to train or load previously saved model
train = False

if not train:
    if torch.cuda.is_available():
        map_location=lambda storage, loc: storage.cuda()
    else:
        map_location='cpu'

    net.load_state_dict(torch.load(model_save_path, map_location=map_location))
    optimizer.load_state_dict(torch.load(optim_save_path, map_location=map_location))
if train:
    epochs = 10
    clip = 5 # max gradient for clipping

    counter = 0
    print_every = 100

    net.train()
    # train for some number of epochs
    for e in range(epochs):
        # batch loop
        for inputs, labels in train_loader:
            counter += 1

            inputs, labels = inputs.to(device, dtype=torch.long), labels.to(device)

    #         # Creating new variables for the hidden state, otherwise
    #         # we'd backprop through the entire training history
    #         h = tuple([each.data for each in h])

            # zero accumulated gradients
            net.zero_grad()

    #         s = inputs.shape
    #         describe(s)
    #         inputs = torch.randint(low=0, high=3+1, size=s, dtype=torch.long, device=device) 

            # initialize hidden state
            h = net.init_hidden(inputs.shape[0])

            # get the output from the model
            output, h = net(inputs, h)

            # calculate the loss and perform backprop
            loss = criterion(output.squeeze(), labels.float())
            loss.backward()
            # `clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.
            nn.utils.clip_grad_norm_(net.parameters(), clip)
            optimizer.step()

            # loss stats
            if counter % print_every == 0:
                # Get validation loss
    #             val_h = net.init_hidden(batch_size)

    #             describe([val_h[0].shape, val_h[1].shape])

                val_losses = []
                net.eval()
                for inputs, labels in valid_loader:
                    inputs, labels = inputs.to(device, dtype=torch.long), labels.to(device)

    #                 # Creating new variables for the hidden state, otherwise
    #                 # we'd backprop through the entire training history
    #                 val_h = tuple([each.data for each in val_h])

    #                 val_h = net.init_hidden(inputs.shape[0])

    #                 describe([val_h[0].shape, val_h[1].shape])

                    h = net.init_hidden(inputs.shape[0])

                    output, val_h = net(inputs, h)
                    val_loss = criterion(output.squeeze(), labels.float())

                    val_losses.append(val_loss.item())

                net.train()
                print("Epoch: {}/{}...".format(e+1, epochs),
                      "Step: {}...".format(counter),
                      "Loss: {:.6f}...".format(loss.item()),
                      "Val Loss: {:.6f}".format(np.mean(val_losses)))
if train:
    torch.save(net.state_dict(), model_save_path)
    torch.save(optimizer.state_dict(), optim_save_path)<jupyter_output><empty_output><jupyter_text>--- TestingThere are a few ways to test your network.* **Test data performance:** First, we'll see how our trained model performs on all of our defined test_data, above. We'll calculate the average loss and accuracy over the test data.* **Inference on user-generated data:** Second, we'll see if we can input just one example review at a time (without a label), and see what the trained model predicts. Looking at new, user input data like this, and predicting an output label, is called **inference**.<jupyter_code># Get test data loss and accuracy

test_losses = [] # track loss
num_correct = 0
num_examples = 0

net.eval()
# iterate over test data
for inputs, labels in valid_loader:
    inputs, labels = inputs.to(device, dtype=torch.long), labels.to(device)
    
    batch_size = labels.size(0)
    
    # init hidden state
    h = net.init_hidden(batch_size)

    # get predicted outputs
    output, h = net(inputs, h)
    
    output.squeeze_()
    
    # calculate loss
    test_loss = criterion(output, labels.float())
    test_losses.append(test_loss.item())
    
    output.sigmoid_()
    
#     describe(output)

    # Round output probabilities to predicted class (0 or 1)
    pred = torch.round(output).byte()
    
#     describe(pred)

    # Compare predictions to true label
    correct_tensor = pred.eq(labels)
    
    
    
# #     correct_tensor = pred.eq(labels.float().view_as(pred))
#     correct = np.squeeze(correct_tensor.numpy()) if not train_on_gpu else np.squeeze(correct_tensor.cpu().numpy())
#     num_correct += np.sum(correct)
    
#     num=28
    
#     describe(sum(correct_tensor).item())

#     describe(sum(correct_tensor).item() / labels.shape[0])
    
    num_examples += batch_size
    num_correct += sum(correct_tensor).item()
    
    
#     print('labels:')
#     print(labels[0:num])
#     print()
#     print('pred:')
#     print(pred[0:num])
#     print('---------------------------------------------------')
    
#     describe(labels.shape)
    
#     correct_tensor = pred.eq(labels.float()) # .view_as(pred))
#     correct = np.squeeze(correct_tensor.numpy()) if not train_on_gpu else np.squeeze(correct_tensor.cpu().numpy())
#     num_correct += correct.sum()


# -- stats! -- ##
# avg test loss
print("Test loss: {:.3f}".format(np.mean(test_losses)))  # TODO: account for heterogenous batch sizes

# accuracy over all test data
test_acc = num_correct / num_examples
print("Test accuracy: {:.3f}".format(test_acc))<jupyter_output>Test loss: 0.165
Test accuracy: 0.957<jupyter_text>Inference on a test reviewYou can change this test_review to any text that you want. Read it and think: is it pos or neg? Then see if your model predicts correctly!    > **Exercise:** Write a `predict` function that takes in a trained net, a plain text_review, and a sequence length, and prints out a custom statement for a positive or negative review!* You can use any functions that you've already defined or define any helper functions you want to complete `predict`, but it should just take in a trained net, a text review, and a sequence length.<jupyter_code>def predict(net, test_review, seq_length=seq_length):
    ''' Prints out whether a give review is predicted to be 
        positive or negative in sentiment, using a trained model.
        
        params:
        net - A trained net 
        test_review - a review made of normal text and punctuation
        sequence_length - the padded length of a review
        '''
    review_ints = reviews_to_ints(clean_data(test_review))
    padded = torch.tensor(pad_features(review_ints, seq_length=seq_length), device=device)
    y = net(padded, net.init_hidden(batch_size=1))[0]
    y = F.sigmoid(y).item()
    review = 'negative' if y < 0.5 else 'positive'
    print(f'The review is {review} (sigmoid: {y:.3f})')
# negative test review
test_review_neg = 'The worst movie I have seen; acting was terrible and I want my money back. This movie had bad acting and the dialogue was slow.'
# test_review_neg = 'I want my money back.'
# positive test review
test_review_pos = 'This movie had convincing acting and the dialogue was believable.'
test_review_pos = 'Definitely worth the money.'
# test_review_pos = 'This movie was better than good'
# call function
# try negative and positive reviews!

predict(net, test_review_neg, seq_length)
predict(net, test_review_pos, seq_length)<jupyter_output>The review is negative (sigmoid: 0.002)
The review is positive (sigmoid: 0.774)
{'fraction_non_alphanumeric': 0.09865175061344028, 'fraction_numerical': 0.023086462098625367, 'mean_word_length': 3.140937397574566, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 74, 'https://': 12, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 36, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '422647', 'license': 'MIT', 'n_tokens_mistral': 11809, 'n_tokens_neox': 11117, 'n_words': 4285, 'path': 'sentiment-rnn/Sentiment_RNN_Exercise.ipynb', 'repo_name': 'HaleTom/pytorch-udacity-intro', 'size': '37901'}