<jupyter_start><jupyter_text>Text Generation with RNNs credits to **<NAME>** (who was a teaching assistant together with me for the last class)<jupyter_code>import numpy as np
import torch
from torch import nn
import torch.nn.functional as F<jupyter_output><empty_output><jupyter_text>1 DatasetDefine the path of the file, you want to read and train the model on<jupyter_code>'''TODO: set the path of the file'''
# Moby Dick text file from: https://gist.github.com/StevenClontz/4445774

path_to_file = '/content/mobydick.txt'
text = open(path_to_file, encoding='utf-8').read()<jupyter_output><empty_output><jupyter_text>Inspect the datasetTake a look at the first 250 characters in text<jupyter_code>print(text[:250])


# The unique characters in the file
vocab = sorted(set(text))
print ('{} unique characters'.format(len(vocab)))<jupyter_output>88 unique characters<jupyter_text>2 Process the dataset for the learning taskThe task that we want our model to achieve is: given a character, or a sequence of characters, what is the most probable next character?To achieve this, we will input a sequence of characters to the model, and train the model to predict the output, that is, the following character at each time step. RNNs maintain an internal state that depends on previously seen elements, so information about all characters seen up until a given moment will be taken into account in generating the prediction. Vectorize the textBefore we begin training our RNN model, we'll need to create a numerical representation of our text-based dataset. To do this, we'll generate two lookup tables: one that maps characters to numbers, and a second that maps numbers back to characters. Recall that we just identified the unique characters present in the text.<jupyter_code># Creating a mapping from unique characters to indices
char2idx = {u:i for i, u in enumerate(vocab)}
text_as_int = np.array([char2idx[c] for c in text])

# Create a mapping from indices to characters
idx2char = np.array(vocab)<jupyter_output><empty_output><jupyter_text>This gives us an integer representation for each character. Observe that the unique characters (i.e., our vocabulary) in the text are mapped as indices from 0 to len(unique). Let's take a peek at this numerical representation of our dataset:<jupyter_code>print('{')
for char,_ in zip(char2idx, range(200)):
    print('  {:4s}: {:3d},'.format(repr(char), char2idx[char]))
print('  ...\n}')<jupyter_output>{
  '\n':   0,
  ' ' :   1,
  '!' :   2,
  '"' :   3,
  '$' :   4,
  '&' :   5,
  "'" :   6,
  '(' :   7,
  ')' :   8,
  '*' :   9,
  ',' :  10,
  '-' :  11,
  '.' :  12,
  '/' :  13,
  '0' :  14,
  '1' :  15,
  '2' :  16,
  '3' :  17,
  '4' :  18,
  '5' :  19,
  '6' :  20,
  '7' :  21,
  '8' :  22,
  '9' :  23,
  ':' :  24,
  ';' :  25,
  '<' :  26,
  '>' :  27,
  '?' :  28,
  'A' :  29,
  'B' :  30,
  'C' :  31,
  'D' :  32,
  'E' :  33,
  'F' :  34,
  'G' :  35,
  'H' :  36,
  'I' :  37,
  'J' :  38,
  'K' :  39,
  'L' :  40,
  'M' :  41,
  'N' :  42,
  'O' :  43,
  'P' :  44,
  'Q' :  45,
  'R' :  46,
  'S' :  47,
  'T' :  48,
  'U' :  49,
  'V' :  50,
  'W' :  51,
  'X' :  52,
  'Y' :  53,
  'Z' :  54,
  '\\':  55,
  '^' :  56,
  '_' :  57,
  'a' :  58,
  'b' :  59,
  'c' :  60,
  'd' :  61,
  'e' :  62,
  'f' :  63,
  'g' :  64,
  'h' :  65,
  'i' :  66,
  'j' :  67,
  'k' :  68,
  'l' :  69,
  'm' :  70,
  'n' :  71,
  'o' :  72,
  'p' :  73,
  'q' :  74,
  'r' :  75,
  's' :  7[...]<jupyter_text>We can also look at how the first part of the text is mapped to an integer representation:<jupyter_code>print ('{} ---- characters mapped to int ---- > {}'.format(repr(text[:13]), text_as_int[:13]))<jupyter_output>'\nMOBY-DICK \n\n' ---- characters mapped to int ---- > [ 0 41 43 30 53 11 32 37 31 39  1  0  0]<jupyter_text>Defining method to encode one hot labels<jupyter_code>def one_hot_encode(arr, n_labels):
    # Initialize the the encoded array
    one_hot = np.zeros((np.multiply(*arr.shape), n_labels), dtype=np.float32)

    # Fill the appropriate elements with ones
    one_hot[np.arange(one_hot.shape[0]), arr.flatten()] = 1.

    # Finally reshape it to get back to the original array
    one_hot = one_hot.reshape((*arr.shape, n_labels))

    return one_hot<jupyter_output><empty_output><jupyter_text>Defining method to make mini-batches for training<jupyter_code>def get_batches(arr, batch_size, seq_length):
    '''Create a generator that returns batches of size
       batch_size x seq_length from arr.

       Arguments
       ---------
       arr: Array you want to make batches from
       batch_size: Batch size, the number of sequences per batch
       seq_length: Number of encoded chars in a sequence
    '''

    batch_size_total = batch_size * seq_length
    # total number of batches we can make
    n_batches = len(arr) // batch_size_total

    # Keep only enough characters to make full batches
    arr = arr[:n_batches * batch_size_total]
    # Reshape into batch_size rows
    arr = arr.reshape((batch_size, -1))

    # iterate through the array, one sequence at a time
    for n in range(0, arr.shape[1], seq_length):
        # The features
        x = arr[:, n:n + seq_length]
        # The targets, shifted by one
        y = np.zeros_like(x)
        try:
            y[:, :-1], y[:, -1] = x[:, 1:], arr[:, n + seq_length]
        except IndexError:
            y[:, :-1], y[:, -1] = x[:, 1:], arr[:, 0]
        yield x, y<jupyter_output><empty_output><jupyter_text>3 The Recurrent Neural Network (RNN) model  Check if GPU is available<jupyter_code>train_on_gpu = torch.cuda.is_available()
print ('Training on GPU' if train_on_gpu else 'Training on CPU')<jupyter_output>Training on GPU<jupyter_text>Declaring the model<jupyter_code>class CharRNN(nn.Module):
    def __init__(self, vocab, n_hidden=256, n_layers=2,
                 drop_prob=0.5, lr=0.001):
        super().__init__()
        self.drop_prob = drop_prob
        self.n_layers = n_layers
        self.n_hidden = n_hidden
        self.lr = lr

        # creating character dictionaries
        self.vocab = vocab

        # define the LSTM
        self.lstm = nn.LSTM(len(self.vocab), n_hidden, n_layers,
                            dropout=drop_prob, batch_first=True)

        # define a dropout layer
        self.dropout = nn.Dropout(drop_prob)

        # define the final, fully-connected output layer
        self.fc = nn.Linear(n_hidden, len(self.vocab))

    def forward(self, x, hidden):
        ''' Forward pass through the network.
            These inputs are x, and the hidden/cell state `hidden`. '''

        # get the outputs and the new hidden state from the lstm
        r_output, hidden = self.lstm(x, hidden)

        # pass through a dropout layer
        out = self.dropout(r_output)

        # Stack up LSTM outputs using view
        out = out.contiguous().view(-1, self.n_hidden)

        # put x through the fully-connected layer
        out = self.fc(out)

        # return the final output and the hidden state
        return out, hidden

    def init_hidden(self, batch_size):
        ''' Initializes hidden state '''
        # Create two new tensors with sizes n_layers x batch_size x n_hidden,
        # initialized to zero, for hidden state and cell state of LSTM
        weight = next(self.parameters()).data

        if (train_on_gpu):
            hidden = (weight.new(self.n_layers, batch_size, self.n_hidden).zero_().cuda(),
                      weight.new(self.n_layers, batch_size, self.n_hidden).zero_().cuda())
        else:
            hidden = (weight.new(self.n_layers, batch_size, self.n_hidden).zero_(),
                      weight.new(self.n_layers, batch_size, self.n_hidden).zero_())

        return hidden<jupyter_output><empty_output><jupyter_text>Declaring the train method<jupyter_code>def train(model, data, epochs=10, batch_size=10, seq_length=50, lr=0.001, clip=5, val_frac=0.1, print_every=10):
    ''' Training a network

        Arguments
        ---------

        model: CharRNN network
        data: text data to train the network
        epochs: Number of epochs to train
        batch_size: Number of mini-sequences per mini-batch, aka batch size
        seq_length: Number of character steps per mini-batch
        lr: learning rate
        clip: gradient clipping
        val_frac: Fraction of data to hold out for validation
        print_every: Number of steps for printing training and validation loss

    '''
    model.train()

    opt = torch.optim.Adam(model.parameters(), lr=lr)
    criterion = nn.CrossEntropyLoss()

    # create training and validation data
    val_idx = int(len(data) * (1 - val_frac))
    data, val_data = data[:val_idx], data[val_idx:]

    if (train_on_gpu):
        model.cuda()

    counter = 0
    n_vocab = len(model.vocab)
    for e in range(epochs):
        # initialize hidden state
        h = model.init_hidden(batch_size)
        
        '''TODO: use the get_batches function to generate sequences of the desired size'''
        dataset = get_batches(data, batch_size, seq_length)

        for x, y in dataset:
            counter += 1

            # One-hot encode our data and make them Torch tensors
            x = one_hot_encode(x, n_vocab)
            inputs, targets = torch.from_numpy(x), torch.from_numpy(y)

            if (train_on_gpu):
                inputs, targets = inputs.cuda(), targets.cuda()

            # Creating new variables for the hidden state, otherwise
            # we'd backprop through the entire training history
            h = tuple([each.data for each in h])

            # zero accumulated gradients
            model.zero_grad()

            # get the output from the model
            output, h = model(inputs, h)

            # calculate the loss and perform backprop
            loss = criterion(output, targets.view(batch_size * seq_length).long())
            loss.backward()
            # `clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.
            nn.utils.clip_grad_norm_(model.parameters(), clip)
            opt.step()

            # loss stats
            if counter % print_every == 0:
                # Get validation loss
                val_h = model.init_hidden(batch_size)
                val_losses = []
                model.eval()
                for x, y in get_batches(val_data, batch_size, seq_length):
                    # One-hot encode our data and make them Torch tensors
                    x = one_hot_encode(x, n_vocab)
                    x, y = torch.from_numpy(x), torch.from_numpy(y)

                    # Creating new variables for the hidden state, otherwise
                    # we'd backprop through the entire training history
                    val_h = tuple([each.data for each in val_h])

                    inputs, targets = x, y
                    if (train_on_gpu):
                        inputs, targets = inputs.cuda(), targets.cuda()

                    output, val_h = model(inputs, val_h)
                    val_loss = criterion(output, targets.view(batch_size * seq_length).long())

                    val_losses.append(val_loss.item())

                model.train()  # reset to train mode after iterationg through validation data

                print("Epoch: {}/{}...".format(e + 1, epochs),
                      "Step: {}...".format(counter),
                      "Loss: {:.4f}...".format(loss.item()),
                      "Val Loss: {:.4f}".format(np.mean(val_losses)))
                print(sample(model, 1000, prime='M', top_k=10))<jupyter_output><empty_output><jupyter_text>Defining a method to generate the next character<jupyter_code>def predict(model, char, h=None, top_k=None):
    ''' Given a character, predict the next character.
        Returns the predicted character and the hidden state.
    '''

    # tensor inputs
    if char == 'و':
      print(char)
      return 0, h
    x = np.array([[char2idx[char]]])
    x = one_hot_encode(x, len(model.vocab))
    inputs = torch.from_numpy(x)

    if (train_on_gpu):
        inputs = inputs.cuda()

    # detach hidden state from history
    h = tuple([each.data for each in h])
    # get the output of the model
    out, h = model(inputs, h)

    # get the character probabilities
    p = F.softmax(out, dim=1).data
    if (train_on_gpu):
        p = p.cpu()  # move to cpu

    # get top characters
    if top_k is None:
        top_ch = np.arange(len(model.vocab))
    else:
        p, top_ch = p.topk(top_k)
        top_ch = top_ch.numpy().squeeze()

    # select the likely next character with some element of randomness
    p = p.numpy().squeeze()
    char = np.random.choice(top_ch, p=p / p.sum())

    # return the encoded value of the predicted char and the hidden state
    return idx2char[char], h<jupyter_output><empty_output><jupyter_text>Declaring a method to generate new text<jupyter_code>def sample(net, size, prime='The', top_k=None):
    if (train_on_gpu):
        net.cuda()
    else:
        net.cpu()

    net.eval()  # eval mode

    # First off, run through the prime characters
    chars = [ch for ch in prime]
    h = net.init_hidden(1)
    for ch in prime:
        char, h = predict(net, ch, h, top_k=top_k)

    chars.append(char)

    # Now pass in the previous character and get a new one
    for ii in range(size):
        char, h = predict(net, chars[-1], h, top_k=top_k)
        chars.append(char)

    net.train()
    return ''.join(chars)<jupyter_output><empty_output><jupyter_text>Generate new Text using the RNN model  Define and print the net<jupyter_code>''''TODO: Try changing the number of units in the network to see how it affects performance'''
n_hidden = 1024
n_layers = 2

model = CharRNN(vocab, n_hidden, n_layers)
print(model)<jupyter_output>CharRNN(
  (lstm): LSTM(88, 1024, num_layers=2, batch_first=True, dropout=0.5)
  (dropout): Dropout(p=0.5, inplace=False)
  (fc): Linear(in_features=1024, out_features=88, bias=True)
)<jupyter_text>Declaring the hyperparameters<jupyter_code>batch_size = 64
seq_length = 10
n_epochs = 10  # start smaller if you are just testing initial behavior<jupyter_output><empty_output><jupyter_text>train the model<jupyter_code>'''TODO'''
train(model, text_as_int, epochs=n_epochs, batch_size=batch_size, seq_length=seq_length, lr=0.001, print_every=500)<jupyter_output>Epoch: 1/10... Step: 500... Loss: 1.6626... Val Loss: 1.5590
Mink was so inridnest 
hunting to me ; he his pricies, but the sea of the shell cormer, the particulal sevoatian woll. 
W' ' The sumble of the fisher, and hus ofcemado arm on the ross, trunse from 
at march on our one present meaniar, whether the way, and 
seen him. I were has a farly of the whole,' said I was indougded wind, I see so it spope all horners ; hing or fret there had no blong. 

I Tertlidal fither bitce ; bet I an as a matire, whale ! 



CHAPTER VI. 

IL ! 

StHall a wasterouses, that at sea, and that where arm his owner, had shoulding seemed to hist the 
more crew. Starbuck, whose threasons shake there only idled by in origh-nore-ships all sort, and 
sillering wifd a maintre-corserance of 
dispas all our flongery but still,. 

BOOTOR. 

II these lass, ship's hast took throw this to mo,t tammucant, 
as wooding only 
all stleets white captain against a parmowing sinness 
as in the white whale went, when the fore [...]
{'fraction_non_alphanumeric': 0.11248124225223462, 'fraction_numerical': 0.020812944477066615, 'mean_word_length': 2.6538736591179974, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 63, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '588049', 'license': 'Apache-2.0', 'n_tokens_mistral': 5329, 'n_tokens_neox': 5003, 'n_words': 1604, 'path': 'Machine Learning/Sample solutions/_U12/RNN_Solution.ipynb', 'repo_name': 'qiaw99/WS2020-21', 'size': '15332'}