<jupyter_start><jupyter_text>Recurrent Neural Network trained on AeneisThis project is based on <NAME>'s ["Recurrent Neural Network - The Math of Intelligence (Week 5)"](https://www.youtube.com/watch?v=BwmddtPFWtA) Video. [This](https://github.com/llSourcell/recurrent_neural_network/blob/master/RNN.ipynb) is his Jupyter Notebook which is based on <NAME>'s [Code](https://gist.github.com/karpathy/d4dee566867f8291f086) The code contains 4 parts* Load the trainning data  * encode char into vectors* Define the Recurrent Network* Define a loss function  * Forward pass  * Loss  * Backward pass* Define a function to visualize samples from the model* Train the network  * Feed the network  * Calculate gradient and update the model parameters  * Output a text to see the progress of the training   Loading the dataIn this case I am using Virgil's Aeneis as training data, which I got from [gutenberg.org](http://www.gutenberg.org/ebooks/227).<jupyter_code>data = open('aeneis.txt').read()

chars = list(set(data))
data_size, vocab_size = len(data), len(chars)
print(f'Virgil\'s Aeneis is {data_size} characters long including {len(chars)} unique ones.')<jupyter_output>Virgil's Aeneis is 484506 characters long including 82 unique ones.<jupyter_text>encoding/decoding First let's create two dictionaries<jupyter_code>char_to_ix = { ch:i for i,ch in enumerate(chars)}
ix_to_char = { i:ch for i, ch in enumerate(chars)}
print(char_to_ix)
print(ix_to_char)<jupyter_output>{'1': 0, 't': 1, 'K': 2, ':': 3, 'e': 4, 'H': 5, '5': 6, 'N': 7, '/': 8, 'O': 9, 'w': 10, ';': 11, 'g': 12, 'i': 13, 'V': 14, 'z': 15, 'S': 16, 'y': 17, 'G': 18, ',': 19, '?': 20, '-': 21, 'x': 22, 'v': 23, 'R': 24, '[': 25, '8': 26, '9': 27, '3': 28, 'r': 29, 's': 30, '(': 31, '@': 32, '!': 33, 'J': 34, 'd': 35, ')': 36, 'k': 37, ']': 38, '*': 39, 'q': 40, 'A': 41, '2': 42, 'Y': 43, 'f': 44, 'j': 45, 'B': 46, 'l': 47, 'o': 48, 'b': 49, '"': 50, 'P': 51, 'Z': 52, '$': 53, 'a': 54, 'n': 55, 'm': 56, '.': 57, 'D': 58, '6': 59, 'L': 60, '0': 61, 'U': 62, 'W': 63, 'T': 64, 'I': 65, 'M': 66, '%': 67, '7': 68, '4': 69, 'c': 70, 'E': 71, "'": 72, 'X': 73, '\n': 74, 'Q': 75, 'C': 76, ' ': 77, 'p': 78, 'u': 79, 'F': 80, 'h': 81}
{0: '1', 1: 't', 2: 'K', 3: ':', 4: 'e', 5: 'H', 6: '5', 7: 'N', 8: '/', 9: 'O', 10: 'w', 11: ';', 12: 'g', 13: 'i', 14: 'V', 15: 'z', 16: 'S', 17: 'y', 18: 'G', 19: ',', 20: '?', 21: '-', 22: 'x', 23: 'v', 24: 'R', 25: '[', 26: '8', 27: '9', 28: '3', 29: 'r', 30: 's', [...]<jupyter_text>Now, let's make a function to create vectors from characters<jupyter_code>import numpy as np
def vec_from_char(char):
    # two-dimensional!
    vector = np.zeros((vocab_size, 1))
    vector[char_to_ix[char]][0] = 1
    return vector

print(vec_from_char('a').ravel())
len(vec_from_char('a').shape)<jupyter_output>[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]<jupyter_text>Now, let's make a function to decode those vectors<jupyter_code>import numpy as np
def char_from_vec(vec):
    # one-dimensional!
    if len(vec.shape) == 2:
        vec = vec.ravel()
    index = int(np.where(vec == 1)[0])
    return ix_to_char[index]

char_from_vec(vec_from_char('a'))<jupyter_output><empty_output><jupyter_text>Definition of the Network<jupyter_code># hyper parameters

hidden_size = 100
seq_length = 25
learning_rate = 1e-1
# model parameters

# weights from the standard normal distribution
# multiply by 0.01 to scale the values for character level RNNs

Wxh = np.random.randn(hidden_size, vocab_size) * 0.01 # input to hidden
Whh = np.random.randn(hidden_size, hidden_size) * 0.01 # input to hidden
Why = np.random.randn(vocab_size, hidden_size) * 0.01 # input to hidden

# biases are initialised as zero

bh = np.zeros((hidden_size, 1))
by = np.zeros((vocab_size, 1))<jupyter_output><empty_output><jupyter_text>Loss function forward**pseudo code**```pythonhidden = tanh(weights_input_hidden * input + weights_hidden_hidden * previous_hidden + biases_hidden)output = weights_hidden_output * hidden + biases_hiddennormalized_probabilities = e^(output) / sum(e^(output))```**implementation**```pythonhs[t] = np.tanh(np.dot(Wxh, xs[t]) + np.dot(Whh, hs[t-1]) + bh)  hidden stateys[t] = np.dot(Why, hs[t]) + by  unnormalized log probabilities for next charsps[t] = np.exp(ys[t]) / np.sum(np.exp(ys[t]))  probabilities for next chars``` lossthe loss of one prediction equals `-log(normalized_prediction_at_true_index)`, which comes from the [cross-entropy loss](https://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.htmlcross-entropy), but only checking the prediction of the true index (where y == 1)<jupyter_code>def lossF(inputs, targets, hprev):
    """
    inputs, targets are both lists of integers of the same length.
    hprev is Hx1 array of initial hidden state
    returns the loss, gradients on model parameters, and last hidden state
    """
    #store our inputs, hidden states, outputs, and probability values
    xs, hs, ys, ps, = {}, {}, {}, {} # Empty dicts
    # Each of these are going to be SEQ_LENGTH(Here 25) long dicts i.e. 1 vector per time(seq) step
    # xs will store 1 hot encoded input characters for each of 25 time steps (vocab_size, 25 times)
    # hs will store hidden state outputs for 25 time steps (100, 25 times)) plus a -1 indexed initial state
    # to calculate the hidden state at t = 0
    # ys will store targets i.e. expected outputs for 25 times (vocab_size, 25 times), unnormalized probabs
    # ps will take the ys and convert them to normalized probab for chars
    
    # We could have used lists BUT we need an entry with -1 to calc the 0th hidden layer
    # -1 as  a list index would wrap around to the final element
    
    # init with previous hidden state
    # Using "=" would create a reference, this creates a whole separate copy
    # We don't want hs[-1] to automatically change if hprev is changed
    hs[-1] = np.copy(hprev)
    # init loss as 0
    loss = 0
    # FORWARD
    for t in range(len(inputs)):
        xs[t] = vec_from_char(ix_to_char[inputs[t]])
        hs[t] = np.tanh(np.dot(Wxh, xs[t]) + np.dot(Whh, hs[t-1]) + bh) # hidden state
        ys[t] = np.dot(Why, hs[t]) + by # unnormalized log probabilities for next chars
        ps[t] = np.exp(ys[t]) / np.sum(np.exp(ys[t])) # normalized probabilities for next chars
        # separate variables for clarity
        true_index = targets[t]
        this_loss = -np.log(ps[t][true_index][0])
        loss += this_loss
        
    # backward pass: compute gradients going backwards
    #initalize vectors for gradient values for each set of weights
    dWxh, dWhh, dWhy = np.zeros_like(Wxh), np.zeros_like(Whh), np.zeros_like(Why)
    dbh, dby = np.zeros_like(bh), np.zeros_like(by)
    dhnext = np.zeros_like(hs[0])
    for t in reversed(range(len(inputs))):
        #output probabilities
        dy = np.copy(ps[t])
        #derive our first gradient
        dy[targets[t]] -= 1 # backprop into y
        #compute output gradient -  output times hidden states transpose
        #When we apply the transpose weight matrix,
        #we can think intuitively of this as moving the error backward
        #through the network, giving us some sort of measure of the error
        #at the output of the lth layer.
        #output gradient
        dWhy += np.dot(dy, hs[t].T)
        #derivative of output bias
        dby += dy
        #backpropagate!
        dh = np.dot(Why.T, dy) + dhnext # backprop into h
        dhraw = (1 - hs[t] * hs[t]) * dh # backprop through tanh nonlinearity
        dbh += dhraw #derivative of hidden bias
        dWxh += np.dot(dhraw, xs[t].T) #derivative of input to hidden layer weight
        dWhh += np.dot(dhraw, hs[t-1].T) #derivative of hidden layer to hidden layer weight
        dhnext = np.dot(Whh.T, dhraw)
    for dparam in [dWxh, dWhh, dWhy, dbh, dby]:
        np.clip(dparam, -5, 5, out=dparam) # clip to mitigate exploding gradients
    return loss, dWxh, dWhh, dWhy, dbh, dby, hs[len(inputs)-1]<jupyter_output><empty_output><jupyter_text>Sample Function<jupyter_code>from matplotlib import pyplot as plt
import seaborn as sns
sns.set()
def sample(h, seed_ix, n, plot=False):
    """ 
    sample a sequence of integers from the model 
    h is memory state, seed_ix is seed letter for first time step
    """
    x = np.zeros((vocab_size, 1))
    x[seed_ix] = 1
    ixes = []
    for t in range(n):
        # feed forward
        h = np.tanh(np.dot(Wxh, x) + np.dot(Whh, h) + bh)
        y = np.dot(Why, h) + by
        p = np.exp(y) / np.sum(np.exp(y))
        # choose proportional to probabilty
        ix = np.random.choice(range(vocab_size), p=p.ravel())
        # set input to chosen character
        x = np.zeros((vocab_size, 1))
        x[ix] = 1
        ixes.append(ix)
    if plot:
        last_p = p
        plt.plot(p.ravel())
        plt.show()
        
    txt = ''.join(ix_to_char[ix] for ix in ixes)
    print(f'----\n{txt}\n----')
    return ixes

prediction = sample(np.zeros((hidden_size, 1)), char_to_ix['a'], 200, plot=True)<jupyter_output><empty_output><jupyter_text>TrainingWe're using [Adagrad](https://ml-cheatsheet.readthedocs.io/en/latest/optimizers.html?highlight=adagradadagrad)<jupyter_code>n, p = 0, 0 # n is training step (of a 25 sequence), p is pointer within data
mWxh, mWhh, mWhy = np.zeros_like(Wxh), np.zeros_like(Whh), np.zeros_like(Why)
mbh, mby = np.zeros_like(bh), np.zeros_like(by) # memory variables for Adagrad
smooth_loss = -np.log(1.0/vocab_size)*seq_length # loss at iteration 0
while n <= 1000*100:
    # prepare inputs (we're sweeping from left to right in steps seq_length long)
    # check "How to feed the loss function to see how this part works
    if p+seq_length+1 >= len(data) or n == 0:
        hprev = np.zeros((hidden_size,1)) # reset RNN memory
        p = 0 # go from start of data
    inputs = [char_to_ix[ch] for ch in data[p:p+seq_length]]
    targets = [char_to_ix[ch] for ch in data[p+1:p+seq_length+1]]

    # forward seq_length characters through the net and fetch gradient
    loss, dWxh, dWhh, dWhy, dbh, dby, hprev = lossF(inputs, targets, hprev)
    smooth_loss = smooth_loss * 0.999 + loss * 0.001

    # sample from the model now and then
    if n % 1000 == 0:
        print(f'iter {n}, loss: {smooth_loss:.2f}') # print progress
        sample(hprev, inputs[0], 200, plot=False)

    # perform parameter update with Adagrad
    for param, dparam, mem in zip([Wxh, Whh, Why, bh, by],
                                  [dWxh, dWhh, dWhy, dbh, dby],
                                  [mWxh, mWhh, mWhy, mbh, mby]):
        mem += dparam * dparam # increases memory variable, which decreases learning rate
        param += -learning_rate * dparam / np.sqrt(mem + 1e-8) # adagrad update

    p += seq_length # move data pointer
    n += 1 # iteration counter<jupyter_output>iter 0, loss: 110.17
----
*wZ GICiLxmkxpq/!f(adw*8qY59s :OGCY$82QFuAXGApbck
3dM]p$sRd$RZ,r"c:,DYfN-f!Zy-.G1F%dY,"DYOjKKr;O2*DPo55uKR];ZNj';2N]Xo]]!.G
m!aok?w4YrU/Uf7?T;IwGlf%r]]4YwAU.qx!CwdBe"
F@X04n(GKb$a$HJm6)b.bNU;z!9Ok(abF
----
iter 1000, loss: 85.57
----
te  ahas bepusqupioquas rafuebiiDromnis farduonesca li :. equenat pites
 Keomqus usHactridod, nTeranhhoa tenne. hroveriant ar,ce ,iramece venolenu. s rodora toraturihem psas mderunolnotves
 amassad re
----<jupyter_text>resultLet's create a function which displays our work more beautifully!<jupyter_code>def tell_a_story(beginning, n):
    """
    write a story of length n with the given beginning
    beginning is a string
    n is an integer
    """
    h = np.zeros((hidden_size, 1)) # start with hidden state zero
    # build hidden state from beginning
    for char in beginning[:-1]:
        # create input vector
        x = vec_from_char(char)
        h = np.tanh(np.dot(Wxh, x) + np.dot(Whh, h) + bh)
        # we don't need to calculate y and p as we are 
        # just building up the hidden state
        
    # print of function is story
    story = beginning
    x = vec_from_char(beginning[-1])
    for t in range(n):
        # predict
        h = np.tanh(np.dot(Wxh, x) + np.dot(Whh, h) + bh)
        y = np.dot(Why, h) + by
        p = np.exp(y) / np.sum(np.exp(y))
        # set prediction
        char = ix_to_char[np.random.choice(range(vocab_size), p=p.ravel())]
        story += char
        # set new x to this time's prediction
        x = vec_from_char(char)
        
    print(story)
tell_a_story('  ARMA virumque cano', 1000)<jupyter_output>ARMA virumque canorem, qui regea capis
  insustum Guris adveribus altero talis,, sissi dum comum
  ieserius, patrunt illue prnorus, quaeterit
  salius urat--' Allucerior ac tergent aeter et virere dacre fircet enos,
  modine prosaborsa anus urbe audaque et fecoliis admour pirmine masso, cumproro
  Tenis
  dito camerem nuverat, exat parrpit Gitetiens impiateres;
  ux, siseremque perditur haurupe
  pora saca replomediusque octes,
  et omis,
, primo mivi egileante cramis ic voce mini cocristam, suin et eb orbitersis, ersus aunti, ignare mutra sdindis:
  forroqiam tanum
  et comorrisque restixine Trulames, heves, fypfluat
  cridipe tani; aucama furgum fucidas
  excurngunt
  camporis et facum ritan inserricuis mallusque aerus ceriiscum
  erona quam stimus proea canu copus,
  quid mora hamnia peditur, seres mecuia cur coneri.
  He pitis et moentia Teulasque orbaque turium corredi, aorra donitere, soriabis
  nae et victurer; agerum
  dis semptes, peria subeltis Daens et eter.
  Malla caplum[...]
{'fraction_non_alphanumeric': 0.13982988708021704, 'fraction_numerical': 0.03556239917876521, 'mean_word_length': 3.2782308657465498, 'pattern_counts': {'":': 1, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 5, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '518810', 'license': 'MIT', 'n_tokens_mistral': 5604, 'n_tokens_neox': 5041, 'n_words': 1610, 'path': 'rnn-aeneis.ipynb', 'repo_name': 'MoriMotonari/rnn-aeneis', 'size': '13652'}