<jupyter_start><jupyter_text>Writing Shakespeare with RNNsBy <NAME>In this project we're going to train a character level RNN on the collected works of Shakespeare, and use it to write some Shakespeare of our own. The Shakespeare corpus used for training was made available by [Project Gutenberg](https://www.gutenberg.org/files/100/100-0.txt).  Contents1. Model Architecture2. Project Setup    * 2.1 Import Libraries    * 2.2 Data Overview3. Model Training4. Model Parameter Tuning  1. Model ArchitectureThis project will use a recurrent neural network (RNN) to build a character model. Lets break down what these mean. Character ModelThis model will try to predict the next single character in a string of characters. For example, if the last sequence of characters the RNN has seen are "tomorro", it should predict "w" as the next character.This is different from a word level model that tries to predict whole words with every step. Recurrent Neural NetworkMost deep learning architectures have a series of layers that process an input into an output. Each layer has a set of weights that perform some functional operation to convert inputs to outputs. RNNs, rather than having a deep stack of layers, have a single hidden layer that is updated every time an input comes in. This allows the activations of a previous input to affect the activations of the next input. This allows RNNs to learn complex relationships along a series of inputs.This has made RNNs very effective architectures for NLP problems, where understanding a word in a sentence requires understanding what came before it.For example, lets say the RNN is processing the word "thanks". Since this is a character level model, each character is fed into the network one at a time.First the network processes "t", and updates the hidden state accordingly.Then the network processes "h", using the activations generated by the previous letter, "t", to process "h" and generate a new set of activations.Then the network processes "a", using the activations generated by "h" and "t". And so on.In Pytorch, the standard RNN module processes each input into a new set of activations using the following equation:$h_t = tanh(w_{ih} x_t + b_{ih}  +  w_{hh} h_{(t-1)} + b_{hh})$The network starts with an existing hidden state $h_{(t-1)}$. When a new input $x_t$ enters the network, it is processed by a linear layer with weights $w_{ih}$ and bias $b_{ih}$. The hidden state $h_{(t-1)}$ is also processed by a linear layer with weights $w_{hh}$ and bias $b_{hh}$. The outputs of these two linear layers are summed together and processed by the $tanh$ activation function to create the new hidden state $h_t$.For this project, we'll use a more advanced version of an RNN called an LSTM. The key improvement of LSTMs over regular RNNs is LSTMs introduce a set of gates that control how much information from the old hidden state and the current input move into the new hidden state. The parameters that control these gates are learned by the network. This allows LSTMs to learn and understand information over longer periods.The equations governing LSTMs:$\begin{array}{ll}i_t = \sigma(W_{ii} x_t + b_{ii} + W_{hi} h_{(t-1)} + b_{hi}) \\f_t = \sigma(W_{if} x_t + b_{if} + W_{hf} h_{(t-1)} + b_{hf}) \\g_t = \tanh(W_{ig} x_t + b_{ig} + W_{hg} h_{(t-1)} + b_{hg}) \\o_t = \sigma(W_{io} x_t + b_{io} + W_{ho} h_{(t-1)} + b_{ho}) \\c_t = f_t c_{(t-1)} + i_t g_t \\h_t = o_t \tanh(c_t)\end{array}$$i_{t}$ is the input gate. This controls how much of the new input is taken into the cell state.$f_{t}$ is the forget gate. This gate controls how much of the old cell state passes on to the new cell state.$g_{t}$ is the cell gate, which is the same equation we used to define our new hidden state in the standard RNN.$o_{t}$ is the output gate which controls how much of the cell state passes on to the hidden state.$c_{t}$ is the current cell state. You will notice $c_{t}$ is composed of the old cell state, the current forget gate, the current input gate, and the current cell gate. This shows how the various gates are used to filter how much of the input and previous cell state pass on to the new hidden state.$h_{t}$ is the new hidden state. It is composed of the current cell state activated by $tanh$ and filtered by the output gate.Visually:[](attachment:LSTM.png)When the optimizer updates the weights by backpropagation, the loss is propagated through all the hidden activations, essentially "unrolling" the hidden state into each set of activations created over the course of generating the final activations.Visually:[](attachment:RNN.png)[Image source](https://colah.github.io/posts/2015-08-Understanding-LSTMs/)This leads to one of the most important parameters in training RNNs: backprop through time (bptt). The bptt value sets how many hidden activations are saved and propagated through during backpropagation. Bptt is a double edged sword - on the one hand, a larger bptt value allows the network to train over longer sequences. This can allow the network to understand language over a longer sequence length. On the other, the larger the bptt value, the deeper the network becomes and the harder it is to train.   2. Project Setup 2.1 Load LibrariesThis project uses FastAI, Pytorch, and their associated dependencies.<jupyter_code>%reload_ext autoreload
%autoreload 2
%matplotlib inline

from fastai.io import *
from fastai.conv_learner import *
from fastai.column_data import *
from torchtext import vocab, data
from fastai.nlp import *
from fastai.lm_rnn import *
#Set GPU
torch.cuda.set_device(1)
torch.cuda.is_available()
torch.backends.cudnn.enabled<jupyter_output><empty_output><jupyter_text>2.2 Data OverviewOur corpus is a single text file containing the complete works of Shakespeare<jupyter_code>PATH='data/Shakespeare/'
text = open(f'{PATH}Shakespeare.txt', encoding='utf-8').read()
print('corpus length:', len(text))
print(text[4000000:4001000])<jupyter_output>or love
    As for another secret close intent
    By marrying her which I must reach unto.
    But yet I run before my horse to market.
    Clarence still breathes; Edward still lives and reigns;
    When they are gone, then must I count my gains.         Exit

SCENE 2.

London. Another street

Enter corpse of KING HENRY THE SIXTH, with halberds to guard it;
LADY ANNE being the mourner, attended by TRESSEL and BERKELEY

  ANNE. Set down, set down your honourable load-
    If honour may be shrouded in a hearse;
    Whilst I awhile obsequiously lament
    Th' untimely fall of virtuous Lancaster.
    Poor key-cold figure of a holy king!
    Pale ashes of the house of Lancaster!
    Thou bloodless remnant of that royal blood!
    Be it lawful that I invocate thy ghost
    To hear the lamentations of poor Anne,
    Wife to thy Edward, to thy slaughtered son,
    Stabb'd by the self-same hand that made these wounds.
    Lo, in these windows that let forth thy life
    I pour the helpless ba[...]<jupyter_text>Looks like Shakespeare.Since this is a character model, the model vocabulary will be every character type that appears in the corpus. It looks like we have 94 characters in the corpus. We will also add a "\0" value for padding.<jupyter_code>chars = sorted(list(set(text)))
vocab_size = len(chars)+1
print('total chars:', vocab_size)
chars.insert(0, "\0")

''.join(chars[1:-6])<jupyter_output><empty_output><jupyter_text>To create a train and validation set, we don't want to randomly shuffle or select data the way we would with other types of data. This is because we want to preserve the text/sentence structure, because that's exactly what we want the model to learn!A simple way to do this is to put the last 20% of the text into the validation set, and train on the first 80%.<jupyter_code>PATH='data/Shakespeare/'

TRN_PATH = 'trn/'
VAL_PATH = 'val/'
TRN = f'{PATH}{TRN_PATH}'
VAL = f'{PATH}{VAL_PATH}'

os.listdir(f'{PATH}')
with open('data/Shakespeare/Shakespeare.txt', encoding='utf-8') as f:
    text = f.readlines()
    text_line_length = len(text)
    trn_index = int(text_line_length*.8)
    trn = text[:trn_index]
    tst = text[trn_index:]
    with open('data/Shakespeare/trn/trn.txt','w', encoding='utf-8') as f2:
        f2.writelines(trn)
    with open('data/Shakespeare/val/val.txt','w', encoding='utf-8') as f3:
        f3.writelines(tst)<jupyter_output><empty_output><jupyter_text>3. Model TrainingThis is where we define the data loader for the model. TEXT is a torchtext field object. This contains the tokenization fuction used to process the text for the model. Since this is a character level model, the tokenizer is just the list function (ie 'abcd' --> ['a', 'b', 'c', 'd'])bs is the batch size and bptt is the backprop through time value. These two values decide how the text corpus is split up and fed into the model. Our batch size is 64, so the corpus is split into 64 equal sized sections. Bptt is 16. This is the depth of text taken in each minibatch from the corpus after it has been split by the batch size. This means the batch fed to the GPU will be 16x64, with each column representing 16 characters occurring in sequence.n_fac sets the size of the embedding matrix for each character in the vocabulary. n_hidden sets the size of the hidden layer.<jupyter_code>TEXT = data.Field(lower=True, tokenize=list)
bs=64; bptt=16; n_fac=32; n_hidden=512

FILES = dict(train=TRN_PATH, validation=VAL_PATH, test=VAL_PATH)
md = LanguageModelData.from_text_files(PATH, TEXT, **FILES, bs=bs, bptt=bptt, min_freq=3)

len(md.trn_dl), md.nt, len(md.trn_ds), len(md.trn_ds[0].text)
em_sz = n_fac  # size of each embedding vector
nh = n_hidden     # number of hidden activations per layer
nl = 2       # number of layers<jupyter_output><empty_output><jupyter_text>We will use the Adam optimizer to train the model.The model itself has 5 dropout parameters.dropouth is applied to the activations going from one LSTM block to another.dropouti is applied to the input layer.dropoute is applied to the embedding layer.wdrop is applied to the LSTM's hidden weights.dropout is applied to the linear decoder that determines the final prediction.<jupyter_code>opt_fn = partial(optim.Adam, betas=(0.7, 0.99))
learner = md.get_model(opt_fn, em_sz, nh, nl,
               dropouti=0.1, dropout=0.1, wdrop=0.2, dropoute=0.04, dropouth=0.1)
learner.reg_fn = partial(seq2seq_reg, alpha=2, beta=1)
learner.clip=0.3<jupyter_output><empty_output><jupyter_text>This is the model structure. We start with an embedding matrix encoding our vocabulary. Then we have two LSTM modules, followed by a linear decoder.<jupyter_code>learner<jupyter_output><empty_output><jupyter_text>We select an optimum learning rate and proceed with training the model using stochastic gradient descent with restarts<jupyter_code>learner.lr_find(wds=1e-5)
learner.sched.plot()
learner.fit(1e-2, 2, wds=1e-5)
learner.lr_find(start_lr = 1e-7, end_lr = 1e-2, wds=1e-5)
learner.sched.plot()
learner.fit(1e-3, 3, wds=1e-5, cycle_len=1, cycle_mult=2)<jupyter_output><empty_output><jupyter_text>So our model's validation loss is about 1.51. What does this actually look like?<jupyter_code>def sample_model(m, s, l=50):
    t = TEXT.numericalize(s)
    m[0].bs=1
    m.eval()
    m.reset()
    res,*_ = m(t)
    print('...')

    for i in range(l):
        r = torch.multinomial(res[-1].exp(), 2)
        if r.data[0] == 0:
            r = r[1]
        else:
            r = r[0]
        word = TEXT.vocab.itos[to_np(r)[0]]
        res, *_ = m(r[0].unsqueeze(0))
        print(word, end='')
    m[0].bs=bs
m = learner.model
m.eval()
sample_model(m, "Good, speak to the mariners: fall to't, yarely", l=1000)<jupyter_output>...
 you can in, let he treft the sport rosper!who sing you princess fetch. refuse you will know:_break your gfgalianes know ourgree4behold; and not he thinch.            demeans the feace near wto indeed, all these;  for ever at said i envire in the yuty; he did dution    in gar5 marry, go a9hood from money.  iago. reason picpets; for he comes the great stood.      king us no surrey    that have not psrupks lik’d me dnines,'  with fool, possession. 6lweet dropping, said to mught come for thine husband spoke,send it a zoque. sek'st,    and viadwhich his princers and like of his sport of rest;the fman-toa sings.d servant.’ the days must gaved him that i think not not fo4 out again?air thas ehe were that you would brief all and counsel,power to starching guirts; yet'ss lidy,and he are me nature were to thee this will he were to   resimming only so the devil at equire to tongue.    'gainst me, by this, she dare with his dad itself. these done.let, my soul, that most bad —here is not a' th[...]<jupyter_text>So this isn't half bad for just a few epochs of training. Most of the words spelled are real words. The model has learned basic spelling and punctuation.But it's far from perfect. Some of the words have extra letters or even a number stuck in them. A few of the "words" look like someone smashing their keyboard. The model misses spaces sometimes.Most importantly, the whole sentences put out by the model don't make much sense. Some are pretty good. Some have a few sensible chunks but are otherwise meaningless. Some are just word vomit. Lets train the model a bit more and see if it gets better.<jupyter_code>learner.fit(1e-3, 3, wds=1e-5, cycle_len=2, cycle_mult=2)
m = learner.model
m.eval()
sample_model(m, "Good, speak to the mariners: fall to't, yarely", l=1000)<jupyter_output>...
.  peay6your king-here must she virtue.  gloucester. i do remember it is them.  ilas. hasthis so writing villain's son    as gentlemen with love at is thy s0boulp to do up a and hisade-several thank    good hiljures of witness.  nerissa. am i heard the court actionraoes on?  [unbrawledge sent of the bonds' houseenter gremio, isficy his sons. but the high crumpets are      as sworn at all in my tardiness and this holy blood?  somerset. o whe'er he was done and look    out of the heart, and me with suddstracge _henry,      good antony's enrivous natures, and i have server    for.  attended. i have say me name.    i'll find thy outlary trumpet in the    bnack with none.  boyet. what, says.  angelo. to the merry it my course.  gloucester. upon portly, who mores and wear with his pess'd    away, let mischance live all you, !  lafeu. ley me barren, the poet of single blood left white,    i dore to d if henry tald and devise so fair face:    both of offendest sweet wrongs in fearance,    [...]<jupyter_text>This looks a bit better. All the problems we identified before are still present, but the writing seems a little clearer.  4. Model Parameter TuningLets see if the model improves with more weights to learn on and a higher bptt value. Note the model will need slightly higher dropout parameters to account for the increased complexity.<jupyter_code>TEXT = data.Field(lower=True, tokenize=list)
bs=64; bptt=94; n_fac=74; n_hidden=1024

FILES = dict(train=TRN_PATH, validation=VAL_PATH, test=VAL_PATH)
md = LanguageModelData.from_text_files(PATH, TEXT, **FILES, bs=bs, bptt=bptt, min_freq=3)

len(md.trn_dl), md.nt, len(md.trn_ds), len(md.trn_ds[0].text)
em_sz = n_fac  # size of each embedding vector
nh = n_hidden     # number of hidden activations per layer
nl = 2       # number of layers
opt_fn = partial(optim.Adam, betas=(0.7, 0.99))
learner = md.get_model(opt_fn, em_sz, nh, nl,
               dropouti=0.11, dropout=0.11, wdrop=0.22, dropoute=0.05, dropouth=0.11)
learner.reg_fn = partial(seq2seq_reg, alpha=2, beta=1)
learner.clip=0.3
learner
learner.lr_find(wds=1e-5)
learner.sched.plot()
learner.fit(1e-2, 2, wds=1e-5)
learner.lr_find(start_lr = 1e-7, end_lr = 1e-2, wds=1e-6)
learner.sched.plot()
learner.fit(1e-3, 3, wds=1e-5, cycle_len=1, cycle_mult=2)<jupyter_output><empty_output><jupyter_text>It looks like the model is still training well despite the high bptt value. I was a little nervous setting it so high would make the model unstable.Right now our loss is comprable to where we tested the previous model. Lets see if the increased bptt results in better results at the same loss<jupyter_code>m = learner.model
m.eval()
sample_model(m, "Good, speak to the mariners: fall to't, yarely", l=1000)<jupyter_output>...
 sayor sox,    die, or embraces the merry clock,  men thhouldst one kind lack'd.      ;    give me the true than 'mind.  gratiano. never love thy faith, revenge, and four i would be think,    and with her atsend beastly to <pad>dost not    have thee down adieu. i could rome me with this —hand.    when you shall hinding the |be made alns    to expect kindred at those clouds?    dear soul commend a it, sir, but knows hrock our son upon't.      enter hostess falstaff, how seft them heary  chesture. i'll had more unhand'st thou young condition    wlifted i cannot be your face, above it this beds,    and there is so, tucked not. sir, je her that lock'd    pistol’s wonder.      claudio.      thf near trees let me more than his soul, and dismiss to attend, and    is not bemar'd in every water more flesh?    most dangerous lancaster! a season is true g[exeunt,    thou hast as my wife, entertain'd no peoplation of gone?  queen margaret. it is everrop me; and show me to reputation shall    h[...]<jupyter_text>There doesn't seem to be an obvious effect.Lets see where we can take this thing<jupyter_code>learner.fit(1e-3, 1, wds=1e-5, cycle_len=8)
learner.lr_find(start_lr = 1e-7, end_lr = 5e-3, wds=1e-5)
learner.sched.plot()
learner.fit(1e-3, 3, wds=1e-5, cycle_len=2, cycle_mult=2)
learner.fit(5e-4, 3, wds=1e-5, cycle_len=1, cycle_mult=2)
learner.fit(1e-4, 3, wds=1e-5, cycle_len=2, cycle_mult=2)
m = learner.model
m.eval()
sample_model(m, "Good, speak to the mariners: fall to't, yarely", l=3000)
learner.save('shakespeare')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08267251560654632, 'fraction_numerical': 0.01569090602328328, 'mean_word_length': 4.2407898614795165, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 52, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '526733', 'license': 'MIT', 'n_tokens_mistral': 5681, 'n_tokens_neox': 5459, 'n_words': 2455, 'path': 'Shakespeare RNN/Shakespeare RNN.ipynb', 'repo_name': 'Holi12345/ML-DL-Projects', 'size': '17785'}