<jupyter_start><jupyter_text>Lesson 12 - FastAI<jupyter_code>#hide
!pip install -Uqq fastbook
import fastbook
fastbook.setup_book()
#hide
from fastbook import *<jupyter_output><empty_output><jupyter_text>A Language Model from ScratchWe have worked with NLP models in the previous lecture, and have seen the many benefits and capabilities of such a model. Now let's try to create our very own model from scratch!  The Data<jupyter_code>from fastai.text.all import *
path = untar_data(URLs.HUMAN_NUMBERS)
#hide
Path.BASE_PATH = path
path.ls()
lines = L()
with open(path/'train.txt') as f: lines += L(*f.readlines())
with open(path/'valid.txt') as f: lines += L(*f.readlines())
lines<jupyter_output><empty_output><jupyter_text>> This is what our data looks like right now<jupyter_code>text = ' . '.join([l.strip() for l in lines]) #Reformating
text[:100]
tokens = text.split(' ') #Now lets tokenize it
tokens[:10]
vocab = L(*tokens).unique() #lets create our vocab
vocab<jupyter_output><empty_output><jupyter_text>> This will be our vocab: Lets also numericalize it.<jupyter_code>word2idx = {w:i for i,w in enumerate(vocab)} #Dictionary of word:id

nums = L(word2idx[i] for i in tokens) #Numericalization
tokens[:10], nums<jupyter_output><empty_output><jupyter_text>Dataloader<jupyter_code>L((tokens[i:i+3], tokens[i+3]) for i in range(0,len(tokens)-4,3))<jupyter_output><empty_output><jupyter_text>> Tokens are created so that the 4th token is the label.<jupyter_code>seqs = L((tensor(nums[i:i+3]), nums[i+3]) for i in range(0,len(nums)-4,3)) #Lets do the above, but this time using 
seqs                                                                       #numericalization form

bs = 64
cut = int(len(seqs) * 0.8) #80% training set, 20% valid set

dls = DataLoaders.from_dsets(seqs[:cut], seqs[cut:], bs=64, shuffle=False) 
dls.one_batch()[0][:2]<jupyter_output><empty_output><jupyter_text>Our Language Model in PyTorchBelow is our language model. As you see, we have created three layers:- The embedding layer (`i_h`, for *input* to *hidden*)- The linear layer to create the activations for the next word (`h_h`, for *hidden* to *hidden*)- A final linear layer to predict the fourth word (`h_o`, for *hidden* to *output*)<jupyter_code>class LMModel1(Module):
    def __init__(self, vocab_sz, n_hidden):
        self.i_h = nn.Embedding(vocab_sz, n_hidden)#input
        self.h_h = nn.Linear(n_hidden, n_hidden)#hidden     
        self.h_o = nn.Linear(n_hidden,vocab_sz)#output
        
    def forward(self, x):
        h = F.relu(self.h_h(self.i_h(x[:,0]))) #word 1
        h = h + self.i_h(x[:,1]) #word 2
        h = F.relu(self.h_h(h))
        h = h + self.i_h(x[:,2]) #word 3
        h = F.relu(self.h_h(h))
        return self.h_o(h) #pred (word 4)<jupyter_output><empty_output><jupyter_text>Train<jupyter_code>learn = Learner(dls, LMModel1(len(vocab), 64), loss_func=F.cross_entropy, 
                metrics=accuracy)
learn.fit_one_cycle(4, 1e-3)<jupyter_output><empty_output><jupyter_text>> Awesome we created our first NLP model!<jupyter_code>n,counts = 0,torch.zeros(len(vocab))
for x,y in dls.valid:
    n += y.shape[0]
    for i in range_of(vocab): counts[i] += (y==i).long().sum()
idx = torch.argmax(counts)
idx, vocab[idx.item()], counts[idx].item()/n<jupyter_output><empty_output><jupyter_text>> Our acc would have been .15 had we used an naive model  Refining our model - Recurrent Neural NetworkLets refactor our above model.<jupyter_code>class LMModel2(Module):
    def __init__(self, vocab_sz, n_hidden):
        self.i_h = nn.Embedding(vocab_sz, n_hidden)  
        self.h_h = nn.Linear(n_hidden, n_hidden)     
        self.h_o = nn.Linear(n_hidden,vocab_sz)
        
    def forward(self, x):
        h = 0
        for i in range(3): #lets use a forloop to create layers
            h = h + self.i_h(x[:,i])
            h = F.relu(self.h_h(h))
        return self.h_o(h)<jupyter_output><empty_output><jupyter_text>> Notice that here h is set to 0 after every batch<jupyter_code>learn = Learner(dls, LMModel2(len(vocab), 64), loss_func=F.cross_entropy, 
                metrics=accuracy)
learn.fit_one_cycle(4, 1e-3)<jupyter_output><empty_output><jupyter_text>> Roughly the same as we expected. However, what we have created this time is actually an RNN!  Improving the RNN  Maintaining the State of an RNNThe first way we can improve our model is by actually remembering the state of h from the previous batches. Recall that before we set h=0 after every batch.<jupyter_code>class LMModel3(Module):
    def __init__(self, vocab_sz, n_hidden):
        self.i_h = nn.Embedding(vocab_sz, n_hidden)  
        self.h_h = nn.Linear(n_hidden, n_hidden)     
        self.h_o = nn.Linear(n_hidden,vocab_sz)
        self.h = 0
        
    def forward(self, x):
        for i in range(3):
            self.h = self.h + self.i_h(x[:,i])
            self.h = F.relu(self.h_h(self.h))
        out = self.h_o(self.h)
        
        self.h = self.h.detach() #detach throws away the stored gradients - However, the activations are still stored
        return out
    
    #Start of each epoch, we should reset out h
    def reset(self): self.h = 0
m = len(seqs)//bs
m,bs,len(seqs)<jupyter_output><empty_output><jupyter_text>MinibatchesRecall from Lecture 10 how we created the minibatches. Where the nth row from a minibatch followed the nth row from the previous minibatch. The function below does exactly that for us.<jupyter_code>#This function formats our minibatches appropriately
def group_chunks(ds, bs):
    m = len(ds) // bs
    new_ds = L()
    for i in range(m): new_ds += L(ds[i + m*j] for j in range(bs))
    return new_ds
group_chunks(seqs, bs)[1:4]
seqs[m,m*2,m*3]<jupyter_output><empty_output><jupyter_text>**Lets recreate our dataloader using our improved minibatch format**<jupyter_code>cut = int(len(seqs) * 0.8)

dls = DataLoaders.from_dsets(
    group_chunks(seqs[:cut], bs), 
    group_chunks(seqs[cut:], bs), 
    bs=bs, drop_last=True, shuffle=False)
learn = Learner(dls, LMModel3(len(vocab), 64), loss_func=F.cross_entropy,
                metrics=accuracy, cbs=ModelResetter) #This will call our reset function
learn.fit_one_cycle(10, 3e-3)<jupyter_output><empty_output><jupyter_text>Creating More SignalRather than predicting every 4th word, why don't we predict every other word.<jupyter_code>sl = 16
seqs = L((tensor(nums[i:i+sl]), tensor(nums[i+1:i+sl+1]))
         for i in range(0,len(nums)-sl-1,sl))

cut = int(len(seqs) * 0.8)

#dataloader
dls = DataLoaders.from_dsets(group_chunks(seqs[:cut], bs),
                             group_chunks(seqs[cut:], bs),
                             bs=bs, drop_last=True, shuffle=False)
seqs[0]
[vocab[s] for s in seqs[0]]
class LMModel4(Module):
    def __init__(self, vocab_sz, n_hidden):
        self.i_h = nn.Embedding(vocab_sz, n_hidden)  
        self.h_h = nn.Linear(n_hidden, n_hidden)     
        self.h_o = nn.Linear(n_hidden,vocab_sz)
        self.h = 0
        
    def forward(self, x):
        outs = [] #list of output
        
        for i in range(sl):
            self.h = self.h + self.i_h(x[:,i])
            self.h = F.relu(self.h_h(self.h))
            
            outs.append(self.h_o(self.h)) #append
            
        self.h = self.h.detach()
        return torch.stack(outs, dim=1) #stack of outputs
    
    def reset(self): self.h = 0
#loss func that flattens
def loss_func(inp, targ):
    return F.cross_entropy(inp.view(-1, len(vocab)), targ.view(-1))
learn = Learner(dls, LMModel4(len(vocab), 64), loss_func=loss_func,
                metrics=accuracy, cbs=ModelResetter)
learn.fit_one_cycle(15, 3e-3)<jupyter_output><empty_output><jupyter_text>> Better than before!  Multilayer RNNsLets create a deeper and more layered RNN. What's unique about this RNN is that each layer will have a different weight matrix. Lets change our model and see how it performs.  The Model<jupyter_code>class LMModel5(Module):
    def __init__(self, vocab_sz, n_hidden, n_layers):
        self.i_h = nn.Embedding(vocab_sz, n_hidden)
        self.h_o = nn.Linear(n_hidden, vocab_sz)
        self.h = torch.zeros(n_layers, bs, n_hidden)
                                             #How many layer to stack  
        self.rnn = nn.RNN(n_hidden, n_hidden, n_layers, batch_first=True) #This is doing what our previous model did:
                                                                          #Looping of the layers      
    def forward(self, x):
        res,h = self.rnn(self.i_h(x), self.h) #Notice that we can do the loop by calling our RNN
        self.h = h.detach() 
        return self.h_o(res)
    
    def reset(self): self.h.zero_()
learn = Learner(dls, LMModel5(len(vocab), 64, 2), 
                loss_func=CrossEntropyLossFlat(), 
                metrics=accuracy, cbs=ModelResetter)
learn.fit_one_cycle(15, 3e-3)<jupyter_output><empty_output><jupyter_text>> Our model did worse. Does that mean our model is bad? No, what most likley happened here is that our gradient has either exploded or disappeared.  LSTMWe can fix the issue of gradients exploding or disappearing by creating another type of architecture, LSTM.  Building an LSTM from Scratch<jupyter_code>class LSTMCell(Module):
    def __init__(self, ni, nh):
        self.forget_gate = nn.Linear(ni + nh, nh)
        self.input_gate  = nn.Linear(ni + nh, nh)
        self.cell_gate   = nn.Linear(ni + nh, nh)
        self.output_gate = nn.Linear(ni + nh, nh)

    def forward(self, input, state):
        h,c = state
        h = torch.cat([h, input], dim=1)
        forget = torch.sigmoid(self.forget_gate(h))
        c = c * forget
        inp = torch.sigmoid(self.input_gate(h))
        cell = torch.tanh(self.cell_gate(h))
        c = c + inp * cell
        out = torch.sigmoid(self.output_gate(h))
        h = out * torch.tanh(c)
        return h, (h,c)<jupyter_output><empty_output><jupyter_text>**We can refactor the above code**<jupyter_code>class LSTMCell(Module):
    def __init__(self, ni, nh):
        self.ih = nn.Linear(ni,4*nh)
        self.hh = nn.Linear(nh,4*nh)

    def forward(self, input, state):
        h,c = state
        # One big multiplication for all the gates is better than 4 smaller ones
        gates = (self.ih(input) + self.hh(h)).chunk(4, 1)
        ingate,forgetgate,outgate = map(torch.sigmoid, gates[:3])
        cellgate = gates[3].tanh()

        c = (forgetgate*c) + (ingate*cellgate)
        h = outgate * c.tanh()
        return h, (h,c)
t = torch.arange(0,10); t
t.chunk(2)<jupyter_output><empty_output><jupyter_text>Training a Language Model Using LSTMs<jupyter_code>class LMModel6(Module):
    def __init__(self, vocab_sz, n_hidden, n_layers):
        self.i_h = nn.Embedding(vocab_sz, n_hidden)
        self.h_o = nn.Linear(n_hidden, vocab_sz)
        self.h = [torch.zeros(n_layers, bs, n_hidden) for _ in range(2)] #more hidden state layers because LSTM has more layers
        
        self.rnn = nn.LSTM(n_hidden, n_hidden, n_layers, batch_first=True) #Replace our RNN to LSTM
        
    def forward(self, x):
        res,h = self.rnn(self.i_h(x), self.h)
        self.h = [h_.detach() for h_ in h]
        return self.h_o(res)
    
    def reset(self): 
        for h in self.h: h.zero_()
learn = Learner(dls, LMModel6(len(vocab), 64, 2), 
                loss_func=CrossEntropyLossFlat(), 
                metrics=accuracy, cbs=ModelResetter)
learn.fit_one_cycle(15, 1e-2)<jupyter_output><empty_output><jupyter_text>> Our model is doing much better now  Regularizing an LSTMWill be using some regularization technique to improve our model's gradients, particularly Dropout. What dropout does is that it randomly drops some neurons each minibatch: This forces the model to become more robust by making it able to produce the correct prediction even with less neurons available.   Dropout<jupyter_code>class Dropout(Module):
    def __init__(self, p): 
        self.p = p #probility that activation gets deleted
        
    def forward(self, x):
        if not self.training:  #NO DROPOUT DURING TESTING (Only occurs during training)
            return x
        
        mask = x.new(*x.shape).bernoulli_(1-self.p) #1's and 0's where 1-p is the prob that we get a 1
        return x * mask.div_(1-self.p)<jupyter_output><empty_output><jupyter_text>Activation Regularization and Temporal Activation Regularization *Activation regularization* (AR) and *temporal activation regularization* (TAR) are two regularization methods very similar to weight decay, which we have discussed before.   For activation regularization, it's the final activations produced by the LSTM that we will try to make as small as possible, instead of the weights.``` pythonloss += alpha * activations.pow(2).mean()``` Temporal activation regularization is there to encourage that behavior by adding a penalty to the loss to make the difference between two consecutive activations as small as possible:``` pythonloss += beta * (activations[:,1:] - activations[:,:-1]).pow(2).mean()```  Training a Weight-Tied Regularized LSTM<jupyter_code>class LMModel7(Module):
    def __init__(self, vocab_sz, n_hidden, n_layers, p):
        self.i_h = nn.Embedding(vocab_sz, n_hidden)
        self.rnn = nn.LSTM(n_hidden, n_hidden, n_layers, batch_first=True)
        
        self.drop = nn.Dropout(p) #Dropout
        self.h_o = nn.Linear(n_hidden, vocab_sz)
        
        self.h_o.weight = self.i_h.weight #Hidden-to-output weights are set identical to input-to-hidden
        
        self.h = [torch.zeros(n_layers, bs, n_hidden) for _ in range(2)]
        
    def forward(self, x):
        raw,h = self.rnn(self.i_h(x), self.h)
        out = self.drop(raw)
        self.h = [h_.detach() for h_ in h]
        return self.h_o(out),raw,out
    
    def reset(self): 
        for h in self.h: h.zero_()<jupyter_output><empty_output><jupyter_text>> Notice that Hidden-to-output and input-to-hidden are linked by the same parameters<jupyter_code>learn = Learner(dls, LMModel7(len(vocab), 64, 2, 0.5),
                loss_func=CrossEntropyLossFlat(), metrics=accuracy,
                cbs=[ModelResetter, RNNRegularizer(alpha=2, beta=1)]) #Although we didn't create our regularizer, we can still
                                                                      #pass it via cbs. 
learn = TextLearner(dls, LMModel7(len(vocab), 64, 2, 0.4),
                    loss_func=CrossEntropyLossFlat(), metrics=accuracy)<jupyter_output><empty_output><jupyter_text>> Calling TextLearner will automatically add ModelResetter, RNNRegularizer(alpha=2, beta=1) for us<jupyter_code>learn.fit_one_cycle(15, 1e-2, wd=0.1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11607569857211177, 'fraction_numerical': 0.0116827218692355, 'mean_word_length': 3.2416690814256737, 'pattern_counts': {'":': 0, '<': 93, '<?xml version=': 0, '>': 105, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '258575', 'license': 'Apache-2.0', 'n_tokens_mistral': 5281, 'n_tokens_neox': 5061, 'n_words': 1345, 'path': '_notebooks/2021-7-22-Lesson12-nlp_dive.ipynb', 'repo_name': 'usama280/PasteBlogs', 'size': '14637'}