<jupyter_start><jupyter_text>Language modelling In this tutorial we will use two character level language models to create dinosaur names. We will recurrent neural networks as main tool for language modellig.  ![picture](https://vignette.wikia.nocookie.net/uncyclopedia/images/7/73/Hankk_the_dino.png/revision/latest?cb=20100127020302)<jupyter_code>!wget https://raw.githubusercontent.com/artemovae/NLP-seminar-LM/master/dinos.txt<jupyter_output>/bin/sh: wget: command not found<jupyter_text>Recurrent neural networksInput:$x_{1:n} = x_1, x_2, \ldots, x_n$, $x_i \in \mathbb{R}^{d_{in}}$For each input $x_{1:i}$ we get an output $y_i$:$y_i = RNN(x_{1:i})$, $y_i \in \mathbb{R}^{d_{out}}$For the whole input sequence $x_{1:n}$:$y_{1:n} = RNN^{*}(x_{1:n})$, $y_i \in \mathbb{R}^{d_{out}}$ $R$ is a recursive activation function with two inputs: $x_i$ и $s_{i-1}$ (state vector)$RNN^{*}(x_{1:n}, s_0) = y_{1:n}$$y_i = O(s_i) = g(W^{out}[s_{i} ,x_i] +b)$$s_i = R(s_{i-1}, x_i)$$s_i = R(s_{i-1}, x_i) = g(W^{hid}[s_{i-1} ,x_i] +b)$  -- concatenate $[s_{i-1}, x]$$x_i \in \mathbb{R}^{d_{in}}$, $y_i \in \mathbb{R}^{ d_{out}}$, $s_i \in \mathbb{R}^{d_{hid}}$$W^{hid} \in \mathbb{R}^{(d_{in}+d_{out}) \times d_{hid}}$, $W^{out} \in \mathbb{R}^{d_{hid} \times d_{out}}$ ![rnn](https://github.com/enggen/Deep-Learning-Coursera/raw/1407e19c98833d2686a0748db26b594f3102301e/Sequence%20Models/Week1/Dinosaur%20Island%20--%20Character-level%20language%20model/images/rnn.png) We are going to create an RNN-LM using pytorch<jupyter_code>import numpy as np
import random
import torch
import torch.nn as nn
import torch.optim as optim
import pdb
from torch.utils.data import Dataset, DataLoader

%load_ext autoreload
%autoreload 2

torch.set_printoptions(linewidth=200)
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
hidden_size = 50

print(device)<jupyter_output>cpu<jupyter_text>**Step 1**. Prepare the dataset<jupyter_code>class DinosDataset(Dataset):
    def __init__(self):
        super().__init__()
        with open('dinos.txt') as f:
            content = f.read().lower()
            self.vocab = sorted(set(content)) + ['<', '>']
            self.vocab_size = len(self.vocab)
            self.lines = content.splitlines()
        self.ch_to_idx = {c:i for i, c in enumerate(self.vocab)}
        self.idx_to_ch = {i:c for i, c in enumerate(self.vocab)}
    
    def __getitem__(self, index):
        line = self.lines[index]
        #teacher forcing
        x_str = '<' + line 
        y_str = line + '>' 
        x = torch.zeros([len(x_str), self.vocab_size], dtype=torch.float)
        y = torch.empty(len(x_str), dtype=torch.long)
        for i, (x_ch, y_ch) in enumerate(zip(x_str, y_str)):
            x[i][self.ch_to_idx[x_ch]] = 1
            y[i] = self.ch_to_idx[y_ch]
        
        return x, y
    
    def __len__(self):
        return len(self.lines)
trn_ds = DinosDataset()
trn_dl = DataLoader(trn_ds, shuffle=True)
trn_ds.lines[1]
print(trn_ds.idx_to_ch)
trn_ds.vocab_size
x, y = trn_ds[1]
x.shape
y.shape<jupyter_output><empty_output><jupyter_text>**Step 2**. Define a model, loss function and optimization algorithm<jupyter_code>class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        self.dropout = nn.Dropout(0.3)
        self.i2o = nn.Linear(input_size + hidden_size, output_size)
    
    def forward(self, h_prev, x):
        combined = torch.cat([h_prev, x], dim = 1) # concatenate x and h
        h = torch.tanh(self.dropout(self.i2h(combined)))
        y = self.i2o(combined)
        return h, y
model = RNN(trn_ds.vocab_size, hidden_size, trn_ds.vocab_size).to(device)
loss_fn = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=1e-2)<jupyter_output><empty_output><jupyter_text>**Step 3**. Declare a sampling procedure ![rnn](https://github.com/enggen/Deep-Learning-Coursera/raw/1407e19c98833d2686a0748db26b594f3102301e/Sequence%20Models/Week1/Dinosaur%20Island%20--%20Character-level%20language%20model/images/dinos3.png) Sidenote: we can use Softmax probabilities to evaluate perplexety, which a standart quality measure for the task$ 2^{{H(p)}}=2^{{-\sum _{x}p(\hat(y)^{} == y^{}_{train} )\log _{2}p(\hat(y)^{} == y^{}_{train})}} $<jupyter_code>def sample(model):
    model.eval()
    word_size=0
    newline_idx = trn_ds.ch_to_idx['>']
    with torch.no_grad():
        h_prev = torch.zeros([1, hidden_size], dtype=torch.float, device=device)
        x = h_prev.new_zeros([1, trn_ds.vocab_size])
        start_char_idx = trn_ds.ch_to_idx['<']
        indices = [start_char_idx]
        x[0, start_char_idx] = 1
        predicted_char_idx = start_char_idx
        
        while predicted_char_idx != newline_idx and word_size != 50:
            h_prev, y_pred = model(h_prev, x)
            y_softmax_scores = torch.softmax(y_pred, dim=1)
            
            np.random.seed(np.random.randint(1, 5000))
            idx = np.random.choice(np.arange(trn_ds.vocab_size), p=y_softmax_scores.cpu().numpy().ravel())
            indices.append(idx)
            
            x = (y_pred == y_pred.max(1)[0]).float()
 
            
            predicted_char_idx = idx
            
            word_size += 1
        
        if word_size == 50:
            indices.append(newline_idx)
    return indices
def print_sample(sample_idxs):
    [print(trn_ds.idx_to_ch[x], end ='') for x in sample_idxs]
    print()<jupyter_output><empty_output><jupyter_text>**Step 4**. Almost done! Train the model<jupyter_code>def train_one_epoch(model, loss_fn, optimizer):
    model.train()
    for line_num, (x, y) in enumerate(trn_dl):
        loss = 0
        optimizer.zero_grad()
        h_prev = torch.zeros([1, hidden_size], dtype=torch.float, device=device)
        x, y = x.to(device), y.to(device)
        for i in range(x.shape[1]):
            h_prev, y_pred = model(h_prev, x[:, i])
            loss += loss_fn(y_pred, y[:, i])
            
        if (line_num+1) % 100 == 0:
            print_sample(sample(model))
        loss.backward()
        optimizer.step()
def train(model, loss_fn, optimizer, dataset='dinos', epochs=1):
    for e in range(1, epochs+1):
        print('Epoch:{}'.format(e))
        train_one_epoch(model, loss_fn, optimizer)
        print()
train(model, loss_fn, optimizer, epochs = 5)<jupyter_output>Epoch:1
<aunuaauras>
<tmrasausus>
<suroiturus>
<rusasaerus>
<gubestarus>
<icsas>
<puattorastp>
<tqrusauras>
<aurubauras>
<tlrasaos>
<uerus>
<duriyrps>
<tapnbauius>
<ahroaoes>
<hcras>

Epoch:2
<puiussurus>
<zurusuarup>
<scrmtanopdor>
<gnbucaurus>
<turesnusus>
<suapsanras>
<taiusaurus>
<aucnsnuous>
<llrur>
<srrusauras>
<aumuaaures>
<sancucaurus>
<ttilosaurus>
<tarocaulus>
<ahrnbamsopaurus>

Epoch:3
<uhrusaurus>
<sprusaurus>
<aurucauris>
<smnasaurus>
<suroiourus>
<purasderus>
<fuaitocem>
<gmbucaurus>
<turasiusus>
<suainaurts>
<tchspaurus>
<aubnteurus>
<llrrpyurus>
<aprapoctosic>
<larhsaueur>

Epoch:4
<xrsasosooeurus>
<putcoodateor>
<agpnaasrus>
<tdruptirus>
<durotturusaurasris>
<tceusaurus>
<auaotpurus>
<llrur>
<sprosantap>
<aupucauras>
<tmrasaurus>
<surorturus>
<purareerus>
<fuabsacros>
<hcranturus>

Epoch:5
<srrusius>
<purarakrus>
<euaetocaurus>
<tcivotaxtsaurus>
<snrusaurss>
<aubtaecuhus>
<gmbuchytrl>
<turistot>
<ltrubgocaurus>
<agrpaoltpterui>
<pucusttrus>
<zurustcrus>
<scrisanrui>
<c[...]
{'fraction_non_alphanumeric': 0.15604365988411265, 'fraction_numerical': 0.02465974935992454, 'mean_word_length': 3.87007874015748, 'pattern_counts': {'":': 0, '<': 99, '<?xml version=': 0, '>': 98, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '549069', 'license': 'MIT', 'n_tokens_mistral': 3127, 'n_tokens_neox': 2993, 'n_words': 538, 'path': 'notebooks/day-5/language_model/lm_solved.ipynb', 'repo_name': 'mzgubic/mlhep2019', 'size': '7421'}