<jupyter_start><jupyter_text>Language modelling In this tutorial we will use two character level language models to create dinosaur names. These models are:* bigram model* recurrent neural network model.Why are they different?<jupyter_code><jupyter_output><empty_output><jupyter_text>![picture](https://vignette.wikia.nocookie.net/uncyclopedia/images/7/73/Hankk_the_dino.png/revision/latest?cb=20100127020302)<jupyter_code>from google.colab import files
uploaded = files.upload()<jupyter_output><empty_output><jupyter_text>Bigram modelUncomment the cell below if you do not have NLTK installed.<jupyter_code>names = ['<' + name.strip().lower() + '>' for name in open('dinos.txt').readlines()]
print(names[:10])
!pip install nltk
import nltk<jupyter_output><empty_output><jupyter_text>**Step 1.** Compute frequency of each character<jupyter_code>chars = [char  for name in names for char in name]
freq = nltk.FreqDist(chars)
print(list(freq.keys()))
freq.most_common(10)<jupyter_output><empty_output><jupyter_text>Define a function to estimate probabilty of character<jupyter_code>l = sum([freq[char] for char in freq])
def unigram_prob(char):
    return freq[char] / l
print('p(a) = %1.4f' %unigram_prob('a'))<jupyter_output>p(a) = 0.1160<jupyter_text>**Step 2.** Compute frequency of each character condtioned on the previous one<jupyter_code>cfreq = nltk.ConditionalFreqDist(nltk.bigrams(chars),)
# кол-во раз, когда какая-то буква встреается ПОСЛЕ а
cfreq['a']<jupyter_output><empty_output><jupyter_text>**Step 3.** Use MLE to estimate condtional probabilities<jupyter_code>cprob = nltk.ConditionalProbDist(cfreq, nltk.MLEProbDist)
print('p(a a) = %1.4f' %cprob['a'].prob('a'))
print('p(a b) = %1.4f' %cprob['a'].prob('b'))
print('p(a u) = %1.4f' %cprob['a'].prob('u'))<jupyter_output>p(a a) = 0.0044
p(a b) = 0.0097
p(a u) = 0.3181<jupyter_text>We can use cprob to generate next characters  Task 1.a. Write a function to generate a dinosaur name of **fixed** length. Use '<' as a start of name symbol.b. Write a to generate a dinosaur names of any length.<jupyter_code>## let's ommit for a while function for a name with fixed length<jupyter_output><empty_output><jupyter_text>HW, Problem 3.1  a) dinosaur name of any length<jupyter_code>def dinosaur_name_any_length(cprob = cprob):
  start_symbol = "<"
  dino_name = "<"

  new_char = cprob[start_symbol].generate()
  dino_name = dino_name + new_char

  while True:
    new_char = cprob[new_char].generate()
    dino_name = dino_name + new_char
    if new_char == ">":
      break
  return dino_name
  

  
name = dinosaur_name_any_length(cprob = cprob)
/print(name)<jupyter_output><gnosarusatiaurus><jupyter_text>b) Add add-one smoothing. What is the difference?<jupyter_code>## add-1 smoothing
cprob_add_one_smoothing = nltk.ConditionalProbDist(cfreq, nltk.LaplaceProbDist)
name = dinosaur_name_any_length(cprob = cprob_add_one_smoothing)
print(name)<jupyter_output><aclelodelis><jupyter_text>With add-one smoothing we can get more names (all possible combinations of letters) as P(letter 1 | letter 2) != 0 for any letter 1 and 2  Recurrent neural networksInput:$x_{1:n} = x_1, x_2, \ldots, x_n$, $x_i \in \mathbb{R}^{d_{in}}$For each input $x_{1:i}$ we get an output $y_i$:$y_i = RNN(x_{1:i})$, $y_i \in \mathbb{R}^{d_{out}}$For the whole input sequence $x_{1:n}$:$y_{1:n} = RNN^{*}(x_{1:n})$, $y_i \in \mathbb{R}^{d_{out}}$ $R$ is a recursive activation function with two inputs: $x_i$ и $s_{i-1}$ (state vector)$RNN^{*}(x_{1:n}, s_0) = y_{1:n}$$y_i = O(s_i) = g(W^{out}[s_{i} ,x_i] +b)$$s_i = R(s_{i-1}, x_i)$$s_i = R(s_{i-1}, x_i) = g(W^{hid}[s_{i-1} ,x_i] +b)$  -- concatenate $[s_{i-1}, x]$$x_i \in \mathbb{R}^{d_{in}}$, $y_i \in \mathbb{R}^{ d_{out}}$, $s_i \in \mathbb{R}^{d_{hid}}$$W^{hid} \in \mathbb{R}^{(d_{in}+d_{out}) \times d_{hid}}$, $W^{out} \in \mathbb{R}^{d_{hid} \times d_{out}}$ ![rnn](https://github.com/enggen/Deep-Learning-Coursera/raw/1407e19c98833d2686a0748db26b594f3102301e/Sequence%20Models/Week1/Dinosaur%20Island%20--%20Character-level%20language%20model/images/rnn.png) We are going to create an RNN-LM using pytorch<jupyter_code>import numpy as np
import random
import torch
import torch.nn as nn
import torch.optim as optim
import pdb
from torch.utils.data import Dataset, DataLoader

%load_ext autoreload
%autoreload 2

torch.set_printoptions(linewidth=200)
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
hidden_size = 50

print(device)<jupyter_output>cuda:0<jupyter_text>**Step 1**. Prepare the dataset<jupyter_code>class DinosDataset(Dataset):
    def __init__(self):
        super().__init__()
        with open('dinos.txt') as f:
            content = f.read().lower()
            self.vocab = sorted(set(content)) + ['<', '>']
            self.vocab_size = len(self.vocab)
            self.lines = content.splitlines()
        self.ch_to_idx = {c:i for i, c in enumerate(self.vocab)}
        self.idx_to_ch = {i:c for i, c in enumerate(self.vocab)}
    
    def __getitem__(self, index):
        line = self.lines[index]
        
        ######################################
        ## teacher forcing
        x_str = '<' + line       # size = len x 28 ~ One hot encoding of letters. There are 'len' letters
        y_str = line + '>'       # size = len  ~ number of letters
        ######################################
        
        x = torch.zeros([len(x_str), self.vocab_size], dtype=torch.float)
        y = torch.empty(len(x_str), dtype=torch.long)
        for i, (x_ch, y_ch) in enumerate(zip(x_str, y_str)):
            x[i][self.ch_to_idx[x_ch]] = 1
            y[i] = self.ch_to_idx[y_ch]
        
        return x, y
    
    def __len__(self):
        return len(self.lines)
trn_ds = DinosDataset()
trn_dl = DataLoader(trn_ds, shuffle=True)
trn_ds.lines[1]
print(trn_ds.idx_to_ch)
trn_ds.vocab_size
x, y = trn_ds[1]
x.shape
y.shape<jupyter_output><empty_output><jupyter_text>**Step 2**. Define a model, loss function and optimization algorithm<jupyter_code>class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)  # input to hidden matrix
        self.dropout = nn.Dropout(0.3)
        self.i2o = nn.Linear(input_size + hidden_size, output_size)  # input to output matrix
    
    def forward(self, h_prev, x):
        combined = torch.cat([h_prev, x], dim = 1) # concatenate x and h
        h = torch.tanh(self.dropout(self.i2h(combined)))
        y = self.i2o(combined)
        return h, y
model = RNN(trn_ds.vocab_size, hidden_size, trn_ds.vocab_size).to(device)
loss_fn = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=1e-2)<jupyter_output><empty_output><jupyter_text>**Step 3**. Declare a sampling procedure ![rnn](https://github.com/enggen/Deep-Learning-Coursera/raw/1407e19c98833d2686a0748db26b594f3102301e/Sequence%20Models/Week1/Dinosaur%20Island%20--%20Character-level%20language%20model/images/dinos3.png)<jupyter_code>def sample(model):
    model.eval()
    word_size=0
    newline_idx = trn_ds.ch_to_idx['>']
    with torch.no_grad():
        h_prev = torch.zeros([1, hidden_size], dtype=torch.float, device=device)
        x = h_prev.new_zeros([1, trn_ds.vocab_size])
        start_char_idx = trn_ds.ch_to_idx['<']
        indices = [start_char_idx]
        x[0, start_char_idx] = 1
        predicted_char_idx = start_char_idx
        
        while predicted_char_idx != newline_idx and word_size != 50:
            h_prev, y_pred = model(h_prev, x)
            y_softmax_scores = torch.softmax(y_pred, dim=1)
            
            np.random.seed(np.random.randint(1, 5000))
            idx = np.random.choice(np.arange(trn_ds.vocab_size), p=y_softmax_scores.cpu().numpy().ravel())
            indices.append(idx)
            
            x = (y_pred == y_pred.max(1)[0]).float()
 
            
            predicted_char_idx = idx
            
            word_size += 1
        
        if word_size == 50:
            indices.append(newline_idx)
    return indices
def print_sample(sample_idxs):
    [print(trn_ds.idx_to_ch[x], end ='') for x in sample_idxs]
    print()<jupyter_output><empty_output><jupyter_text>**Step 4**. Almost done! Train the model<jupyter_code>def train_one_epoch(model, loss_fn, optimizer):
    model.train()
    for line_num, (x, y) in enumerate(trn_dl):
        loss = 0
        optimizer.zero_grad()
        h_prev = torch.zeros([1, hidden_size], dtype=torch.float, device=device)
        x, y = x.to(device), y.to(device)
        for i in range(x.shape[1]):
            h_prev, y_pred = model(h_prev, x[:, i])
            loss += loss_fn(y_pred, y[:, i])
            
        if (line_num+1) % 100 == 0:
            print_sample(sample(model))
        loss.backward()
        optimizer.step()
def train(model, loss_fn, optimizer, dataset='dinos', epochs=1):
    for e in range(1, epochs+1):
        print('Epoch:{}'.format(e))
        train_one_epoch(model, loss_fn, optimizer)
        print()
print(model)
train(model, loss_fn, optimizer, epochs = 50)<jupyter_output>Epoch:1
<nba>
<juuoat>
<oaalonasruslhs>
<lacusasnus>
<aubot>
<xerusiurus>
<tnrusasras>
<aunuaauras>
<tinasausus>
<turooourus>
<rtraseerus>
<guaascarus>
<haras>
<pugus>
<mirus>

Epoch:2
<tnotrahpapseur>
<aeroainrus>
<tatsosaurus>
<sipttrnuras>
<aaucontcrusaurus>
<ancrusturus>
<eiurysaurui>
<lamudtssas>
<marhsauius>
<yrsas>
<splushurus>
<taoocaurus>
<agrraiurus>
<tcrus>
<bururaurut>

Epoch:3
<ltrua>
<lbrucspras>
<lardsaurus>
<xtsesturus>
<jyrostudus>
<eubuptcrusaurus>
<aubosourus>
<llrus>
<snrusasras>
<alnsakgunus>
<hncucaurus>
<turhshusus>
<sualoauras>
<tbesraurus>
<aubmtiurus>

Epoch:4
<llrur>
<srostcruo>
<saurucauras>
<snrasaurus>
<surohsaurus>
<suairauros>
<tcgspaurus>
<auaivotieurus>
<rrvurjurasairus>
<eubesacrus>
<gbrantosaurus>
<ruryshurul>
<lcnucturis>
<laresaunus>
<xstasturus>

Epoch:5
<kyroseunus>
<eubuotdrusaur>
<gncuchurus>
<turcshurus>
<suaioauros>
<tceuraurus>
<auaiuosaurus>
<ruryscurul>
<lcmucnurds>
<larbsaurus>
<xssas>
<salrosuurus>
<anrapnnrus>
<curaneunanrus>
<pueutoras[...]<jupyter_text>HW Problem 3.2<jupyter_code>class Neural_Language_Model(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        self.i2h = nn.Linear(input_size, hidden_size)  # input to hidden matrix
        self.i2o = nn.Linear(3*hidden_size, output_size)  # input to output matrix
    
    def forward(self, x):
        
        x_1 = self.i2h(x[:, 0])
        x_2 = self.i2h(x[:, 1])
        x_3 = self.i2h(x[:, 2])
        x = torch.cat([x_1, x_2, x_3], dim = 1)
        x = torch.tanh(x)
        x = self.i2o(x)
        return x
      
def train_one_epoch(model, loss_fn, optimizer):
    model.train()
    for line_num, (x, y) in enumerate(trn_dl):
        loss = 0
        optimizer.zero_grad()
        x, y = x.to(device), y.to(device)
        if x.shape[1] <= 3:
          continue
        for i in range(x.shape[1] - 3):
            y_pred = model(x[:, i:i+3])
            loss += loss_fn(y_pred, y[:, i+3])
        loss.backward()
        optimizer.step()
        
        
def train(model, loss_fn, optimizer, dataset='dinos', epochs=1):
    for e in range(1, epochs+1):
        print('Epoch:{}'.format(e))
        train_one_epoch(model, loss_fn, optimizer)
        
model_2 = Neural_Language_Model(trn_ds.vocab_size, hidden_size, trn_ds.vocab_size).to(device)
loss_fn = nn.CrossEntropyLoss()
optimizer = optim.SGD(model_2.parameters(), lr=1e-2)
train(model_2, loss_fn, optimizer, epochs = 50)<jupyter_output>Epoch:1
Epoch:2
Epoch:3
Epoch:4
Epoch:5
Epoch:6
Epoch:7
Epoch:8
Epoch:9
Epoch:10
Epoch:11
Epoch:12
Epoch:13
Epoch:14
Epoch:15
Epoch:16
Epoch:17
Epoch:18
Epoch:19
Epoch:20
Epoch:21
Epoch:22
Epoch:23
Epoch:24
Epoch:25
Epoch:26
Epoch:27
Epoch:28
Epoch:29
Epoch:30
Epoch:31
Epoch:32
Epoch:33
Epoch:34
Epoch:35
Epoch:36
Epoch:37
Epoch:38
Epoch:39
Epoch:40
Epoch:41
Epoch:42
Epoch:43
Epoch:44
Epoch:45
Epoch:46
Epoch:47
Epoch:48
Epoch:49
Epoch:50<jupyter_text>Hyperparameters are:  - k  - hidden_size **Increasing k to ~5** would yield a bit better results. Hidden size equal to 40 or 60 approximately does not change anything   HW, Problem 3.3<jupyter_code>## generate a few name of dinosaurs

few = 5

for i in range(few):
  name_idx = sample(model)
  print_sample(name_idx)<jupyter_output><lbrcpnicaurus>
<busolmosaurus>
<tbmianodauris>
<lbrcpnicaurus>
<busolmosaurus>
{'fraction_non_alphanumeric': 0.14708246262658736, 'fraction_numerical': 0.029175373734126347, 'mean_word_length': 4.037651821862348, 'pattern_counts': {'":': 1, '<': 151, '<?xml version=': 0, '>': 147, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '332917', 'license': 'Apache-2.0', 'n_tokens_mistral': 5180, 'n_tokens_neox': 4926, 'n_words': 985, 'path': 'basic/NLP_simple_RNN.ipynb', 'repo_name': 'geangohn/nlp_projects', 'size': '12442'}