<jupyter_start><jupyter_text>German to English translatorBased on these tutorials: * YouTube: https://www.youtube.com/watch?v=EoGUlvhRYpk   * Code here: https://github.com/aladdinpersson/Machine-Learning-Collection/blob/master/ML/Pytorch/more_advanced/Seq2Seq/seq2seq.py   * Utilities here: https://github.com/aladdinpersson/Machine-Learning-Collection/blob/master/ML/Pytorch/more_advanced/Seq2Seq/utils.py * PyTorch documentation: https://pytorch.org/tutorials/beginner/torchtext_translation_tutorial.html   **TODO**: * Sort training data by length (putting shorter sentences at beginning makes it easier to train). * During training, periodically print out a translation of a sentence, to see the translation improve with epoch. (Add a callback argument to do this?) * detokenizing (converting back to text string) seq2seq models that were tokenized with spacy: https://stackoverflow.com/questions/50330455/how-to-detokenize-spacy-text-without-doc-context  **Other notes**Tips for profiling GPU: * https://developer.nvidia.com/blog/profiling-and-optimizing-deep-neural-networks-with-dlprof-and-pyprof/   * `nvidia-smi dmon`   Improving performance: * https://towardsdatascience.com/7-tips-for-squeezing-maximum-performance-from-pytorch-ca4a40951259   * DataLoader(num_workers=4, pin_memory=True)   * Don't unnecssarily move data of GPU (.cpu(), .item(), .numpy()). Clear up attached computational graph with .detach() instead   * Construct tensors directly on GPU: instead of `tensor.rand(5).cuda()`, use `tensor.rand(5, device=torch.device("cuda:0"))`<jupyter_code>%config InlineBackend.figure_format = "retina"
%load_ext autoreload
%autoreload 2
from IPython.core.display import display, HTML
display(HTML("<style>.container { width:95% !important; }</style>"))

import random
import sys
from time import time
from typing import List

import pandas as pd
import spacy
import torch
from torch import LongTensor
import torch.nn as nn
import torch.optim as optim
from torchtext.data.metrics import bleu_score
from torchtext.data.utils import get_tokenizer
from torchtext.utils import download_from_url, extract_archive

sys.path.insert(0, "../../bentorched")
from utils import print_torch_setup
from text.text import TextPreprocessorSeq2Seq, FitModelSeq2Seq, indices_to_text, indices_to_tokens, evaluate_bleu_score
from text.encdec import Encoder, Decoder, Seq2Seq
#from text import TextPreprocessorSeq2Seq, Encoder, Decoder, Seq2Seq, indices_to_text, FitModelSeq2Seq, evaluate_bleu_score, list_of_indices_to_text

print_torch_setup()<jupyter_output><empty_output><jupyter_text>Download and load data<jupyter_code>def load_corpus(filename: str) -> List[str]:
    """
    Read lines of corpus.
    """
    with open(filename, encoding="utf8") as f:
        corpus = f.readlines()
        
    return [t.strip() for t in corpus]
url_base = 'https://raw.githubusercontent.com/multi30k/dataset/master/data/task1/raw/'
urls = ('train.de.gz', 'train.en.gz', 'val.de.gz', 'val.en.gz', 'test_2016_flickr.de.gz', 'test_2016_flickr.en.gz')

# Download and extract all the files
filenames = [extract_archive(download_from_url(url_base+url, root="data"))[0] for url in urls]

filenames
df_train = pd.DataFrame({"de": load_corpus("data/train.de"), "en": load_corpus("data/train.en")})
df_val = pd.DataFrame({"de": load_corpus("data/val.de"), "en": load_corpus("data/val.en")})
df_test = pd.DataFrame({"de": load_corpus("data/test_2016_flickr.de"), "en": load_corpus("data/test_2016_flickr.en")})

display(df_train.head(5))
display(df_val.head(5))
display(df_test.head(5))<jupyter_output><empty_output><jupyter_text>Preprocess / create DataLoader objects<jupyter_code># Load Spacy's German and English tokenizers
tokenizer_de = get_tokenizer('spacy', language='de')
tokenizer_en = get_tokenizer('spacy', language='en')
batch_size = 64

preprocessor = TextPreprocessorSeq2Seq(tokenizer_de, tokenizer_en, batch_size=batch_size, max_size=10000, min_freq=2)
preprocessor.fit(df_train["de"], df_train["en"])

dataloader_train = preprocessor.generate_dataloader(df_train["de"], df_train["en"])
dataloader_val = preprocessor.generate_dataloader(df_val["de"], df_val["en"])
dataloader_test = preprocessor.generate_dataloader(df_test["de"], df_test["en"])
# Get batch of data
x, y = next(iter(dataloader_test))


print(len(preprocessor.vocab_in))
print(len(preprocessor.vocab_out))
print(preprocessor.vocab_in.itos[:15])
print(preprocessor.vocab_out.itos[:15])
print(x.shape)
#print(x[:, :5])
print(y.shape)
#print(y[:, :5])

# TODO: there should be no <unk> if you are looking at the training set and you didn't restrict the vocabulary size
batch_in = indices_to_text(x, preprocessor.vocab_in, ignore_tokens=["<pad>", "<bos>", "<eos>"])
batch_out = indices_to_text(y, preprocessor.vocab_out, ignore_tokens=["<pad>", "<bos>", "<eos>"])
for i in range(10):
    print(batch_in[i])
    print(batch_out[i])
    print()<jupyter_output>8015
6191
['<unk>', '<pad>', '<bos>', '<eos>', '.', 'Ein', 'einem', 'in', ',', 'und', 'mit', 'auf', 'Mann', 'einer', 'Eine']
['<unk>', '<pad>', '<bos>', '<eos>', 'a', '.', 'A', 'in', 'the', 'on', 'is', 'and', 'man', 'of', 'with']
torch.Size([28, 64])
torch.Size([28, 64])
Eine <unk> und zwei Kinder machen vor ihrem <unk> Gartenarbeit wie den Rasen mit einer <unk> zu bearbeiten und einen Baum zu pflanzen .
A father - figure and two children outside their home doing yard work such as using a hoe on the grass and planting a tree .

Zwei Radfahrer mit Helmen fahren an ein paar <unk> Feldern vorbei .
Two bicyclists with helmets on are riding past some empty country fields .

Zwei Silhouetten paddeln in einem Kanu auf dem Ozean bei Sonnenuntergang .
Two silhouetted people paddle a canoe on the ocean during sunset .

Ein Mann , der aus einer gelben Tasse <unk> , umgeben von Menschen auf einer geschäftigen Veranstaltung .
A man drinking out of a yellow cup around people at a busy event

Leute s[...]<jupyter_text>Define and train model<jupyter_code>device = "cuda"

# --- Model parameters ---

n_tokens_enc = len(preprocessor.vocab_in)
n_tokens_dec = len(preprocessor.vocab_out)

n_embedding_enc = 300
n_embedding_dec = 300

n_hidden_enc = 1024
n_hidden_dec = 1024

n_layers_enc = 2
n_layers_dec = 2

dropout_enc = 0.5
dropout_dec = 0.5

# --- Training parameters ---

# n_epochs = 100
# learning_rate = 0.001
# teacher_force_ratio = 0.5
# # Gradient clipping
# max_norm = 1.0

n_epochs = 10
learning_rate = 0.001
teacher_forcing_ratio = 0.5
# Gradient clipping
max_norm = 1.0

encoder_net = Encoder(n_tokens_enc, n_embedding_enc, n_hidden_enc, n_layers_enc, dropout_enc).to(device)
decoder_net = Decoder(n_tokens_dec, n_embedding_dec, n_hidden_dec, n_layers_dec, dropout_dec).to(device)
model = Seq2Seq(encoder_net, decoder_net, device).to(device)

# Calculate loss for each element in sequence (compare true element to predicted element of target sequence).
# Ignore the elements where the true element is "<pad>".
pad_idx = preprocessor.vocab_out.stoi["<pad>"]
loss_func = nn.CrossEntropyLoss(ignore_index=pad_idx)

optimizer = optim.Adam(model.parameters(), lr=learning_rate)
scheduler = None
fm = FitModelSeq2Seq(
    dataloader_train, 
    model,
    loss_func,
    optimizer,
    n_epochs,
    max_norm = max_norm,
    teacher_forcing_ratio = teacher_forcing_ratio,
    show_interval = 100,
    n_score = 5,
    vocab_out=preprocessor.vocab_out,
    scoring_func = evaluate_bleu_score,
    device = "cuda",
    dataloader_val = dataloader_val,
    scheduler = scheduler
)
fm.fit()
torch.save(model.to("cpu"), "ger2eng_model.pt")<jupyter_output><empty_output><jupyter_text>Evaluation  Compare translations to ground truth<jupyter_code>model = torch.load("ger2eng_model.pt")

x, y = next(iter(dataloader_test))

ignore_tokens = ["<pad>", "<bos>", "<eos>", "<unk>"]

# Don't use ground truth when making predictions (teacher_forcing_ratio=0)
test_pred = model(x, y, teacher_forcing_ratio=0.0)
test_tokens = test_pred.argmax(dim=2)

x_text = indices_to_text(x, preprocessor.vocab_in, ignore_tokens=ignore_tokens)
y_text = indices_to_text(y, preprocessor.vocab_out, ignore_tokens=ignore_tokens)
y_text_pred = indices_to_text(test_tokens, preprocessor.vocab_out, ignore_tokens=ignore_tokens)


for i in range(10):
    print(f"German:\t\t\t{x_text[i]}")
    print(f"English ground truth:\t{y_text[i]}")
    print(f"English translation:\t{y_text_pred[i]}")
    print()<jupyter_output>German:			Eine und zwei Kinder machen vor ihrem Gartenarbeit wie den Rasen mit einer zu bearbeiten und einen Baum zu pflanzen .
English ground truth:	A father - figure and two children outside their home doing yard work such as using a hoe on the grass and planting a tree .
English translation:	A woman and her children and kids throwing martial arts - jump to the the of a and frozen - out of the the .

German:			Ein hell gekleideter Mann fotografiert eine Gruppe von Männern in dunklen Anzügen und mit Hüten , die um eine Frau in einem Kleid herum stehen .
English ground truth:	A man in light colored clothing photographs a group of men wearing dark suits and hats standing around a woman dressed in a gown .
English translation:	A man dressed in traditional clothing is dancing in a group of men in formal attire , and a woman with a woman and a woman .

German:			Ein kleines Mädchen in einem schwarzen Badeanzug hält eine Schaufel am Strand .
English ground truth:	A little girl wearing a bla[...]<jupyter_text>Beam search<jupyter_code>preprocessor.vocab_out.stoi["<bos>"], preprocessor.vocab_out.stoi["<eos>"]
from queue import PriorityQueue
import operator

# SOS_token = 2
SOS_token = 0
EOS_token = 3
MAX_LENGTH = 50
class BeamSearchNode(object):
    def __init__(self, hiddenstate, cellstate, previousNode, wordId, logProb, length):
        """
        Args:
            hiddenstate:
            previousNode:
            wordId: Index of token
            logProb: Log probability (Output of decoder after applying logsoftmax)
            length: f
        """
        self.h = hiddenstate
        self.c = cellstate
        self.prevNode = previousNode
        self.wordid = wordId
        self.logp = logProb
        self.leng = length

    def eval(self, alpha=1.0):
        reward = 0
        # Add here a function for shaping a reward

        return self.logp / float(self.leng - 1 + 1e-6) + alpha * reward



def beam_decode(target_tensor, decoder_hiddens, decoder_cells, decoder, device, beam_width=10, topk=1):
    """
    TODO: shape = [B,T] is the opposite order of what I have
    
    Args:
        target_tensor: target indexes tensor of shape [B, T] where B is the batch size and T is the maximum length of the output sentence
        decoder_hidden: input tensor of shape [1, B, H] for start of the decoding
        decoder_cells:
        beam_width:
        topk: how many sentence do you want to generate
    
    Returns: 
        decoded_batch
    """
    decoded_batch = []

    # for idx in range(target_tensor.size(0)):
    for idx in range(target_tensor.size(1)):
        
        # --- Beam search for a single sentence in batch ---
        # decoder_hiddens/cells.shape = (n_layers, N, n_hidden)
        # encoder_output.shape = ()
        decoder_hidden = decoder_hiddens[:, idx, :].unsqueeze(dim=1).contiguous() # GRU/RNN
        decoder_cell = decoder_cells[:, idx, :].unsqueeze(dim=1).contiguous() # For LSTM
        # decoder_hidden/cell.shape = (n_layers, 1, n_hidden)

        # Start with the start of the sentence token
        # TODO: Change 0 to the actual "<bos>" token value
        decoder_input = torch.LongTensor([SOS_token]).to(device)

        # Number of sentences to generate
        endnodes = []
        number_required = min((topk + 1), topk - len(endnodes))

        # starting node -  hidden vector, previous node, word id, logp, length
        node = BeamSearchNode(decoder_hidden, decoder_cell, None, decoder_input, 0, 1)
        
        # Priority Queue will sort the nodes by the first value (-logp) in ascending order. 
        # So, the highest value of logp will be selected first when nodes.get() is called.
        nodes = PriorityQueue()

        # start the queue
        nodes.put((-node.eval(), node))
        qsize = 1

        # start beam search
        while True:
            # give up when decoding takes too long
            if qsize > 2000: break

            # fetch the best node
            score, n = nodes.get()
            decoder_input = n.wordid
            decoder_hidden = n.h
            decoder_cell = n.c

            if n.wordid.item() == EOS_token and n.prevNode != None:
                endnodes.append((score, n))
                # if we reached maximum # of sentences required
                if len(endnodes) >= number_required:
                    break
                else:
                    continue

            # decode for one step using decoder
            #print("Before:", decoder_input.shape, decoder_hidden.shape, decoder_cell.shape)
            decoder_output, decoder_hidden, decoder_cell = decoder(decoder_input, decoder_hidden, decoder_cell)
            #print("After:", decoder_output.shape, decoder_hidden.shape, decoder_cell.shape)
            
            # decoder_output.shape = (1, n_vocab)
            
            # PUT HERE REAL BEAM SEARCH OF TOP
            log_prob, indexes = torch.topk(decoder_output.log_softmax(dim=1), beam_width)
            #print(log_prob, indexes)

            # Create a new node for each of the topk, connected to the previous node n
            nextnodes = []
            for new_k in range(beam_width):
                # TODO: should write indexes[0, new_k]
                # decoded_t = indexes[0][new_k].view(1, -1)
                decoded_t = indexes[0][new_k].view(1) # shape=(1,)
                log_p = log_prob[0][new_k].item() # scalar
                node = BeamSearchNode(decoder_hidden, decoder_cell, n, decoded_t, n.logp + log_p, n.leng + 1)
                score = -node.eval()
                nextnodes.append((score, node))

            # put them into queue
            for i in range(len(nextnodes)):
                score, nn = nextnodes[i]
                nodes.put((score, nn))
                # increase qsize
            qsize += len(nextnodes) - 1

        #print(qsize)
        
        # choose nbest paths, back trace them
        if len(endnodes) == 0:
            # Get the topk nodes with the lowest scores (highest logp)
            endnodes = [nodes.get() for _ in range(topk)]
        
        
        utterances = []
        for score, n in sorted(endnodes, key=operator.itemgetter(0)):
            utterance = []
            utterance.append(n.wordid.item())
            # back trace
            while n.prevNode != None:
                n = n.prevNode
                utterance.append(n.wordid.item())
            
            # Flip order of utterances (words) to go first to last word
            utterance = utterance[::-1]
            
            utterances.append(utterance)
            utterances = torch.LongTensor(utterances).to(device).squeeze()
            
        decoded_batch.append(utterances)

    return decoded_batch
source, target = next(iter(dataloader_test))

source = source.to(device)
target = target.to(device)
model = model.to(device)
encoder = model.encoder
decoder = model.decoder


print(source.shape, target.shape, model.device, model.training)


# --- get initial hidden states for decoder ---

batch_size = source.shape[1]
# TODO: If your are making predictions and you don't have a ground truth target,
#       you'll need another way of choosing target_len.
target_len = target.shape[0]

# Encode source sentence
hidden, cell = encoder(source)

print(hidden.shape, cell.shape)
# target_tensor = target
# decoder_hiddens = hidden
# decoder_cells = cell

# decoded_batch = beam_decode(target_tensor, decoder_hiddens, decoder_cells, decoder, device, beam_width=10, topk=1)
# decoded_batch[0]
model = torch.load("ger2eng_model.pt")

source, target = next(iter(dataloader_test))

ignore_tokens = ["<pad>", "<bos>", "<eos>", "<unk>"]

# Don't use ground truth when making predictions (teacher_forcing_ratio=0)
test_pred = model(source, target, teacher_forcing_ratio=0.0)
test_tokens = test_pred.argmax(dim=2)

x_text = indices_to_text(source, preprocessor.vocab_in, ignore_tokens=ignore_tokens)
y_text = indices_to_text(target, preprocessor.vocab_out, ignore_tokens=ignore_tokens)



source = source.to(device)
target = target.to(device)
model = model.to(device)
encoder = model.encoder
decoder = model.decoder

hidden, cell = encoder(source)
target_tensor = target
decoder_hiddens = hidden
decoder_cells = cell

decoded_batch = beam_decode(target_tensor, decoder_hiddens, decoder_cells, decoder, device, beam_width=10, topk=1)
y_text_pred = list_of_indices_to_text(decoded_batch, preprocessor.vocab_out, ignore_tokens=ignore_tokens)


for i in range(10):
    print(f"German:\t\t\t{x_text[i]}")
    print(f"English ground truth:\t{y_text[i]}")
    print(f"English translation:\t{y_text_pred[i]}")
    print()
decoded_batch[9]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10994672443065394, 'fraction_numerical': 0.012235817575083427, 'mean_word_length': 3.6633906633906634, 'pattern_counts': {'":': 6, '<': 58, '<?xml version=': 0, '>': 61, 'https://': 8, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '56306', 'license': 'MIT', 'n_tokens_mistral': 5774, 'n_tokens_neox': 5491, 'n_words': 1488, 'path': 'examples/seq2seq/ger2eng_enc_dec.ipynb', 'repo_name': 'benjaminlackey/bentorched', 'size': '17081'}