<jupyter_start><jupyter_text>Sequence to sequence model  Installation<jupyter_code># !pip install -q unpackai==0.1.8.9
# !pip install -q transformers
# !pip install -Uqq fastai
# !pip install -q datasets<jupyter_output><empty_output><jupyter_text>Imports<jupyter_code>from datasets import load_dataset
from fastai.text.all import *
from unpackai.nlp import *<jupyter_output><empty_output><jupyter_text>Data<jupyter_code># also en-ru, en-fr, en-it, etc
dataset = load_dataset("ted_iwlst2013", 'en-zh') 
dataset
entire = dataset['train']<jupyter_output><empty_output><jupyter_text>Split data to train/valid The dataset we donwloaded doesn't contain validation, let's split to make one<jupyter_code>splited = entire.train_test_split(test_size = .1,)
splited
train = splited['train']
valid = splited['test']
train[30]<jupyter_output><empty_output><jupyter_text>Tokenizer and pretrained model> Tokenizer will be a part the data pipeline, so let's download the pretrained tokenizer and pretrained model before we create a data block<jupyter_code>from transformers import (
    AutoTokenizer,
    AutoModelForCausalLM,
    AutoModel,
    EncoderDecoderModel
    )
# we find a English parsing encoder, as a pretrained bert is good at understanding english
# BERT is short for Bidirectional **Encoder** Representations from Transformers, which consists fully of encoder blocks
ENCODER_PRETRAINED = "bert-base-uncased"
# we find a Chinese writing model for decoder, as decoder is the part of the model that can write stuff
DECODER_PRETRAINED = "uer/gpt2-chinese-poem"<jupyter_output><empty_output><jupyter_text>Load pretrained models<jupyter_code># encoder = AutoModel.from_pretrained(ENCODER_PRETRAINED, proxies={"http":"bifrost:3128"})
# decoder = AutoModelForCausalLM.from_pretrained(DECODER_PRETRAINED, add_cross_attention=True,
#                                                proxies={"http":"bifrost:3128"})<jupyter_output><empty_output><jupyter_text>Load pretrained tokenizers<jupyter_code>encoder_tokenizer = AutoTokenizer.from_pretrained(ENCODER_PRETRAINED)
decoder_tokenizer = AutoTokenizer.from_pretrained(DECODER_PRETRAINED)
# ENCODER_MAX_LEN = encoder.config.max_position_embeddings
# DECODER_MAX_LEN = decoder.config.max_position_embeddings

# ENCODER_MAX_LEN, DECODER_MAX_LEN
tokenizer_configuration = dict(
    return_tensors='pt',
    max_length=128,
    padding="max_length",
    truncation=True,
)<jupyter_output><empty_output><jupyter_text>Datablock You can try to change get_x and get_y to fetch the opposite language, you can have model that will train in other direction<jupyter_code>dblock = DataBlock(
    get_x = lambda x:x['translation']['en'],
    get_y = lambda x:x['translation']['zh'],
                   )<jupyter_output><empty_output><jupyter_text>Datasets<jupyter_code>dsets = dblock.datasets(train, valid)<jupyter_output>Collecting items from Dataset({
    features: ['id', 'translation'],
    num_rows: 139121
})
Found 139121 items
2 datasets of sizes 111297,27824
Setting up Pipeline: <lambda>
Setting up Pipeline: <lambda><jupyter_text>Preview a row of dataset, which returns an English sentence vs Chinese sentence pair<jupyter_code>dsets.train[6]<jupyter_output><empty_output><jupyter_text>Dataloaders * Dataset deals data on **row** level, eg. a pair of sentence* Dataloader deals data on **batch** level, eg. a batch of tensor, consists of $n$ rows of data, where $n$ is the batch sizeWe usually call this process of: rows of raw data => pytorch tensor: collateHere we build a collate function that will transform rows of 2 sentences into tokenized/numericalize tensors using given tokenizers<jupyter_code>def batch_tokenize_collate(data):
    input_seq, target_seq = list(zip(*data))
    # tokenizing for encoder
    x = encoder_tokenizer(list(input_seq),**tokenizer_configuration)
    input_ids = x.input_ids
    attention_mask = x.attention_mask

    # tokenizing for decoder
    y = decoder_tokenizer(list(target_seq),**tokenizer_configuration)
    decoder_input_ids = y.input_ids
    decoder_attention_mask = y.attention_mask
    
    # return the output in format of (x, y), y
    # As the model forward pipeline will need both x, and y for training
    # and will output loss directly, but fastai learner require x,y formality in datablock
    return (input_ids, attention_mask, decoder_input_ids, decoder_attention_mask),\
        ( decoder_input_ids, decoder_attention_mask)
dls = dsets.dataloaders(bs=64,
                        create_batch=batch_tokenize_collate)
(input_ids, attention_mask, decoder_input_ids, decoder_attention_mask),(
    decoder_input_ids, decoder_attention_mask)=dls.one_batch()<jupyter_output><empty_output><jupyter_text>Reconstruct a batch of data<jupyter_code>for e,c in zip(encoder_tokenizer.batch_decode(input_ids, skip_special_tokens=True),
decoder_tokenizer.batch_decode(decoder_input_ids, skip_special_tokens=True)):
    print(e)
    print(c)
    print("="*10)<jupyter_output>well, there are lots of reasons.
这 有 很 多 原 因
==========
even if you're logged out, one engineer told me, there are 57 signals that google looks at - - everything from what kind of computer you're on to what kind of browser you're using to where you're located - - that it uses to personally tailor your query results.
一 位 工 程 师 告 诉 我 ， 即 使 你 退 出 帐 号 ， 还 会 有 57 种 信 号 可 供 谷 歌 参 考 - - 几 乎 所 有 的 信 息 ： 从 你 使 用 的 电 脑 型 号 到 你 用 的 浏 览 器 到 你 所 在 的 位 置 - - 谷 歌 利 用 这 些 为 你 定 制 出 个 性 化 的 查 询 结 果 。
==========
but it has to be done together.
这 必 须 由 你 们 共 同 参 与 完 成 。
==========
the whole business is run on sustainable energy.
整 间 餐 厅 都 是 使 用 可 再 生 能 源 ，
==========
he said, " you know what, one of the items on the checklist is lack of remorse, but another item on the checklist is cunning, manipulative.
他 说 ， 你 知 道 吗 ， 检 核 表 上 有 一 项 是 缺 乏 懊 悔 但 另 一 项 却 是 狡 猾 ， 且 控 制 欲 强
==========
and one of the most common faces on something faced with beauty, something stupefyingly delicious, is what i call the omg.
面[...]<jupyter_text>Model We create a seq2seq model by using pretrained encoder + pretrained decoder<jupyter_code># loading pretrained model
encoder_decoder = EncoderDecoderModel.from_encoder_decoder_pretrained(
    encoder_pretrained_model_name_or_path=ENCODER_PRETRAINED,
    decoder_pretrained_model_name_or_path=DECODER_PRETRAINED,
)

class Seq2SeqTrain(nn.Module):
    def __init__(self, encoder_decoder):
        super().__init__()
        self.encoder_decoder = encoder_decoder
        
    def forward(self, batch):
        input_ids, attention_mask, decoder_input_ids, decoder_attention_mask = batch

        return self.encoder_decoder(
                input_ids = input_ids,
                attention_mask = attention_mask,
                labels = decoder_input_ids,
                decoder_input_ids=decoder_input_ids,
            )
model = Seq2SeqTrain(encoder_decoder)<jupyter_output><empty_output><jupyter_text>Training<jupyter_code>learn = Learner(dls, model, loss_func=lambda output, target:output.loss,)
learn.fit(10, lr=5e-5)<jupyter_output><empty_output><jupyter_text>Inference<jupyter_code>model = model.cpu()
model = model.eval()
def inference(text, starter=''):
    tk_kwargs = dict(truncation=True, max_length=128, padding="max_length",
                     return_tensors='pt')
    inputs = encoder_tokenizer([text,],**tk_kwargs)
    with torch.no_grad():
        return decoder_tokenizer.batch_decode(
            model.encoder_decoder.generate(
            inputs.input_ids,
            attention_mask=inputs.attention_mask,
            num_beams=3,
            bos_token_id=101,
        ),
                                              skip_special_tokens=False)
inference(
    'And now my mission to control and predict had turned up the answer that the way to live is with vulnerability and to stop controlling and predicting.')
inference("I'm going to enjoy this movie")
inference("Why does this matter")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08717624763108023, 'fraction_numerical': 0.008970309538850285, 'mean_word_length': 4.459310344827586, 'pattern_counts': {'":': 2, '<': 61, '<?xml version=': 0, '>': 63, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 24, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '355712', 'license': 'MIT', 'n_tokens_mistral': 2776, 'n_tokens_neox': 2652, 'n_words': 764, 'path': 'examples/nlp_seq2seq_en_to_zh_translation.ipynb', 'repo_name': 'vtecftwy/unpackai', 'size': '7915'}