<jupyter_start><jupyter_text>Batch translation<jupyter_code>def translate(tokenizer, model, text, num_outputs):
    """Use beam search to get a reasonable translation of 'text'"""
    batch = tokenizer.prepare_translation_batch([text]).to(model.device)
    num_beams = num_outputs
    translated = model.generate(**batch, num_beams=num_beams, num_return_sequences=num_outputs)
    return [tokenizer.decode(t, skip_special_tokens=True, clean_up_tokenization_spaces=False) for t in translated]

translate(en_ROMANCE_tokenizer, en_ROMANCE, ">>es<< I ran to the store.", 5)<jupyter_output><empty_output><jupyter_text>Incremental translation English to token ids<jupyter_code>tokenizer = en_ROMANCE_tokenizer
model = en_ROMANCE
english = ">>es<< I ran to the store."
input_ids = tokenizer.encode(english, return_tensors="pt").to(device)<jupyter_output><empty_output><jupyter_text>English token ids to "concept space"<jupyter_code>batch = tokenizer.prepare_translation_batch([english]).to(device)
english_encoded = model.get_encoder()(**batch)<jupyter_output><empty_output><jupyter_text>What we have so far, as token ids:<jupyter_code>decoder_start_token = model.config.decoder_start_token_id
decoder_start_token
partial_decode = torch.LongTensor([decoder_start_token]).to(device).unsqueeze(0)<jupyter_output><empty_output><jupyter_text>Ask the model for what comes next<jupyter_code># one-time setup
past = (english_encoded, None)
model_inputs = model.prepare_inputs_for_generation(
    partial_decode, past=past, attention_mask=batch['attention_mask'], use_cache=model.config.use_cache
)
with torch.no_grad():
    model_outputs = model(**model_inputs)

next_token_logits = model_outputs[0][:, -1, :]
past = model_outputs[1]<jupyter_output><empty_output><jupyter_text>Find the most likely<jupyter_code>next_token_to_add = next_token_logits[0].argmax()<jupyter_output><empty_output><jupyter_text>Notice that this is a one-item tensor (zero-dim)... so it can't concat with anything:<jupyter_code>next_token_to_add.shape<jupyter_output><empty_output><jupyter_text>But this is the kind of shape that it needs to have:<jupyter_code>partial_decode.shape<jupyter_output><empty_output><jupyter_text>So here's how to give it that extra dimension:<jupyter_code>next_token_to_add.unsqueeze(0).unsqueeze(0)<jupyter_output><empty_output><jupyter_text>Ok, those shapes align, so we can concatenate them.<jupyter_code>partial_decode = torch.cat((partial_decode, next_token_to_add.unsqueeze(0).unsqueeze(0)), -1)
partial_decode<jupyter_output><empty_output><jupyter_text>Now we have a new output, with one additional token:<jupyter_code>tokenizer.convert_ids_to_tokens(partial_decode[0])<jupyter_output><empty_output><jupyter_text>We can now do all that again, to ask for the next thing after that token.<jupyter_code>model_inputs = model.prepare_inputs_for_generation(
    partial_decode, past=past, attention_mask=batch['attention_mask'], use_cache=model.config.use_cache
)
with torch.no_grad():
    model_outputs = model(**model_inputs)

next_token_logits = model_outputs[0][:, -1, :]
past = model_outputs[1]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11089743589743589, 'fraction_numerical': 0.004807692307692308, 'mean_word_length': 9.403333333333334, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 53, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '635164', 'license': 'MIT', 'n_tokens_mistral': 1100, 'n_tokens_neox': 1066, 'n_words': 190, 'path': 'demo_marian.ipynb', 'repo_name': 'adomaatobrah/simplewebpage', 'size': '3120'}