<jupyter_start><jupyter_text>Understanding Incremental Decoding> Walks through some of the performance hacks in BartForConditionalGeneration- toc: true - badges: true- comments: true- categories: [jupyter]- image: images/chart-preview.png Here is a really slow (pseudocode) way to greedily generate summaries:    ```pythonoutput_tokens = [bos]while not done:     encoder_hidden_state = model.encoder(article_input_ids)     logits = model.decoder(encoder_hidden_state, output_tokens)     next_word = logits.argmax()     output_tokens.append(next_word)     if next_word == eos: break``` Let's say `N = len(article_input_ids)` and `M = len(output_tokens)` when we return. Let's also ignore the cost of encoder-decoder attention.Then the complexity of this approach is roughly- $(N \cdot M)$ (we call the encoder `M` times on `N` tokens)-  $+ \sum_{m=1}^{M} m = \frac{M(M+1)}{2}$   (we call the decoder `M` times on 1 token then 2 tokens, all the way to M tokens).- Total: $N \cdot M  + \frac{M(M+1)}{2}$ Let's say we are generating a 100 token summary of a 1024 token article, then we have to "process"$N \cdot M  + \frac{M(M+1)}{2} =  107,450$ tokens<jupyter_code>#collapse-hide
# show work
N,M=1024, 100
complexity_simple = int(N*M + ((M*(M+1))/2))
msg = "To generate a 100 token summary of a 1024 token article using this approach, we have to process {:,} tokens."
#display(Markdown(msg.format(complexity_simple)))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12465373961218837, 'fraction_numerical': 0.02631578947368421, 'mean_word_length': 4.995850622406639, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '227812', 'license': 'Apache-2.0', 'n_tokens_mistral': 503, 'n_tokens_neox': 459, 'n_words': 164, 'path': '_notebooks/2020-03-11-Decoding.ipynb', 'repo_name': 'goodhamgupta/blog_v2', 'size': '1444'}