<story><title>Better and Faster Large Language Models via Multi-Token Prediction</title><url>https://arxiv.org/abs/2404.19737</url></story><parent_chain><item><author>Xcelerate</author><text>Do LLMs not consider the probability distribution over all combinations of tokens up to a certain output length with regard to sequence prediction? I assumed they did that already.&lt;p&gt;If they don’t, I’m amazed they work as well as they do. Consider 2-bit sequence prediction with the following possible outcomes and associated probabilities:&lt;p&gt;&lt;pre&gt;&lt;code&gt;    00: p=0.36
    01: p=0.04
    10: p=0.30
    11: p=0.30
&lt;/code&gt;&lt;/pre&gt;
So the most likely 2-bit sequence is 00. But on the basis of predicting the next token (bit) alone, we have:&lt;p&gt;&lt;pre&gt;&lt;code&gt;     0: p=0.40
     1: p=0.60
&lt;/code&gt;&lt;/pre&gt;
which suggests that 1 is the next bit and leads to a suboptimal starting point for predicting the bit after that. The error is even more prominent with longer sequences as the joint probability distribution becomes more unfactorizable into marginal distributions (as I would expect any minimal algorithmic description of real-world data to be).&lt;p&gt;Edit: now that I think about this a bit more, a cool research project that would be really simple to carry out might be to modify the cross-entropy loss function to consider &lt;i&gt;only&lt;/i&gt; the nth future token in the text training data, and then plot LLM performance vs n, assuming that for all current LLM models we just have n=1.&lt;p&gt;My hypothesis is that you can mostly bypass all of the resource blow-up involved in predicting the joint probability distribution over the next 1 through n tokens (which scales as x^n) by just predicting the nth token directly, since doing so would implicitly require a better data model (at least for human-generated text; this wouldn’t be the case for all types of data).</text></item></parent_chain><comment><author>elcomet</author><text>I think you&amp;#x27;re not looking at this from the right perspective. A LLM is designed to sample text, that follows the training distribution. It is not designed to tell you the &amp;quot;most likely&amp;quot; text that follows, and we don&amp;#x27;t actually want that. This would mean you have no diversity in your outputs.&lt;p&gt;In your example, sampling a 0 in 40% of cases and a 1 in 60% of cases does make sense for chat applications.&lt;p&gt;For applications where we &lt;i&gt;do&lt;/i&gt; care about the most likely sentence (e.g. question answering), then beam search helps, as others have mentioned.&lt;p&gt;Another thing to consider is that the model can &amp;quot;look ahead&amp;quot; and precompute what the future tokens might be. And it can then use this to predict the current token. In fact, some work have been investigating this, such as [1].&lt;p&gt;And a final note, predicting one token at a time is what we are doing as humans when we speak, so clearly it is not a &lt;i&gt;wrong&lt;/i&gt; approach. We are doing this &amp;quot;look ahead&amp;quot; in our mind before speaking.&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;arxiv.org&amp;#x2F;abs&amp;#x2F;2404.00859&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;arxiv.org&amp;#x2F;abs&amp;#x2F;2404.00859&lt;/a&gt;</text></comment>
<story><title>Better and Faster Large Language Models via Multi-Token Prediction</title><url>https://arxiv.org/abs/2404.19737</url></story><parent_chain><item><author>Xcelerate</author><text>Do LLMs not consider the probability distribution over all combinations of tokens up to a certain output length with regard to sequence prediction? I assumed they did that already.&lt;p&gt;If they don’t, I’m amazed they work as well as they do. Consider 2-bit sequence prediction with the following possible outcomes and associated probabilities:&lt;p&gt;&lt;pre&gt;&lt;code&gt;    00: p=0.36
    01: p=0.04
    10: p=0.30
    11: p=0.30
&lt;/code&gt;&lt;/pre&gt;
So the most likely 2-bit sequence is 00. But on the basis of predicting the next token (bit) alone, we have:&lt;p&gt;&lt;pre&gt;&lt;code&gt;     0: p=0.40
     1: p=0.60
&lt;/code&gt;&lt;/pre&gt;
which suggests that 1 is the next bit and leads to a suboptimal starting point for predicting the bit after that. The error is even more prominent with longer sequences as the joint probability distribution becomes more unfactorizable into marginal distributions (as I would expect any minimal algorithmic description of real-world data to be).&lt;p&gt;Edit: now that I think about this a bit more, a cool research project that would be really simple to carry out might be to modify the cross-entropy loss function to consider &lt;i&gt;only&lt;/i&gt; the nth future token in the text training data, and then plot LLM performance vs n, assuming that for all current LLM models we just have n=1.&lt;p&gt;My hypothesis is that you can mostly bypass all of the resource blow-up involved in predicting the joint probability distribution over the next 1 through n tokens (which scales as x^n) by just predicting the nth token directly, since doing so would implicitly require a better data model (at least for human-generated text; this wouldn’t be the case for all types of data).</text></item></parent_chain><comment><author>faabian</author><text>Language models factor the joint probability p(y, x) as p(y, x) = p(y|x) p(x) which is exact. I.e. if you train a language model on your distribution &lt;i&gt;and sample with temperature 1&lt;/i&gt;, you will get the exact same distribution out. If you sample at lower temperature or even greedily, evidently, you will get other distributions.</text></comment>