<story><title>Lossless Acceleration of LLM via Adaptive N-Gram Parallel Decoding</title><url>https://arxiv.org/abs/2404.08698</url></story><parent_chain><item><author>SonOfLilit</author><text>Good idea. Generate 8 tokens with a small model, then give them to a large model as one batch (much faster) and if tokens 1..3 are in agreement but 4..8 nut, take only 1..4 and start again. Most tokens are easy to guess so you&amp;#x27;ll get x3.5 gains.&lt;p&gt;I do have a feeling of dejavú, like I&amp;#x27;ve seen this before on hn.</text></item></parent_chain><comment><author>milkey_mouse</author><text>&amp;gt; I do have a feeling of dejavú, like I&amp;#x27;ve seen this before on hn.&lt;p&gt;You&amp;#x27;re either thinking of speculative encoding more generally, or Medusa: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;arxiv.org&amp;#x2F;abs&amp;#x2F;2401.10774&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;arxiv.org&amp;#x2F;abs&amp;#x2F;2401.10774&lt;/a&gt;</text></comment>
<story><title>Lossless Acceleration of LLM via Adaptive N-Gram Parallel Decoding</title><url>https://arxiv.org/abs/2404.08698</url></story><parent_chain><item><author>SonOfLilit</author><text>Good idea. Generate 8 tokens with a small model, then give them to a large model as one batch (much faster) and if tokens 1..3 are in agreement but 4..8 nut, take only 1..4 and start again. Most tokens are easy to guess so you&amp;#x27;ll get x3.5 gains.&lt;p&gt;I do have a feeling of dejavú, like I&amp;#x27;ve seen this before on hn.</text></item></parent_chain><comment><author>johntb86</author><text>They mention previous work on speculative decoding using similar techniques, but &amp;quot;ANPD dynamically generates draft outputs
via an adaptive N-gram module using real-time
statistics, after which the drafts are verified by the
LLM. This characteristic is exactly the difference
between ANPD and the previous speculative decoding methods.&amp;quot;</text></comment>