<story><title>Unlimiformer: Long-Range Transformers with Unlimited Length Input</title><url>https://arxiv.org/abs/2305.01625</url></story><parent_chain><item><author>mxwsn</author><text>1. This is not exact attention, but an approximation of it. Specifically, they use k-nearest neighbors to retrieve the top-k most similar tokens, out of an &amp;quot;unlimited-length input&amp;quot; say of size N, where k &amp;lt;&amp;lt; N.&lt;p&gt;2. This idea is quite similar to retrieval transformers and Hopfield networks which have been known and published for several years now. It&amp;#x27;s not really that novel.&lt;p&gt;3. Due to the preceding points, the title can easily mislead people. It&amp;#x27;s not really a conventional transformer, and it&amp;#x27;s not a breakthrough.&lt;p&gt;4. This paper is a preprint and not peer-reviewed.&lt;p&gt;&amp;quot;I generally don&amp;#x27;t enjoy seeing preprints like this going to the top of Hacker News. This would be a higher quality submission if the paper was peer-reviewed or put into a greater context, like a blog post discussion or something like that.&amp;quot;&lt;p&gt;Let me retract this and say something a bit nicer :) 
I personally think there this specific preprint making it to the top of HN is potentially harmful, because of the hype around LLMs, the diverse audience of readers here, and the specific title that implies a claim of &amp;quot;transformer with unlimited context length&amp;quot;, when this is misleading. I don&amp;#x27;t have anything against preprints in general - a lot of work outside of the peer-review process ends up being very impactful.</text></item></parent_chain><comment><author>cs702</author><text>After a very quick read, that&amp;#x27;s my understanding too: It&amp;#x27;s just KNN search with some bells and whistles. So I agree on points 1-3.&lt;p&gt;When something works well, I don&amp;#x27;t care much about point 4.&lt;p&gt;Personally, I&amp;#x27;ve had only mixed success with KNN search on long sequences. Maybe I haven&amp;#x27;t done it right? I don&amp;#x27;t know. In my experience, nothing seems to work quite as well as explicit token-token interactions by some form of attention, which as we all know is too costly for long sequences (O(n²)). Lately I&amp;#x27;ve been playing with &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;hazyresearch&amp;#x2F;safari&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;hazyresearch&amp;#x2F;safari&lt;/a&gt; , which uses a lot less compute and seems promising, though it reminds me of things like FNet. Otherwise, for long sequences I&amp;#x27;ve yet to find something better than &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;HazyResearch&amp;#x2F;flash-attention&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;HazyResearch&amp;#x2F;flash-attention&lt;/a&gt; for n×n interactions and &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;glassroom&amp;#x2F;heinsen_routing&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;glassroom&amp;#x2F;heinsen_routing&lt;/a&gt; for n×m interactions. If anyone has other suggestions, I&amp;#x27;d love to hear about them.</text></comment>
<story><title>Unlimiformer: Long-Range Transformers with Unlimited Length Input</title><url>https://arxiv.org/abs/2305.01625</url></story><parent_chain><item><author>mxwsn</author><text>1. This is not exact attention, but an approximation of it. Specifically, they use k-nearest neighbors to retrieve the top-k most similar tokens, out of an &amp;quot;unlimited-length input&amp;quot; say of size N, where k &amp;lt;&amp;lt; N.&lt;p&gt;2. This idea is quite similar to retrieval transformers and Hopfield networks which have been known and published for several years now. It&amp;#x27;s not really that novel.&lt;p&gt;3. Due to the preceding points, the title can easily mislead people. It&amp;#x27;s not really a conventional transformer, and it&amp;#x27;s not a breakthrough.&lt;p&gt;4. This paper is a preprint and not peer-reviewed.&lt;p&gt;&amp;quot;I generally don&amp;#x27;t enjoy seeing preprints like this going to the top of Hacker News. This would be a higher quality submission if the paper was peer-reviewed or put into a greater context, like a blog post discussion or something like that.&amp;quot;&lt;p&gt;Let me retract this and say something a bit nicer :) 
I personally think there this specific preprint making it to the top of HN is potentially harmful, because of the hype around LLMs, the diverse audience of readers here, and the specific title that implies a claim of &amp;quot;transformer with unlimited context length&amp;quot;, when this is misleading. I don&amp;#x27;t have anything against preprints in general - a lot of work outside of the peer-review process ends up being very impactful.</text></item></parent_chain><comment><author>chaxor</author><text>There&amp;#x27;s nothing really wrong with a preprint making it to the top - there can be genuinely good work that stays in preprint for quite some time.
I believe the original ELMo work that spurred the Sesame street gang is &lt;i&gt;still&lt;/i&gt; in preprint despite its importance in NLP (:shocked Pikachu face: not a transformer?!).&lt;p&gt;But yes, you&amp;#x27;re correct in this instance that it&amp;#x27;s not necessarily &amp;#x27;huge news&amp;#x27; since it is highly similar to a long list of the Reformer (LSH-based), Performer (FAVOR**), FNet (Fourier-based), Routing Transformer, Sparse Transformer, Longformer (task specific sparse), blockbert,  XLNet&amp;#x2F;xfmr-xl (slide + relative PE), BP-Transformer (binary partition), BigBird (global and rand attn),  RWKV which is..., etc.&lt;p&gt;** FAVOR actually is innovative and different in this space, but towards &lt;i&gt;similar&lt;/i&gt; ends anyway</text></comment>