<story><title>From deep to long learning?</title><url>https://hazyresearch.stanford.edu/blog/2023-03-27-long-learning</url></story><parent_chain><item><author>cs702</author><text>This looks really interesting! If these guys succeed in bringing self-attention&amp;#x27;s computational cost down from O(n²) to O(n log n), that would be a huge win. The quadratic cost makes it very difficult to increase sequence length on current hardware. I&amp;#x27;m going to take a closer look.&lt;p&gt;There are other interesting ongoing efforts to increase sequence length. One that has worked for me is this dynamic routing algorithm, related to self-attention, that can handle sequences with 1M+ tokens in a single GPU: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;glassroom&amp;#x2F;heinsen_routing&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;glassroom&amp;#x2F;heinsen_routing&lt;/a&gt; . Right now, you can take 1,000 sequences of hidden states computed by a pretrained transformer, each sequence with, say, 1024 tokens, concatenate them into a single ultra-long sequence with 1,024,000 hidden states, slap 1,024,000 position encodings on top, and feed the whole thing to that routing algorithm to predict the next token (or whatever other training objective you want to optimize for). It works. Search the README for &amp;quot;Very Long Sequences&amp;quot;.&lt;p&gt;If anyone here has other suggestions for working with long sequences (hundreds of thousands to millions of tokens), &lt;i&gt;I&amp;#x27;d love to learn about them&lt;/i&gt;.</text></item></parent_chain><comment><author>smaddox</author><text>Other approaches for long sequence lengths:&lt;p&gt;- &amp;quot;Hyena Hierarchy: Towards Larger Convolutional Language Models&amp;quot; &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;arxiv.org&amp;#x2F;pdf&amp;#x2F;2302.10866.pdf&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;arxiv.org&amp;#x2F;pdf&amp;#x2F;2302.10866.pdf&lt;/a&gt;&lt;p&gt;- &amp;quot;Resurrecting Recurrent Neural Networks for Long Sequences&amp;quot; &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;arxiv.org&amp;#x2F;pdf&amp;#x2F;2303.06349.pdf&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;arxiv.org&amp;#x2F;pdf&amp;#x2F;2303.06349.pdf&lt;/a&gt;&lt;p&gt;It&amp;#x27;s unclear (at least to me) if any of these scale as well as transformers at this point.</text></comment>
<story><title>From deep to long learning?</title><url>https://hazyresearch.stanford.edu/blog/2023-03-27-long-learning</url></story><parent_chain><item><author>cs702</author><text>This looks really interesting! If these guys succeed in bringing self-attention&amp;#x27;s computational cost down from O(n²) to O(n log n), that would be a huge win. The quadratic cost makes it very difficult to increase sequence length on current hardware. I&amp;#x27;m going to take a closer look.&lt;p&gt;There are other interesting ongoing efforts to increase sequence length. One that has worked for me is this dynamic routing algorithm, related to self-attention, that can handle sequences with 1M+ tokens in a single GPU: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;glassroom&amp;#x2F;heinsen_routing&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;glassroom&amp;#x2F;heinsen_routing&lt;/a&gt; . Right now, you can take 1,000 sequences of hidden states computed by a pretrained transformer, each sequence with, say, 1024 tokens, concatenate them into a single ultra-long sequence with 1,024,000 hidden states, slap 1,024,000 position encodings on top, and feed the whole thing to that routing algorithm to predict the next token (or whatever other training objective you want to optimize for). It works. Search the README for &amp;quot;Very Long Sequences&amp;quot;.&lt;p&gt;If anyone here has other suggestions for working with long sequences (hundreds of thousands to millions of tokens), &lt;i&gt;I&amp;#x27;d love to learn about them&lt;/i&gt;.</text></item></parent_chain><comment><author>logicchains</author><text>&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;BlinkDL&amp;#x2F;RWKV-LM&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;BlinkDL&amp;#x2F;RWKV-LM&lt;/a&gt; this claims to work well with long sequences.</text></comment>