<story><title>xLSTM: Extended Long Short-Term Memory</title><url>https://arxiv.org/abs/2405.04517</url></story><parent_chain></parent_chain><comment><author>albertzeyer</author><text>It seems Sepp Hochreiter has talked already about this model since Oct 2023: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;huggingface&amp;#x2F;transformers&amp;#x2F;issues&amp;#x2F;27011&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;huggingface&amp;#x2F;transformers&amp;#x2F;issues&amp;#x2F;27011&lt;/a&gt;&lt;p&gt;In the scaling law comparison, I wonder if it is reasonable to compare number of parameters between Llama, Mamba, RWKV, xLSTM? Isn&amp;#x27;t compute time more relevant? E.g. in the figure about scaling laws, replace num of params by compute time.&lt;p&gt;Specifically, the sLSTM has still recurrence (memory mixing) in it, i.e. you cannot fully parallelize the computation. So scaling up Transformer could still look better when you look at compute time.&lt;p&gt;It seems neither the code nor the model params are released. I wonder if that will follow.</text></comment>
<story><title>xLSTM: Extended Long Short-Term Memory</title><url>https://arxiv.org/abs/2405.04517</url></story><parent_chain></parent_chain><comment><author>KhoomeiK</author><text>For those who don&amp;#x27;t know, the senior author on this paper (Sepp Hochreiter) was the first author on the original paper with Schmidhuber introducing LSTMs in 1997.</text></comment>