<story><title>The RWKV language model: An RNN with the advantages of a transformer</title><url>https://johanwind.github.io/2023/03/23/rwkv_overview.html</url></story><parent_chain></parent_chain><comment><author>Straw</author><text>Unfortunately its not very good at longer context lengths, which sort of defeats the point of efficient scaling with context. See &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;twitter.com&amp;#x2F;arankomatsuzaki&amp;#x2F;status&amp;#x2F;1639000379978403853&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;twitter.com&amp;#x2F;arankomatsuzaki&amp;#x2F;status&amp;#x2F;16390003799784038...&lt;/a&gt;&lt;p&gt;Its also not really an RNN. The best way to describe the key time mixing operation is a normalized exponentially weighted moving average (EMA)- no non-linearity. Once viewed this way, its not surprising that it struggles at longer contexts- everything decays, and it has limited space to put things. Of course, it does have some clever tricks, and can choose to remember things for a while by upweighting them, but not forever.</text></comment>
<story><title>The RWKV language model: An RNN with the advantages of a transformer</title><url>https://johanwind.github.io/2023/03/23/rwkv_overview.html</url></story><parent_chain></parent_chain><comment><author>imranq</author><text>The stats shown are pretty compelling that this transformer + RNN approach works. Looking back at the Llama paper (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;arxiv.org&amp;#x2F;pdf&amp;#x2F;2302.13971.pdf&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;arxiv.org&amp;#x2F;pdf&amp;#x2F;2302.13971.pdf&lt;/a&gt;), many of the results are comparable between llama and RWKV. E.g. Llama 13bn scores 80.1 on PIQA and RWKV scores 77.5&lt;p&gt;I wonder if RLHF would boost performance for this architecture in the same way</text></comment>