<story><title>RWKV: Reinventing RNNs for the Transformer Era</title><url>https://arxiv.org/abs/2305.13048</url></story><parent_chain><item><author>andy_xor_andrew</author><text>One thing I&amp;#x27;m keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models?&lt;p&gt;All these 2k&amp;#x2F;4k&amp;#x2F;8k context sizes that we&amp;#x27;ve had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer questions about the text coherently.&lt;p&gt;But what about 32k contexts, or beyond? At some point, as token size increases, the ability of a human to give a highly precise and detailed answer decreases. They must start to generalize. A human could not read Infinite Jest in one pass and then answer details about every single sentence. But could a transformer, or a RNN? As the context grows, is it harder to keep a high granularity of detail? Or am I wrong in trying to think of these models the way I think about the human mind, and they are actually able to handle this problem just fine?&lt;p&gt;I&amp;#x27;m aware that we can cheat a bit, by adding a lookup step into an embedding database, to provide &amp;quot;infinite&amp;quot; context with &amp;quot;infinite&amp;quot; precision. But to me, that is analogous to a human looking up information in a library in order to answer a question. I&amp;#x27;m interested in the inherent, emergent memory that these models have available to them in just one forward pass.</text></item></parent_chain><comment><author>coppsilgold</author><text>The word &amp;quot;attention&amp;quot; has been stretched pretty far to explain what is happening inside a transformer.&lt;p&gt;What&amp;#x27;s actually happening is that every token embedding interacts with every other token embedding before it and as the product of this interaction (dot product + softmax) it takes a fraction of every other token embedding and adds it to itself. Technically, it&amp;#x27;s different transforms&amp;#x2F;functions of the embedding.&lt;p&gt;You can view it as every token embedding mixing information from other embeddings into itself. Done ~100 times in parallel (&amp;quot;attention heads&amp;quot;), ~100 times in sequence (layers). As per GPT-3 model (175B).</text></comment>
<story><title>RWKV: Reinventing RNNs for the Transformer Era</title><url>https://arxiv.org/abs/2305.13048</url></story><parent_chain><item><author>andy_xor_andrew</author><text>One thing I&amp;#x27;m keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models?&lt;p&gt;All these 2k&amp;#x2F;4k&amp;#x2F;8k context sizes that we&amp;#x27;ve had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer questions about the text coherently.&lt;p&gt;But what about 32k contexts, or beyond? At some point, as token size increases, the ability of a human to give a highly precise and detailed answer decreases. They must start to generalize. A human could not read Infinite Jest in one pass and then answer details about every single sentence. But could a transformer, or a RNN? As the context grows, is it harder to keep a high granularity of detail? Or am I wrong in trying to think of these models the way I think about the human mind, and they are actually able to handle this problem just fine?&lt;p&gt;I&amp;#x27;m aware that we can cheat a bit, by adding a lookup step into an embedding database, to provide &amp;quot;infinite&amp;quot; context with &amp;quot;infinite&amp;quot; precision. But to me, that is analogous to a human looking up information in a library in order to answer a question. I&amp;#x27;m interested in the inherent, emergent memory that these models have available to them in just one forward pass.</text></item></parent_chain><comment><author>teruakohatu</author><text>&amp;gt; But what about 32k contexts, or beyond? At some point, as token size increases, the ability of a human to give a highly precise and detailed answer decreases&lt;p&gt;War and Peace is over 580,000 words long. Chapter one is ~2,020 words which encoded for GPT3 is ~2,956 tokens (lots of longer, older words and proper nouns eg. &amp;quot;scarlet-liveried footman&amp;quot; is six tokens), so we might expect the entire book to be ~750,000 tokens long.&lt;p&gt;Many people could reason about the book in its entirety. They would not have an encyclopedic recall of random dates and side characters or be able to quote any passage, but they could perform deep analysis on it.</text></comment>