<story><title>Mamba Explained: The State Space Model Taking On Transformers</title><url>https://www.kolaayonrinde.com/blog/2024/02/11/mamba.html</url></story><parent_chain><item><author>Straw</author><text>The SSMs papers and blogs always have unnecessarily complicated explanations. At this point I almost wonder if its to hide how simple the underlying algorithms are, or to make them seem fancy.&lt;p&gt;SSMs are doing exponentially weighted moving averages (EMA). That&amp;#x27;s it- to summarize the past, you take an EMA of a variable output at each time step. Mamba changes one key thing- instead of decaying the past by a fixed amount each step as in a constant-time EMA, we have another output which decides how much to forget, or equivalently, how much &amp;#x27;time&amp;#x27; has passed since the last observation in our EMA.&lt;p&gt;All of the matrix equations, continuous time, discretization, etc, will end up with a dynamic-forgetting EMA as I describe above. This also makes the benefits and limitations clear- finite state size, has to decide at a given layer what to forget before it sees the past at that layer.</text></item></parent_chain><comment><author>logicchains</author><text>Are there any fundamental differences between Mamba, Retnet and RWKV, or are they all variants of this same architecture?</text></comment>
<story><title>Mamba Explained: The State Space Model Taking On Transformers</title><url>https://www.kolaayonrinde.com/blog/2024/02/11/mamba.html</url></story><parent_chain><item><author>Straw</author><text>The SSMs papers and blogs always have unnecessarily complicated explanations. At this point I almost wonder if its to hide how simple the underlying algorithms are, or to make them seem fancy.&lt;p&gt;SSMs are doing exponentially weighted moving averages (EMA). That&amp;#x27;s it- to summarize the past, you take an EMA of a variable output at each time step. Mamba changes one key thing- instead of decaying the past by a fixed amount each step as in a constant-time EMA, we have another output which decides how much to forget, or equivalently, how much &amp;#x27;time&amp;#x27; has passed since the last observation in our EMA.&lt;p&gt;All of the matrix equations, continuous time, discretization, etc, will end up with a dynamic-forgetting EMA as I describe above. This also makes the benefits and limitations clear- finite state size, has to decide at a given layer what to forget before it sees the past at that layer.</text></item></parent_chain><comment><author>binarymax</author><text>I hadn’t heard of Mamba before reading this article, and I was wondering if anyone has tried setting importance of a token as a TF-IDF or BM25 lookup.  Requires a first pass to construct the token index but otherwise it seems like it would address the big issue that all these architectures have - they don’t know how “important” a token is.  Interestingly this seems to be the crux of Mamba - deciding what tokens to forget!  EMA other treats all tokens equally at sequence time.  What if the tokens were weighted beforehand and the weights were passed as an attention mechanism?  I wonder if anyone has tried something like this.</text></comment>