Linear Recurrence has proven to be a powerful tool for modeling long sequences efficiently. In this work, we show that existing models fail to take full advantage of its potential. Motivated by this finding, we develop GateLoop, a foundational sequence model that generalizes linear recurrent models such as S4, S5, LRU and RetNet, by employing data-controlled state transitions.
Utilizing this theoretical advance, GateLoop empirically outperforms existing models for auto-regressive language modeling. Our method comes with a low-costrecurrent mode and an efficientparallel mode, whereis the sequence length, making use of highly optimized associative scan implementations. Furthermore, we derive ansurrogate attention mode, revealing remarkable implications for Transformer and recently proposed architectures.
Specifically, we prove that our approach can be interpreted as providing data-controlled relative-positional information to Attention.
While many existing models solely rely on data-controlled cumulative sums for context aggregation, our findings suggest that incorporating data-controlled complex cumulative products may be a crucial step towards more powerful sequence models.