Recent work has shown that attention-based language models excel atrecall, the ability to ground generations in tokens previously seen in context. However, the efficiency of attention-based models is bottle-necked during inference by the KV-cache’s aggressive memory consumption.
In this work, we explore whether we can improve language model efficiency (e.g.by reducing memory consumption)  without compromising on recall.
By applying experiments and theory to a broad set of architectures, we identify a key tradeoff between a model’sstate sizeand recall ability.
We show that efficient alternatives to attention (e.g.H3, Mamba, RWKV)  maintain a fixed-size recurrent state, but struggle at recall.
We proposeBaseda simple architecture combining linear and sliding window attention.
By varyingBasedwindow size and linear attention feature dimension, we can dial the state size and traverse the pareto frontier of the recall-memory tradeoff curve, recovering the full quality of attention on one end and the small state size of attention-alternatives on the other.
We train language models up tob parameters and show thatBasedmatches the strongest sub-quadratic models (e.g.Mamba)  in perplexity and outperforms them on real-world recall-intensive tasks by 6.22 accuracy points.
Implementations of linear attention are often less efficient than optimized standard attention implementations. To makeBasedcompetitive, we develop IO-aware algorithms that enablehigher throughput on language generation than FlashAttention-2, when generating 1024 tokens usingb parameter models.