Efficiently modeling sequences with infinite context length has been a long-standing problem. Past works suffer from either the quadratic computation complexity or the limited extrapolation ability on length generalization. In this work, we presentSamba, a simple hybrid architecture that layer-wise combines Mamba, a selective State Space Model (SSM) , with Sliding Window Attention (SWA) .Sambaselectively compresses a given sequence into recurrent hidden states while still maintaining the ability to precisely recall memories with the attention mechanism.
We scaleSambaup to 3.8B parameters with 3.2T training tokens and show thatSambasubstantially outperforms the state-of-the-art models based on pure attention or SSMs on a wide range of benchmarks. When trained on 4K length sequences,Sambacan be efficiently extrapolated to 256K context length with perfect memory recall and show improved token predictions up to 1M context length. As a linear-time sequence model,Sambaenjoys ahigher throughput compared to Transformers with grouped-query attention when processing user prompts of 128K length, andspeedup when generating 64K tokens with unlimited streaming. A sample implementation ofSambais publicly available inhttps://github.com/microsoft/Samba.