Papers
arxiv:2312.00752

Mamba: Linear-Time Sequence Modeling with Selective State Spaces

Published on Dec 1, 2023
· Submitted by akhaliq on Dec 4, 2023
#1 Paper of the day
Authors:

Abstract

Foundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module. Many subquadratic-time architectures such as linear attention, gated convolution and recurrent models, and structured state space models (SSMs) have been developed to address Transformers' computational inefficiency on long sequences, but they have not performed as well as attention on important modalities such as language. We identify that a key weakness of such models is their inability to perform content-based reasoning, and make several improvements. First, simply letting the SSM parameters be functions of the input addresses their weakness with discrete modalities, allowing the model to selectively propagate or forget information along the sequence length dimension depending on the current token. Second, even though this change prevents the use of efficient convolutions, we design a hardware-aware parallel algorithm in recurrent mode. We integrate these selective SSMs into a simplified end-to-end neural network architecture without attention or even MLP blocks (Mamba). Mamba enjoys fast inference (5times higher throughput than Transformers) and linear scaling in sequence length, and its performance improves on real data up to million-length sequences. As a general sequence model backbone, Mamba achieves state-of-the-art performance across several modalities such as language, audio, and genomics. On language modeling, our Mamba-3B model outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation.

Community

When we get to play with the model?

is this the new Hyena? I have been holding my breath for SSM moment for so long I plumb nearly asphyxiated

Hello guys, greate work. Just saw your work on Twitter. Are you planning to release a 6B-7B version in the future?

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

Excellent research! Would love to collaborate with you guys (UCSB).

This Mamba paper has some investigation of DNA modeling, @rlacombe (hat/tip @lvwerra for the info)

Thanks @julien-c and @lvwerra ! Super helpful 🙏

Edit: @albertgu @tridao have you/do you plan on releasing the weights for Mamba-7m trained on HG38?

(They don’t seem to be on HF, but my apologies if I missed them!)

I’m exploring gLMs for CRISPR edits property predictions and would love to fine-time a pre-trained model if available.

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

Are the DNA models available somewhere? I'm only seeing the NLP models. Thanks.

·

would love to know too actually

Sign up or log in to comment

Models citing this paper 53

Browse 53 models citing this paper

Datasets citing this paper 1

Spaces citing this paper 12

Collections including this paper 50