Model Card
This model is pretrained as a reference baseline to the Based model provided here: https://huggingface.co/hazyresearch/based-1b-50b.
Both checkpoints are pretrained on 50Bn tokens of the Pile in the exact same data order using next token prediction.
Model Sources
The model is a standard Transformer model, using the Llama architecture (Rotary encodings, SwiGLU, RMS Norm, etc.)
The training code is provided here and can be used to reproduce training: https://github.com/HazyResearch/based
The paper for the work is here, and the appendix includes additional experimental details/hyperparameters: https://arxiv.org/abs/2402.18668
Uses
The purpose of this work is to evaluate the language modeling quality of a new efficient architecture, Based.
We include a series of benchmarks that you can use to evaluate quality:
- FDA: https://huggingface.co/datasets/hazyresearch/based-fda
- SWDE: https://huggingface.co/datasets/hazyresearch/based-swde
- SQUAD: https://huggingface.co/datasets/hazyresearch/based-squad
Citation
Please consider citing this paper if you use our work:
@article{arora2024simple,
title={Simple linear attention language models balance the recall-throughput tradeoff},
author={Arora, Simran and Eyuboglu, Sabri and Zhang, Michael and Timalsina, Aman and Alberti, Silas and Zinsley, Dylan and Zou, James and Rudra, Atri and Ré, Christopher},
journal={arXiv:2402.18668},
year={2024}
}
Please reach out to [email protected], [email protected], and [email protected] with questions.
- Downloads last month
- 6