---
license: mit
datasets:
- cerebras/SlimPajama-627B
language:
- en
---


This checkpoint of the 1.3B GLA model used in the paper [Gated Linear Attention](https://arxiv.org/abs/2312.06635).  The model is trained with 100B tokens from the SlimPajama dataset tokenized with Llama2 tokenizer.

See the model and loading script in this [repo](https://github.com/berlino/gated_linear_attention).