|
--- |
|
license: mit |
|
datasets: |
|
- togethercomputer/RedPajama-Data-V2 |
|
language: |
|
- en |
|
library_name: transformers |
|
--- |
|
|
|
This is a set of sparse autoencoders (SAEs) trained on the residual stream of [Llama 3 8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) using the 10B sample of the [RedPajama v2 corpus](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-V2), which comes out to roughly 8.5B tokens using the Llama 3 tokenizer. The SAEs are organized by layer, and can be loaded using the EleutherAI [`sae` library](https://github.com/EleutherAI/sae). |
|
|
|
These are early checkpoints of an ongoing training run which can be tracked [here](https://wandb.ai/eleutherai/sae/runs/7r5puw5z?nw=nwusernorabelrose). They will be updated as the training run progresses. The last upload was at 7,000 steps. |
