Multi-Layer Sparse Autoencoders with Transformers
Collection
Single SAEs trained on the residual stream activation vectors from every transformer layer simultaneously (including the transformers).
•
22 items
•
Updated
mlsae-pythia-70m-deduped-x128-k32-tfm
A Multi-Layer Sparse Autoencoder (MLSAE) trained on the residual stream activation vectors from every layer of EleutherAI/pythia-70m-deduped with an expansion factor of 128 and k = 32, over 1 billion tokens from monology/pile-uncopyrighted. This model includes the underlying transformer.
For more details, see:
- Paper: https://arxiv.org/abs/2409.04185
- GitHub repository: https://github.com/tim-lawson/mlsae
- Weights & Biases project: https://wandb.ai/timlawson-/mlsae
- Downloads last month
- 48