Multi-Layer Sparse Autoencoders
Collection
Single SAEs trained on the residual stream activation vectors from every transformer layer simultaneously: https://arxiv.org/abs/2409.04185
•
30 items
•
Updated
A Multi-Layer Sparse Autoencoder (MLSAE) trained on the residual stream activation vectors from every layer of EleutherAI/pythia-70m-deduped with an expansion factor of 8 and k = 32, over 1 billion tokens from monology/pile-uncopyrighted.
For more details, see: