@inproceedings{
csord{\'a}s2024switchhead,
title={SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention},
author={R{\'o}bert Csord{\'a}s and Piotr Pi{\k{e}}kos and Kazuki Irie and J{\"u}rgen Schmidhuber},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=80SSl69GAz}
}
Despite many recent works on Mixture of Experts (MoEs) for resource-efficient Transformer language models, existing methods mostly focus on MoEs for feedforward layers. Previous attempts at extending MoE to the self-attention layer fail to match the performance of the parameter-matched baseline. Our novel SwitchHead is an effective MoE method for the attention layer that successfully reduces both the compute and memory requirements, achieving wall-clock speedup, while matching the language modeling performance of the baseline Transformer. Our novel MoE mechanism allows SwitchHead to compute up to 8 times fewer attention matrices than the standard Transformer. SwitchHead can also be combined with MoE feedforward layers, resulting in fully-MoE "SwitchAll" Transformers. For our 262M parameter model trained on C4, SwitchHead matches the perplexity of standard models with only 44% compute and 27% memory usage. Zero-shot experiments on downstream tasks confirm the performance of SwitchHead, e.g., achieving more than 3.5% absolute improvements on BliMP compared to the baseline with an equal compute resource.
SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention
Róbert Csordás Piotr Piękos Kazuki Irie Jürgen Schmidhuber