How to reduce compute and memory requirements of neural networks (NNs)  without sacrificing performance?
Many recent works use sparse Mixtures of Experts (MoEs)  to build resource-efficient large language models (LMs) .
Here we introduce several novel perspectives on MoEs, presenting
a general framework thatunifiesvarious methods toapproximate two-layer NNs(e.g., feedforward blocks of Transformers) , including product-key memories (PKMs) .
Leveraging insights from this framework, we propose methods to improve both MoEs and PKMs.
Unlike prior work that compares MoEs with dense baselines under thecompute-equalcondition, our evaluation condition isparameter-equal, which is crucial to properly evaluate LMs.
We show that our MoEs are competitive with thedenseTransformer-XL on both the WikiText-103 and enwiki8 datasets at two different scales, while being much more resource-efficient.
This demonstrates that MoEs are relevant not only to extremely large LMs but also to any-scale resource-efficient LMs.
Our code is public.111https://github.com/robertcsordas/moe