DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
Paper
•
2401.06066
•
Published
•
42
Sparsity
-
-
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Paper • 2006.16668 • Published • 3 -
OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models
Paper • 2402.01739 • Published • 26 -
BlackMamba: Mixture of Experts for State-Space Models
Paper • 2402.01771 • Published • 22