We utilize the tutel package for implementing Mixture-of-Experts (MoE) on GPT2 models [18]. In MoE, the parameter \(topk\) determines the number of experts each token is routed to. It is recommended by some practitioners to use \(topk=2\) during training and \(topk=1\) during testing. Additionally, the \(cap\_factor\) parameter ensures that, given \(M\) experts, each expert receives no more than \(\frac{cap\_factor}{M}\) fraction of the data.