Consider, for instance, GPT2, but with its MLP layer (\(d\to 4d\to d\)) replaced by 32 experts, each following a \(d\to d\to d\) configuration. This setup uses \(64d^{2}\) total parameters, but during inference, only \(2d^{2}\) parameters are used per token (e.g., when using \(topk=1\)). After including the Attention layers, which each have \(4d^{2}\) parameters, the ratio between the total and the effective number of parameters for the 32-expert MoE models is approximately \(\frac{4d^{2}+64d^{2}}{4d^{2}+2d^{2}}\approx 11.3\).