Mixture-of-ExperstThe mixture-of-experts (MoE) architecture has gained popularity with demonstrated superior performance compared to its dense counterpart [14, 15]. It would be highly beneficial to derive a similar scaling law applicable to the MoE architecture. In MoE architectures, each input interacts with only a subset of the network's parameters - chosen independently for each datapoint [12, 1]. This changing of the architecture would inevitably impact the form of \(L(N)\) because both the number of both activated and total parameters influence the loss values [10]. The following steps, such as Eq 3.9 and Eq 3.10 are general and should not be affected.