An important way to enhance efficiency in modern language models is the incorporation of sparsity. The Mixture of Experts (MoE) architecture plays a crucial role in this regard [9, 30]. A question arises: does the MoE model scale differently in terms of the capacity ratio? For an MoE model, let \(P\) denote the **total number of parameters in the model**, including all experts. Due to its inherent sparsity, the effective number of parameters can be significantly less than \(P\). Our primary observation is that MoE models scale _similarly_ to dense models, even with 32 experts per layer.