We delve into the reasons behind this. By adjusting LLaMA's architecture (e.g., switching GatedMLP back to normal MLP), as shown in Figure 5, we find that replacing LLaMA's GatedMLP with a standard MLP is necessary to match GPT2's scaling law. Notably, for a strong comparison, when using GatedMLP we select the best result from three learning rates, whereas for a standard MLP, akin to GPT2, we use a single learning rate.