For other model architectures, Mistral, GPT2\({}_{0}\), and GPT2\({}_{1/4}\), their scaling laws in the 100-exposure setting are presented in Figure 4. Figure 4(c) confirms that the Mistral architecture also underperforms GPT2 due to its use of gated MLP. Figure 4(d) reveals that reducing GPT2\({}_{1/4}\)'s MLP layer size by a quarter has a _negligible impact_ on model capacity. However, removing the MLP layers entirely in GPT2\({}_{0}\) significantly reduces the model's capacity, see Figure 4(e).