* _Even for large models_, LLaMA architecture's capacity ratio can be 1.3x worse than GPT2, even after optimally tuning learning rates. The results are similar for Mistral. * Reducing GPT2's MLP size by 1/4 has a negligible impact on the capacity ratio. * Removing MLPs decreases the capacity ratio by more than 1.5x.