* The GatedMLP architecture _slows down_ the model's learning speed, and we observe less stable training with its use.27 Footnote 27: For example, mixed-precision fp16 training can sometimes fail for LLaMA/Mistral models smaller than 100M; hence, we use mixed-precision bf16 instead. Conversely, GPT2 models up to 1B can be trained with fp16. * Removing MLP layers entirely _slows down_ the model's learning speed, whereas adjusting the size of MLP layers (e.g., from \(8d^{2}\) to \(10.5d^{2}\) or down to \(2d^{2}\)) may not have a significant impact.