Additionally, we experimented with enabling trainable biases in LLaMA's layernorms and switching from \(silu\) to \(gelu\) (to more closely resemble GPT2), in a similar way as Figure 5, but found these changes do not affect the model's capacities. We ignore those experiments for clarity.