* For large models, replacing LLaMA architecture's gated MLP with a standard MLP (while keeping \(silu\) unchanged) noticeably improves LLaMA's capacity ratio.17 Footnote 17: As discussed in Appendix B, gated MLP layers are less stable to train, thus requiring more time. * For tiny LLaMA models, switching back to the GPT2Tokenizer is also necessary to match GPT2's performance, though this is a minor issue.18 Footnote 18: This only applies to tiny models and is specific to the biography data we consider here: GPT2Tokenizer may tokenize years such as 1991 into a single token, while LLaMATokenizer will tokenize it into four digit tokens. * Other modifications, such as changing from \(silu\) to \(gelu\) or adding trainable biases to layer-norms, do not noticeably affect the capacity ratios (so we ignore those figures).