1. LLaMA and Mistral employ MLP layers with gated activation, using \(V(\sigma(W_{1}x)\cdot(W_{2}x))\) instead of \(V\sigma(Wx)\). Shazeer [29] noted that gated activation appears to yield slightly better performance. 2. Unlike GPT2, which ties the weights of the embedding layer and the output (LMHead) layer, LLaMA and Mistral do not. 3. For a hidden dimension \(d\), GPT2/LLaMA have \(4d^{2}\) parameters in the attention layer and \(8d^{2}\) in the MLP layer, whereas Mistral allocates a larger \(10.5d^{2}\) for its MLP layer. 4. Mistral promotes group-query attention (e.g., using 4 groups, thus reducing the K/V matrices to \(d^{2}/4\) in size), unlike GPT2. LLaMA does not favor multi-query attention unless in its very large models, such as the 70B variant. 5. LLaMA and Mistral utilize different tokenizers compared to GPT2, with Mistral's tokenizer being nearly identical to LLaMA's. 6. GPT2 employs \(\sigma=gelu\), while LLaMA/Mistral use \(\sigma=silu\). 7. GPT2 incorporates layer normalization with trainable bias, which LLaMA/Mistral do not.