1. LLaMA/Mistral use so-called GatedMLP layers, which is \(V(\sigma(W_{1}x)\cdot(W_{2}x))\) instead of \(V\sigma(Wx)\). Shazeer [29] suggested that gated activation might yield marginally improved performance. 2. Unlike GPT2, LLaMA/Mistral do not tie weights. 3. Mistral features larger MLP layers compared to GPT2/LLaMA. 4. Mistral promotes group-query attention, not so by GPT2/LLaMA. 5. LLaMA/Mistral employ a different tokenizer than GPT2. 6. GPT2 uses the \(gelu\) activation function, LLaMA/Mistral opt for \(silu\). 7. GPT2 implements layer normalization with a trainable bias.