LLaMA/Mistral.Notably, as of the writing of this paper, LLaMA [32, 33] and Mistral [19] stand out as popular, publicly-available large language models. We highlight their key architecture differences from GPT2 -- which we define as having rotary embedding and no dropout.