The model architecture is similar to LLaMA [44] with two differences: we use grouped-query attention [1] to replace the multi-query attention and we apply rotary position embedding on half the dimensions of the query and key vectors.