The Llama family is a well-known class of AutoRegressive language models. Llama(Touvron et al., 2023) was first proposed as a language model that uses only publicly available data. To improve training stability, they normalise the input of each transformer sub-layer instead of normalising the output. They use the RMSNorm normalisation function introduced by Zhang and Sennrich (2019) and replace the ReLU non-linearity with the SwiGLU activation function introduced by Shazeer (2020) to improve performance. Furthermore, the authors replaced the absolute position embedding with the rotational position embedding (RoPE) introduced by Su et al. (2024) at each layer of the network. Llama2(Touvron et al., 2023) used supervised fine-tuning, initial and iterative reward modelling and RLHF in their experiments. they also invented a new technique, Ghost Attention (GAtt), which helps to control the flow of dialogue in multiple turns. Qwen(Bai et al., 2023), based on Llama, the following adjustments were made: 1. The method of loose embedding was chosen instead of bundling the weights of input embedding and output projection to save memory cost. 2. The accuracy of the inverse frequency matrix was improved. 3. For most of the layers, bias was eliminated as per Chowdhery et al. (2023). However, bias was added in the QKV layer to enhance the model's extrapolation capability. The traditional layer normalization technique described in Ba et al. (2016) was replaced with RMSNorm. They chose SwiGLU as their activation function, which is a combination of Swish Ramachandran et al. (2017) and gated linear units (Dauphin et al., 2017). The dimension of the feed-forward network (FFN) is also reduced. Furthermore, Mistral 7b (Jiang et al., 2023) utilises Grouped Query Attention (GQA) to enhance inference speed and combines it with Sliding Window Attention (SWA) to efficiently process sequences of any length with reduced inference cost. These techniques demonstrate superior performance over Llama2.The Llama model is open source and uses publicly available data, providing researchers (Yan et al., 2024) (Asai et al., 2023) (Wang et al., 2023) with more opportunities to expand. As a result, many researchers use the Llama family as language models in the RALM architecture.