**RMSNorm** Same to Llama2 model (Touvron et al., 2023) 7B serious. We normalize the input of each transformer sub-layer, the attention layer, and the feedforward layer, with RMSNorm (Zhang and Sennrich, 2019).