\(\bullet\)_RMSNorm._ To improve the training speed of LayerNorm (LN), RMSNorm [257] is proposed by re-scaling the activations with only the root mean square (RMS) of the summed activations, instead of the mean and variance. Related research has demonstrated its superiority in training speed and performance on Transformer [266]. Representative models that adopt RMSNorm include Gopher [64] and Chinchilla [34].