**Normalization Methods.** Training instability is a challenging issue for pre-training LLMs. To alleviate this issue, normalization is a widely adopted strategy to stabilize the training of neural networks. In the vanilla Transformer [22], LayerNorm [256] is employed. Recently, several advanced normalization techniques have been proposed as alternatives to LayerNorm, _e.g._, RMSNorm, and DeepNorm.