\(\bullet\)_Post-LN._ Post-LN is used in the vanilla Transformer [22], which is placed between residual blocks. However, existing work has found that the training of Transformers with post-LN tends to be instable due to the large gradients near the output layer [267]. Thus, post-LN is rarely employed in existing LLMs except combined with other strategies (_e.g.,_ combining post-LN with pre-LN in GLM-130B [93]).