\(\bullet\)_Sandwich-LN_. Based on pre-LN, Sandwich-LN [255] adds extra LN before the residual connections to avoid the value explosion issues in Transformer layer outputs. However, it has been found that Sandwich-LN sometimes fails to stabilize the training of LLMs and may lead to the collapse of training [93].