**Normalization Position.** In addition to the normalization method, normalization position also plays a crucial role in the LLMs. There are generally three choices for the normalization position, _i.e.,_ post-LN, pre-LN, and sandwich-LN.