\(\bullet\)_DeepNorm._ DeepNorm is proposed by Microsoft [258] to stabilize the training of deep Transformers. With DeepNorm as residual connections, Transformers can be scaled up to 1,000 layers [258], which has shown the advantages of stability and good performance. It has been adopted by GLM-130B [93].