Liu et al. proposed an adaptive model initialization (Admin) to benefit from the ease of training of the post-LN Transformer while achieving the performance of the pre-LN Transformer [1]}. Adaptive model initialization is a technique used in other areas of machine learning to initialize the model such that the dependencies between the input and output variables are better captured [1]}. This initialization technique helps the model learn the relationships between the variables more effectively and improves performance. Another option is to remove the layer normalization altogether. ReZero replaces the layer normalization operation with a trainable parameter, \(\alpha \) , initialized at 0 in each residual layer [3]}. Consequently, the entire network is initialized to compute the identity function, with a gradual and adaptive introduction of the contributions of the self-attention and MLP layers. The residual dependencies seem to balance well with Admin or ReZero for training deeper Transformer models with better generalization performance.
