The original transformer architecture can be referred to as post-layer normalization (post-LN), where the layer normalization is located outside the residual block [1]}. Post-LN converges much slower and requires a learning rate warm-up strategy [1]}. Xiong et al. proposed a pre-layer normalization (pre-LN) Transformer to address this issue, showing that it could help gradients converge faster while requiring no warm-up. Pre-LN Transformer can achieve this by helping control the gradient magnitudes and balancing the residual dependencies [1]}. Although, Pre-LN Transformer architecture does not require learning rate warm-up, they have inferior empirical performance as compared to post-LN Transformer architecture.
<FIGURE>