We have seen several solutions that allow the training of deeper Transformer models with improved performance as compared to the vanilla transformer architecture. Training these deep transformer models from scratch require large datasets, making training with a small dataset challenging. Small datasets require pre-trained models and small batch sizes to perform well, but these two requirements make training additional Transformer layers more difficult. Using a small batch size causes the variance of the updates to be larger, and even with a large batch size, the model may typically generalize poorly. In some cases, better initialization techniques can optimize the model performing well on the smaller datasets. The use of Xavier initialization is the most common scheme for Transformers [1]}, although recently, T-Fixup has been shown to outperform Xavier [2]}. T-Fixup introduced by Huang et al., found that when training Transformer without learning rate warm-up, the variance in the Adam optimizer was amplified by large initial learning rate leading to large updates at the beginning of training [2]}. Hence, the learning rate warm-up requirement comes from the Adam optimizer's instability combined with gradient vanishing through layer normalization. To resolve this, the authors proposed a new weight initialization scheme with theoretical guarantees that keeps model updates bounded and allows to remove both warm-up and layer normalization [2]}. Following the work of T-Fixup, a data-dependent initialization technique was developed by Borealis AI known as DT-Fixup [5]}. DT-Fixup allows for training deeper transformer models with small datasets, given correct optimization procedures are followed.
