The Transformers may not be easy to train from scratch for a beginner. The original transformer architecture [1]} utilized many different strategies to stabilize the gradients during training for deeper networks. The authors' residual connections allow for training a deeper network. Later, layer normalization operations were added alongside an adaptive optimizer (Adam) to provide different learning rates for different parameters. Like most other deep learning models, Transformers are also sensitive to the learning rate. Given an optimal learning rate, Transformers can converge faster than traditional sequence models. During the first few epochs, it is common to observe performance drop. However, after a few epochs, the model will generally start to converge to better values. In the original implementation, the authors used a warm-up learning rate strategy that increases linearly for the first \(N\)  training steps and then decreases proportionally to the inverse square root of the step number, \(\frac{1}{\sqrt{N}}\) .
