Many implementations of Transformers use gradient clipping during the training. This process tends to avoid exploding gradients and potential divergence if the number of steps is too large. Other methods of gradient clipping, such as choosing a step size proportional to the batch size, are not recommended for Transformers as these can lead to slower convergence [1]}.
