Training a deep Transformer model from scratch requires a large amount of labels that is only available in a few applications, such as machine translation where millions of parallel sentences can be used for training [1]}. A more efficient method is to first pre-train a Transformer-based language model on raw text corpora via self-supervised learning  [2]}, [3]}, and then fine-tune the pre-trained language model to downstream tasks with limited amounts of task-specific labels.
