A famous architecture that builds on the original Transformers paper is Bidirectional Encoder Representations from Transformers (BERT) [1]}. The model is built by stacking the Transformer encoder blocks and introducing a new training scheme. The encoder block is pre-trained independently of the task. The decoder block can be added later and fine-tuned for the task at hand. This scheme allows training BERT models on large amounts of unlabeled data.
