Language model pre-training is an active research field.
In this section we describe BERT [1]}, a classic PLM based on the Transformer architecture [2]}, and the pre-training and fine-tuning framework that has been widely used to adapt a PLM to downstream applications. For more detailed surveys we refer to [3]} and encourage readers to check more recent updates in the field.
