Figure REF  shows an example of pre-training BERT using MLM. Since no manual label is required, BERT can be trained on large amounts of raw text corpora.
For example, the BERT base model, a 12-layer Transformer with 110 million parameters, is pre-trained on the Wikipedia and Google Book Corpora which amount to 16 GB words [1]}.
Recent results show that training a larger PLM on a larger dataset significantly improve model's performance, especially in few-shot or zero-shot settings where there is few or zero task labels in the downstream task [2]}, [3]}, [4]}, [5]}, [6]}.
