[1]} explored the idea of pre-training recurrent language model and transferring it to the downstream supervised models. They use unlabelled data from Amazon reviews to pre-train the language model and find that it can improve classification accuracy on the Rotten Tomatoes dataset. [2]} extend the single direction language model to bidirectional. Based on these efforts, [3]} propose the masked language modelling pre-training task to better capture contexts from both sides. Different to [4]} who build two language models—left-to-right and right-to-left—which are trained separately, the masked language modelling is a fill-in-the-blank task. That is, a small set of tokens are masked, and the model needs to use the context tokens to try to predict what the masked tokens should be (Figure REF ).
<FIGURE>