BERT-MLM is evaluated to compare the effectiveness of pre-finetuning tasks. Training a pre-trained NLP model with MLM using data of the downstream task is frequently used for the model to understand the downstream data and improve its performance [1]}, [2]}, [3]}. It is implemented by simply masking 15% tokens of each input sentence.
