Another difference between the masked language modelling and replace token detection pre-training tasks is that the former is performed on only masked tokens, whereas the latter is defined over all input tokens. Therefore, a replace token detection pre-training task requires less compute, measured using floating point operations. [1]} show that it performs comparably to masked language modelling pre-training task while using less than 1/4 of their compute and outperforms masked language modelling when using the same amount of compute. Because pre-training a language representation model using replace token detection task can be done within several days using a single GPU, we thus use it for our investigation in Chapter .
<FIGURE>