The replace token detection task, proposed by [1]}, is a sample efficient variant of masked language modelling (Figure REF ). Instead of replacing some tokens as the special [MASK] token, [1]} employ a small generator network to generate plausible alternatives. Then the discriminator network predicts whether each token in the input is replaced by a generator sample or not.
