The utility of randomized masking. In addition to being used as one of the main training objectives for BERT (the “cloze task”, [1]}), the flexibility afforded by randomized masking in bidirectional models has been utilized in other previous works applied to language [2]}, [3]} and vision [4]} – mostly for the purpose of speeding up auto-regressive decoding, which is not our focus here.
