The technique of masking originates from representation learning on languages [1]}, [2]} and is especially suited for transformer architectures. In computer vision, block-wise masking [3]} and random masking [4]} are investigated to cope with the 2D nature of images.
Aside from BERT-like training approaches [3]}, [4]}, [7]}, [8]}, [9]}, a special type of random masking in the form of small local image crops has also been adopted in contrastive models [10]}, [11]}. However, the local crop augmentation introduced in [11]} is mainly designed for computational efficiency, without considering the impact of substantial content removal on representation quality.
Random erasing [13]} and Cut-out [14]} remove a region from an image and fill it with random or mean pixel values. While similar to block-wise masking, its purpose is mainly to regularize supervised learning instead of for self-supervised representation learning.
Our work reveals the power of masking even when used simply as a data augmentation in contrastive learning, without explicit supervision for token-level prediction.
