In this paper, we aim to study the connections between the instance and the distributed representations, and we explore self-supervision for learning these representations.
We start with an observation that random masking could be viewed as a novel data augmentation scheme not previously exploited in Siamese networks. For the masked area, its potential degrees of freedom grow combinatorially large with its size, allowing for richer self-supervision than conventional augmentations such as cropping and scaling, which are heavily biased towards areas around the image center [1]}.
More importantly, the self-supervision from conventional augmentations lead to a common representation vector that encompasses multiple augmentations of an instance, and this invariance degrades the sensitivity of the representation to spatial locality. On the contrary, random masking preserves the original content of the unmasked area and does not alter the geometrical structure of the data.
