The instance representation from the masked input is trained to predict that from the unmasked input by simply minimizing the cosine distance between the two representations.
We primarily follow BYOL for simplicity in this paper, but our approach is also found to work with the contrastive loss [1]}, [2]} in our experiments.
Our learning objective differs with the conventional BYOL in the following aspect. Conventional BYOL adopts a symmetric loss where the two views are learned to predict each other. This will drive the representation to find the common subspace shared by the two views, with invariance over other information [3]}. In our case, since the shared information between the two views is obvious, finding the commonality between the views does not make for meaningful self-supervision. Thus, we adopt the asymmetric loss. The information gap created by masking encourages the network to extrapolate the masked regions, instead of seeking a shared subspace.
