We propose a simple model, called ExtreMA, where the instance representation from masked input is trained to predict that from the full view, in a plain BYOL fashion [1]}.
The information gap created by masking encourages the student network to encode as much information as possible, and hence bootstrap the teacher network to be stronger.
We adopt the visual transformer ViT [2]} to embed distributed representations over patches, and this is followed by cross-attention blocks [3]} to aggregate the distributed representations into the instance representation. The instance-level learning objective provides the supervision for both of the representations. In our model, the distributed representations are only implicitly learned without the corresponding token-level objective used in masked modeling. However, through investigating and visualizing the attention maps (in Figure REF ), we find that the output representations of our model maintain accurate correspondences with the input tokens and that semantic clusters tend to emerge from the learned distributed representations.
