Results of the averaged attention distance in different attention heads (dots) w.r.t the layer number, on supervised model (DeiT), contrastive learning model (MoCo v3) and SimMIM model with ViT-B as backbone are shown in Figure REF .
We find that the supervised model tends to focus locally at lower layers but more globally at higher layers, which well matches the observations in ViT [1]}. Surprisingly, the contrastive learning model acts very similarly to the supervised counterpart. This may also be understandable, since MoCo v3 has a very high linear evaluation accuracy on ImageNet-1K (76.7% of top-1 accuracy), which indicates that the features of the last layer of MoCo v3 are very similar to that of the supervised counterpart.
But for the model trained by SimMIM, its behavior is significantly different to supervised and contrastive learning models. Each layer has diverse attention heads that tend to aggregate both local and global pixels, and the average attention distance is similar to the lower layers of the supervised model. As the number of layers gets deeper, the averaged attention distance becomes even slightly smaller. That is, MIM brings locality inductive bias to the trained model, that the models tend to aggregate near pixels in part of the attention heads. Also, a similar observation could be observed with Swin-B as the backbone, as shown in Figure REF (b).
<FIGURE>