Studying the behaviors of attention mechanisms is analyzing inside the block, from a micro perspective. Next, we hope to study from a macro perspective of deep networks, such as studying the similarity between feature maps across different layers via the CKA similarity [1]}. Results of CKA similarity between feature representations of different layers of three pre-trained models, supervised model (DeiT), contrastive learning model (MoCo v3), and MIM model (SimMIM) with ViT-B as the backbone, are shown in Figure REF . We surprisingly find that in MIM models, the representation structures of different layers are almost the same, that their CKA similarities are all very large (e.g., [0.9, 1.0]). But for supervised models, as in [2]}, different layers learn different representation structures, that their CKA similarities vary greatly (e.g., [0.5,1.0]). Different from previous visualizations, MoCo v3 behaves similarly to SimMIM in this case.
<FIGURE>