Images are observed to exhibit strong locality: pixels near each other tend to be highly correlated [1]}, motivating the use of local priors in a wide range of visual perception architectures [2]}, [3]}, [4]}, [5]}, [6]}. In the era of Vision Transformers, the usefulness of local priors has still undergone rich discussions and trials [7]}, [6]}, [9]}. Thus it is valuable to investigate whether MIM models bring the locality inductive bias to the models. We do this by computing averaged attention distance in each attention head of each layer.
<FIGURE>