From the previous visualizations on Vision Transformers (ViT) and Swin Transformers, we find that the MIM pre-training brings the locality inductive bias and larger diversity on attention heads to the trained models comparing to the supervised counterpart, which may benefit the optimization of the trained models on downstream tasks. This reminds us that large-kernel ConvNets [1]} without special designs still face the optimization issue, and need the re-parametrization trick with small kernels to bring the locality back and help them optimize. Thus it is valuable to know whether the masked image modeling (MIM) as pre-training could help the large-kernel ConvNets to optimize without the re-parametrization trick.
Thanks to the general applicability of SimMIM [2]}, we could also perform experiments and visualizations on large-kernel ConvNets [1]} with the MIM pre-training.
