Backbone Architectures. Masked image modeling is mostly studied in the Transformer architectures, thus the major understandings and experiments in this paper are performed on Vision Transformers (ViT) [1]} and Swin Transformers [2]}, [3]}. Due to the simple and clear architecture designs of ViT, most of the visualizations are performed on ViT, shown in Section . Due to the general-purpose property of Swin Transformer, most of the experiments on different downstream tasks are conducted on Swin Transformer, shown in Section .
