[leftmargin=0.3in]
We propose a new masked self-supervised transformer approach called MST. It makes full use of self-attention map to guide the masking of local patches, thus enhancing the understanding of local context semantics in pre-training without damaging the crucial structure.

Our method can effectively recover the spatial information of the image by a global image decoder, which is vital for the downstream dense prediction task and greatly improves the versatility and scalability of the pre-training model.

Extensive experiments demonstrate the effectiveness and transfer ability of our method. Specifically, the results on ImageNet [1]}, MS COCO [2]} and Cityscapes [3]} show that our method outperforms previous state-of-the-art methods.

