CNN has been quite successful in image recognition. Since the accuracy of AlexNet[1]} on ImageNet[2]} has greatly surpassed traditional methods, various convolutional architectures have been proposed to improve the accuracy of recognition. For example, ResNet[3]} and DenseNet[4]} have performed quite well on various recognition tasks using deep network structures with shortcut connections, and their pre-trained models on ImageNet have also been successfully transferred to various computer vision tasks. EfficientNet[5]} further discusses the depth, width and input resolution of convolutional networks to optimize the network structure. The method successfully maintains high accuracy with a small number of parameters.
Since the publication of Transformer[6]}, the self-attention mechanism has been widely used in the field of natural language processing and computer vision, and its performance is quite good. For example, Vit[7]} achieved 90.7% accuracy on ImageNet, which convert the patch of the image into a token through linear transformation, and complete the global information exchange through the self-attention mechanism. However, its disadvantage is that there is no hierarchical expression ability for the features of local regions. In SwinTransformer[8]}, the extraction of local region features at different scales is done through a multi-layer self-attention structure. Because of the success of the above architectures, we will use these models as the backbone network to test the capabilities of the proposed modules.
<FIGURE>