Transformer is firstly proposed to address problems concerning natural language processing. It mainly consists of a self-attention module to capture long-range dependencies and feed-forward module to project features to new latent space. Recently, ViT [1]} is proposed to tackle nature image recognition tasks based on Transformer. It first splits the images into several non-overlapping patches, then utilizes Transformer to calculate global information between each token. An additional token is appended for recognition tasks. Inspired by this, many researchers try to leverage this fashion for many purposes, such as TransUNet [2]}, TransReID [3]}, etc.
Furthermore, there are some variations of Transformer, like Swin Transformer [4]} that employs shifted windows to calculate local self-attention, and PVT [5]} that combines the fashion of pyramid network with Transformer to capture features from multiple stages.
