Transformers are a family of neural networks solely based on attention mechanisms to learn representative embedding for various data.
The Transformer model is first proposed in [1]}, which focuses on the machine translation tasks in natural language processing (NLP).
BERT [2]} further applies transformers to multiple deep language understanding tasks by introducing the pre-training technique.
Following Transformer and BERT, a large number of variant works are presented and reach state-of-the-art results on various NLP tasks [3]}, [4]}, [5]}.
Very recently, the transformer model is extended to the field of computer vision (CV) [6]}.
For instance, DETR [7]} first leverages transformers on the object detection task.
ViT [8]} splits an image into multiple patches and uses a pure transformer model to learn the representation for image classification directly.
SETR [9]} utilizes a ViT-like encoder for feature extraction and adopts a multi-level feature aggregation module for pixel-wise image segmentation.
For further details about transformers on NLP and CV please see related surveys [10]}, [11]}.
