For the first issue, previous works directly concatenate CNNs features [1]} or use RNNs [2]}, [3]} to model the inter-image relationships. However, when assigning different orders of the input images there can output different group representations, which makes both training and inferring procedure unstable. Recent studies  [4]}, [5]}, [6]} try to alleviate this limitation by applying some sophisticated modification on CNNs architectures. Unfortunately, their efforts do not address the inherent deficiencies of CNNs and RNNs in sequential order modeling.
To better model the inter-image relationships, we propose to employ the Transformers [7]}, which is a widely used sequence to sequence model in Natural Language Processing (NLP) [23]. The self-attention mechanism is designed to learn all pairwise similarities between the input sequence, which empowers Transformers great ability to capture long-range dependencies. Besides, a Transformer model itself is invariant with respect to re-orderings of the input [7]}, [9]}. These characteristics of Transformers make them naturally suitable for modeling the inter-image relationships across multiple images. Essentially, intra-image saliency and inter-image consistency are both concerned with relationship modeling: intra-image saliency is to learn the pixel-level relationship within a single image and inter-image consistency is to learn the relationship between images. Thus, we construct Transformer based structures for both intra-image saliency and inter-image consistency modeling.
<FIGURE>