On the other hand, to fuse the extracted cross-modal feature maps, we devise a generic fusion block based on the efficient self-attention [1]}. As shown in the bottom of Fig. REF , the fusion block consists of two self-attention modules, which can capture and reinforce the useful components in the global receptive field. In the self-attention module, we reshape the feature maps of \(\mathbb {R}^{H\times W \times C}\)  to vectors of \(\mathbb {R}^{N \times C}\) , where \( N = H\times W \) . Then, we use the linear layers to encode the vectors into quary Q, key K, and value V. We get the attention map by a matrix multiplication \(K^{T}V\) , then we acquire the final attention result \(Q\mbox{Softmax}(K^{T}V)\) . The strengthened component is obtained by element-wise multiplying the attention result and the input feature maps. To reserve the detail information, we further introduce a residual connection.
<FIGURE>