To further improvement the performance of text-vision retrieval methods. Several work  [1]}, [2]}, [3]} inject the cross-modal context when generating image/text representation. For instance, when computing the sentence representation, m-CNN [2]} concatenate the image's feature with words' features as the input of the 1D convolution.
In parallel, when computing image features, BCN [1]} uses text features to guide the generation of image features. Recently, inspired by the triumph achieved by BERT in many natural language processing (NLP) tasks [6]}, some vision-BERT methods  [7]}, [8]} are proposed, achieving excellent performance in many cross-modal tasks such as visual question answering (VQA), image/video captioning and cross-modal retrieval. Basically, the vision-BERT method can be coarsely grouped into two categories. The methods of the first category adopt a single-stream structure. It treats the bounding box features and word features without bias, and directly concatenate them as the input of the self-attention modules. The methods of the second category adopt a two-stream structure. In the text stream, the words features generate query vectors, and the bounding box features are used to generate value vectors and key vectors.
On the other hand, in the vision stream, the bounding box features generate query vectors, and the word features are used to generate value vectors and key vectors. In fact, both single-stream and two-stream vision-BERT methods are computationally costly. Therefore, it is impractical to directly use them to conduct large-scale cross-modal retrieval.
