Nevertheless, the global features used in joint embedding methods might not be able to conduct fine-level matching between words and local regions of an image/video. In many cases, only few words in the query sentence are relevant with some small local regions in the video/video. Therefore, to conduct local matching more effectively, some methods [1]} rely on local features. Basically, they represent an image by a set of bounding box features and represent the query sentence as a set of word features. Then the relevance score is determined by the set-to-set matching.
Recently, inspired by the breakthrough achieved by BERT in NLP, many vision-based BERT methods [2]}, [3]}, [4]}, [5]} have been proposed, and achieve an excellent performance in cross-modal tasks like visual question answering (VQA), image captioning and cross-modal retrieval. In parallel, Baidu has launched a combo-attention network (CAN)  [6]} for an effective query-to-video retrieval in dynamic video adverting (DVA) platform. Nevertheless, since the cross-attention mechanism used in CAN takes expensive computation cost, it is impossible to use CAN to compute the similarity between the query and all video ads due to limited computing resource. Thus, previously, we first conducted the coarse-level retrieval through title-based retrieval and deployed CAN in the re-ranking phase for the search efficiency. Nevertheless, in this case, some relevant videos might be filtered out in the title-based retrieval due to their low-quality titles. A more reasonable way is to incorporate the cross-modal attention in the early stage.
