Traditionally, the text-to-visual cross-modal retrieval is solved by the joint embedding [1]}, [2]}. It maps the features from different modals into the same feature space and thus their similarity can be directly measured by their Euclidean distance in the joint feature space. In the training phase, it seeks to enlarge the distance between the text and its irrelevant images/videos in the joint feature space, and meanwhile minimize the distance between the text and its relevant images/videos in the joint feature space. One of the amazing properties of the joint embedding is that it generates the global text/visual features, and is well compatible with approximate nearest neighbor (ANN) search techniques such as inverted indexing or graph/tree-based indexing. It is hence quite efficient for large-scale cross-modal retrieval applications.
