The traditional cross-modal retrieval tasks are tackled by joint embedding  [1]}, [2]}. It maps the images and texts from their original feature space into a joint feature space so that they can be compared directly. They target to learn a mapping function, which maximizes the distance between an irrelevant image-text pair to be large , and meanwhile minimizes the distance between relevant image-text pairs. To learn the mapping function more effectively. VSE++ [3]} conducts hard negative sampling, which only penalizes the hardest negative sample in each mini-batch. Since the joint-embedding methods generate global features, it well supports feature indexing and thus is quite suitable for large-scale cross-modal retrieval. On the other hand, the global features cannot model the fine-level matching between words and local regions, and thus it might not be effective to capture relevance between an image and a text sentence.
