Inference. During text-image matching inference, a nearest-neighbor search is usually performed to obtain a ranking for each of the queries. It has been pointed out by previous works [1]}, [2]}, [3]} that hubs will emerge in such high-dimensional space and nearest neighbor search can be problematic for this need. Qualitatively, the hubness problem means a small portion of queries becoming “popular” nearest neighbor in the search space. Hubs harm model's performance as we already know that the predicted text-image correspondence should be a bipartite matchingIn Graph Theory, a set of edges is said to be a matching if none of the edges share a common endpoint.. In experiments, we show that the hubness problem is the primary source of error for inference.
Though has not attracted enough attention in text-image matching, hubness problem has been extensively studied in Bilingual Lexicon Induction (BLI) which aims to find a matching between two sets of bilingual word vectors. We thus propose to use similar tools during the inference phase of text-image matching. Specifically, we experiment with Inverted Softmax (Is) [4]} and Cross-modal Local Scaling (Csls) [5]} to mitigate the hubness problem in text-image embeddings.
