Image-Text Retrieval
We collect 1M Catalog Posts for the text-to-image retrieval (T2I) and the image-to-text retrieval (I2T) tasks.
We split a 10K subset for evaluation and leave the rest for model training.
There are two ways to perform the image-text retrieval.
One is using the ITM head to predict the matching score between the input image-text pair and rank the scores of all pairs [1]}, [2]}, [3]}.
One is computing the similarity between image encoder and text encoder and picks the best match without deeper fusion between the two modalities as [4]}, [5]}, [6]}.
While the first approach may achieve better performance [7]}, its computational cost is huge (quadratically increasing w.r.t the size of retrieval pool), which is not applicable in practice.
Thus we follow the second approach to simply use our image and text encoders of CommerceMM for the retrieval tasks.
We apply contrastive learning loss during the fine-tuning.
Recall@1 is used to measure the retrieval performance.
<FIGURE>