E-commerce application as one of the largest multi-modal downstream scenarios has facilitated person's lives.
And there are many product-based tasks in the application scenario, such as similar item retrieval [1]}, online commodity recommendation [2]} and identical product match for cross-price comparison [3]}.
With a wide variety of downstream E-commerce applications in the real world, we focus solely on the pretraining of multi-modality data of products.
In the general scenarios, multi-modal vision-language pre-training model [4]}, [5]}, [6]}, [7]}, [8]}, [9]}, [10]}, [11]}, [12]} such as CLIP [9]} and VilBert [4]} have shown superior performances on the diverse downstream tasks, such as zero-shot classification [15]}, cross-modal retrieval [16]} and open-world detection [17]}.
They learn visual and textual embedding features from a huge number of image-text pairs acquired from different sources and have quite generalization capability and robustness.
The fundamental methodology behind these models is either contrastive semantic alignment of images and texts or specific network architectures designed for better common space learning.
However, these models mentioned above focus more on the attention regions adaptively learned by themselves and are easily influenced by the contents that are not related to the visual objects from each modality, leading to ignoring the importance of concept knowledge, i.e., words or phrases with a certain meaning.
As an illustration in Figure REF , we expect the trained multi-modal network to pay more attention to four typical objects (Fair Water, Facial Cream, Clearner and Clear Lotion) and to ignore extra content (Four piece Set) in a more fine-grained application setting.
If the multi-modal network is trained directly by visual-language contrastive learning, the learned attention region may be SK-II, Cream Cleaner, Lotion Four-piece,
which is unsuitable and incorrect to achieve the expected objects of attention.
In the past few years, there has been increasing attention in designing the knowledge injection [18]}, [19]}, [20]}, [21]} for natural language processing (NLP) tasks to demonstrate and enchance the attention effectiveness of NLP models.
These models are more based on a well fully annotated knowledge graph, which is time-cost and laborious for a new task.
In this paper, we balance the trade-off between knowledge base construction and knowledge extraction, and propose a simple but effective entity-graph enhanced visual-language pretraining model, which explicitly inject the concept knowledge generated from the caption data into the multi-modal model via a special knowledge graph (entity-graph).
<FIGURE>