In the paper, we propose a large-scale Entity-Graph Enhanced Cross-Modal Pretraining framework (EGE-CMP) to address these challenges.
Inspired by the effectiveness of knowledge injection [1]}, [2]}, [3]}, [4]}, [5]} into large-scale pretrained models for single modality, we model the cross-modal semantic alignment under the supervision of an entity-graph instead of directly using image and text interaction, which makes the trained model more focused on meaningful entity objects and thus reduce the influence of irrelevant content.
Specifically, we extract the noun entity directly from the caption data via named entity recognition (NER) tools and obtain the image proposals from the image data via the bottom-up-attention feature representation model [6]}.
Then, EGE-CMP is presented to capture the potential synergy of images, entities and title descriptions via entity-graph learning and some task-agnostic masking training.
We showcase that some widely used cross-modal pretraining methods [7]}, [8]}, [9]}, [10]} might be ineffective under the multi-instance setting without external knowledge guidance.
In contrast, EGE-CMP utilizes a hybrid-stream architecture that encodes data of different modalities separately and fuses them in a unified manner, which has been shown to be advantageous for our suggested purposed tasks.
Additionally, we introduce the cross-modal contrastive loss to compel EGE-CMP to align images, texts, and entities, so avoid the mismatch problem introduced by the unsuitable pretext task.
Besides that, an effective entity-graph enhanced module is adopted to enhance the learning capacity for the whole transformer model by enforcing the model to pay more attention to some entities with real semantic information.
In our entity graph, the node is each entity data and the edge is generated via the relation between each entity.
To encourage the text-encoder to be aware of actual key entities and improve the understanding ability of the transformer model, node-based and subgraph-based ranking losses are also introduced for the model training, requiring it to accurately distinguish the actual positive samples of entities according to the global entity ranking list.
As a result, our EGE-CMP model can focus more on the phrasesor words with real semantic meaning, and the text-image correlation can be aligned better.
Please note that compared with existing knowledge-enhanced pretraining methods based on the knowledge graph, our EGE-CMP model directly utilizes the knowledge directly extracted from the original caption data without any time-consuming and labor-intensive annotated data.
