In the language models, the knowledge-based model aims to incorporate knowledge, concepts and relation into the pretrained models [1]}, [2]}, which proved to be beneficial to language understanding [3]}.
The existing methods can be coarsely broadly classified into two types: implicit knowledge modeling and explicit knowledge injection.
The former attempts on implicit knowledge modeling usually consist of entity-level masked modeling [4]}, [5]}, [6]}, entity-based replacement prediction [7]}, knowledge embedding loss as regularization [8]} and universal knowledge-text prediction [9]}, [10]}.
In contrast to implicit knowledge modeling, the latter separately maintains a group of parameters for representing structural knowledge. Such methods [4]}, [5]}, [7]}, [9]}, [10]}, [6]} usually require a heterogeneous information fusion component to fuse multiple types of features obtained from the text and knowledge graphs.
In our setting, it is very hard to build a knowledge graph with good annotations from scratch to fit the multi-modal data.
In the paper, with the purpose of knowledge modeling and visual-language pretraining, and improving the representation capability of visual-text models, we explore the solutions for knowledge enhanced cross-modal pretraining, where knowledge is represented as an entity parsed from the original caption data and the relationship between different pieces of knowledge are learned via node-based and subgraph-based ranking losses.
