Cross-Modal Schemes. In Table REF , we compare EGE-CMP to many commonly used self-supervised cross-modal pretraining approaches, including state-of-art single- and two-stream Vision-language models, as well as a state-of-art zero-shot classification model, i.e., CLIP [1]}.
CLIP* refers to a CLIP-like architecture that encodes images and text separately and is trained with a contrastive learning.
Notably, EGE-CMP exceeds all of these benchmarks on the majority of three major retrieval criteria.
Two-stream models, such as LXMERT [2]}, ViLBERT [3]}, and CLIP*, do not perform better than single-stream models, indicating that the manner of fusion of cross-modal information is an important component.
Please note that by using entity knowledge, our EGE-CMP outperforms CAPTURE when employing the hybrid transformer architecture.
We attribute EGE-CMP's better performance to its entity knowledge injection. Finally, we examine the influence of various layer types in Section REF .
