We utilize several pretext tasks to enable the self-supervised learning of EGE-CMP and benefit from large-scale multi-modal data.
For modality-wise feature learning, we adopt three masked multi-modal modeling tasks, i.e., Masked Language Modeling task (MLM), Masked Region Prediction task (MRP) and Masked Entity Modeling (MEM) following the standard BERT [1]} and VisualBERT [2]}.
Masked Language Modeling reconstructs the masked token information to improve language understanding with the aid of visual and knowledge representations. Masked Region Prediction recovers the masked visual features to enhance the visual object perception. Masked Entity Modeling improves the semantic knowledge understanding, empowering models to make sense of common concepts with real significance.
