Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training
In this paper, we introduce Cross-View Language Modeling, a simple and
effective language model pre-training framework that unifies cross-lingual
cross-modal pre-training with shared architectures and objectives. Our approach
is motivated by a key observation that cross-lingual and cross-modal
pre-training share the same goal of aligning two different views of the same
object into a common semantic space. To this end, the cross-view language
modeling framework considers both multi-modal data (i.e., image-caption pairs)
and multi-lingual data (i.e., parallel sentence pairs) as two different views
of the same object, and trains the model to align the two views by maximizing
the mutual information between them with conditional masked language modeling
and contrastive learning. We pre-train CCLM, a Cross-lingual Cross-modal
Language Model, with the cross-view language modeling framework. Empirical
results on IGLUE, a multi-lingual multi-modal benchmark, and two multi-lingual
image-text retrieval datasets show that while conceptually simpler, CCLM
significantly outperforms the prior state-of-the-art with an average absolute
improvement of over 10%. Notably, CCLM is the first multi-lingual multi-modal
model that surpasses the translate-test performance of representative English
vision-language models by zero-shot cross-lingual transfer.