Inspired by the success of BERT [1]} in natural language processing, there has been a surge of interest in vision-and-language representation learning, by pre-training on the large-scale multimodal data with a transformer-based [2]} model architecture, then fine-tuning on various downstream tasks [3]}, [4]}, [5]}, [6]}, [7]}, [8]}, [9]}, [10]}, [11]}, [12]}, [13]}, [14]}, [15]}, [16]}, [17]}.
While different works use different architectures (e.g. two-stream [6]}, [17]}, [20]} vs. single-stream [7]}, [8]}, [5]}, [4]}), features (e.g. regions [25]} vs. grids [26]}), backbones (e.g. ConvNets [26]} vs. Transformers [28]}) etc.,
the shared goal of visual-linguistic pre-training is to exploit large-scale, paired image-and-text corpora [29]}, [30]}, [31]}, [32]}, [33]}, [34]} and obtain models that are implicitly pre-built with the appropriate correspondences to understand the multi-modal data.
The level of understanding is benchmarked with multiple multimodal downstream tasks [35]}, [36]}, [37]}, [38]}.
<FIGURE>