In language understanding, strong progress has been made in large-scale learning of contextualized language representations using Transformers [1]}, [2]}, [3]}, [4]}, [5]}, [6]}, [7]}, [8]}.
Riding on the success of Transformers, several recent works have extended it to the multimodal setting by adding an additional vision module to the Transformer framework [9]}, [10]}. However, these models are typically not end-to-end trained; they rely on a language-pretrained BERT [6]}, which is fixed throughout, and train only the visual components. While the pretrained BERT helps accelerate convergence and brings reliable extra supervision signal to the vision component, this partial learning setup can be undesirable if the text data comes from different distributions (of topics, dialects, or foreign languages) or if we want to apply it to different modalities (e.g., audio-visual). Unfortunately, end-to-end training of such multimodal Transformer architectures is challenging for most existing compute environments due to the excessive memory requirement.
<FIGURE>