Multimodal BERT. Extending BERT [1]} to vision-and-language has been actively studied. Existing work typically adopt early fusion [2]}, [3]}, [4]}, [5]}, [6]}, [7]}, [8]}, [9]} or mid fusion [10]}, [11]}, [12]}, [13]} without thorough validation, and they train only visual components while relying on a language-pretrained BERT.
Although there have been some efforts to leverage the Transformer architecture [14]} for audio and visual inputs [15]}, [16]}, our approach is the first to demonstrate multimodal audio-visual BERT trained from scratch in an end-to-end manner. This is enabled by our novel parameter reduction technique, which is one of our main technical contributions.
