Previous work in multimodal visual-and-language tasks [1]}, [2]} point out that using partially fixed Transformers of different modalities is detrimental to multimodal representation learning (c.f., [3]}, [4]}). We make the same observation in our audio-visual learning scenario. We compare two variants of Multi-6 Part in Table REF  (c), each of which pretrains only the audio (or visual) CNN/Transformer in the first half of pretraining stage and then continues pretraining the remaining weights while fixing the weights of the audio (or visual) CNN/Transformer in the second half. This leads to inferior performance (audio-fixed 62.8% and visual-fixed 63.1% vs. ours 67.5%), which is consistent with the results reported in [1]}, [2]}.
