As in Sec. REF , our model design allows us to dynamically change the text encoder and multimodal fusion encoder by assigning different \(K\)  and \(M\)  layers to each.
Previous works [1]}, [2]}, [3]}, [4]} show the modal-agnostic training can be beneficial to the multimodal understanding.
Our approach follows the same spirit.
At each training step, we randomly assign \(K\)  and \(M\)  transformer layers (while keeping the sum of \(K\)  and \(M\)  unchanged) to the text encoder and multimodal encoder, so that every layer can share the knowledge from both text and multimodal modalities.
