Parameter Sharing Schemes. Our parameter reduction scheme reduces the model size from 155M to 31M (by 80%) (Table REF  (a)). We reduce the model size by sharing weights across Transformers and across layers. We validate these ideas in two sets of experiments. Table REF  (c) compares cross-Transformer weight sharing schemes. We use Multi-6 that uses all three Transformers with 6 layers each, and compare four methods that correspond to Figure REF  (a)-(d). Note that No sharing is too large to fit in a Tesla V100 GPU (16GB) even with 2 samples, so we define Multi-2 that uses three Transformers with 2 layers each, and with the reduced number of attention heads \(A\)  to 5, the feature dimension \(D\)  to 320 and the intermediate dimension \(E\)  to 1280.
We see that our proposed approach, Part, achieves the best performance with the least number of parameters. One might ask how Part leads to a smaller model when All shares all the weights across Transformers: We decompose weights \(W = U \Sigma V^\top \)  with low-rank approximation and share only \(U\)  across Transformers, while the \(\Sigma V^\top \)  part learns modality-specific dynamics. Table REF  (d) compares cross-layer weight sharing schemes using the visual Transformer with either 2 (Vis-2) or 6 (Vis-6) layers. The results show that sharing weights across layers does not hurt the performance, confirming the observations by [1]} in the audio-visual setting.
