We emphasize the importance of feeding \(\mathbf {w}^{v}_{0:T}\)  and \(\mathbf {w}^{a}_{0:T}\)  one after another. An alternative would be concatenating them before feeding them to \(h_{AV}\)  and obtaining an output \(\mathbf {z}_{0:T}\)  (instead of \(\mathbf {z}_{0:(2T+1)}\) ). However, this restricts the Transformer to access audio-visual embeddings only from the same time slices, which could be problematic when there is a temporally asynchronous relationship between the two modalities (e.g., a visual clip matches with sound captured a few times steps before) [1]}, [2]}. By arranging the two sequences one after the other, the Transformer can mix-and-match appropriate audio-visual embeddings in an asynchronous manner. Another practical concern with the alternative approach is that it significantly increases the model size; the weight matrices \(W_q, W_k, W_v\)  grow quadratically with the input feature dimension \(D\) . Serializing the input resolves both issues.
