Local Feature Embedding. We feed each of \(T\)  video clips to a visual CNN \( f_{V}(\mathbf {v}_{t})\)  to obtain \(\mathbf {x}^{v}_{1:T} \in \mathbb {R}^{T \times D}\) , and each audio stream to an audio CNN \(f_{A}(\mathbf {a}_{t})\)  to obtain \(\mathbf {x}^{a}_{1:T}\in \mathbb {R}^{T \times D}\) .For notational simplicity, we drop the subscripts to refer to the entire sequence unless distinction is necessary. Intuitively, the CNN outputs are temporally local embeddings as they have access to only a short-range temporal window of the entire video sequence. Thus, they are suitable for representing short-range atomic actions (e.g., sit down, raise arms) that constitute long-range events (e.g., gym workout). We use the SlowFast network [1]} with a ResNet-50 backbone [2]} as a visual CNN \(f_{V}\) , and a ResNet-50 as an audio CNN \(f_{A}\) . The weights of both CNNs are randomly initialized and trained end-to-end with the Transformer layers.
