We leverage the state-of-the-art Hybrid Attention Network (HAN) [1]} as the base for our network architecture.
Given separate streams of pre-trained audio (VGGish [2]}) and visual (ResNet-152 [3]} and 3DResNet [4]} features), HAN uses cross-modal attention to learn modality contextualized features.
As described in Figure REF  these features are then provided as an input to a shared linear layer that produces joint features for audio and video before finally passing them for Attentive MMIL Pooling.
