HAN refers to the current state-of-the-art system for AVVP [1]}. “Base" varies from HAN in two ways: (i) Base uses separate self attention modules for the two modalities while HAN uses a shared module (ii) HAN performs cross modal attention directly over encoded features while Base does modality fusion over self attended features.These modifications in “Base" were beneficial when used in conjunction with our Adv, Skip techniques.
“Skip", “Adv" and “Gcaa" refer to the skip connections, adversarial training and global context-aware attention techniques we described in Section . “Pretrained" and “Pretrain E2E" refers to the two pretraining variants described earlier in this section. We observe that our proposed techniques lead to significant improvements over the state-of-the-art HAN network on all five metrics. Inclusion of “Adv+Skip+Gcaa" improves segment-level and event-level F1 scores on three of the five evaluation metrics.
