Therefore, we propose the Multi-level Attention Fusion network (MAFnet) to fuse dynamically visual and audio information for event recognition. This network dynamically associates a score to each modality and time window in videos. The score highlights a modality at a given time window that may be effective to recognize the event. We also propose to go further than the simple fusion by coupling modalities with a lateral connection between visual and audio paths of MAFnet. Moreover, to overcome the incompatibility of learning dynamics between visual and audio paths, we propose to randomly drop the update of the visual path during training. We evaluate our architecture on multiple datasets: Kinetics-Sound [1]}, UCF51 [2]} and AVE [3]}. In addition, we evaluate the contribution of each module of MAFnet with an ablation study.
