To go further than a simple fusion, we implement a lateral connection between visual and audio paths with the FiLM layer [1]}. With the FiLM layer, the visual modality influences the audio modality. Greater importance is given to some of the audio feature maps based on the visual information. The FiLM layer is placed directly at the output of the audio feature extractor before reducing feature maps into vectors.
<FIGURE>