@inproceedings{
oncescu2024dissecting,
title={Dissecting Temporal Understanding in Text-to-Audio Retrieval},
author={Andreea-Maria Oncescu and Joao F. Henriques and A. Sophia Koepke},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=HTZy9hpoYV}
}
Recent advancements in machine learning have fueled research on multimodal interactions, such as for instance text-to-video and text-to-audio retrieval tasks. These tasks require models to understand the semantic content of input videos, including objects, sounds and characters. The models also need to learn their spatial arrangement and the temporal relationships of sounds. In this work, we tackle the temporal ordering of sounds, which is an understudied problem in the context of text-to-audio retrieval. In particular, we dissect the temporal understanding capabilities of a state-of-the-art model for text-to-audio retrieval on the AudioCaps dataset. Additionally, we introduce a synthetic text-audio dataset that provides a controlled setting for evaluating the temporal understanding of recent models. Lastly, we investigate a new loss function that encourages text-audio models to focus on the temporal ordering of events.
Dissecting Temporal Understanding in Text-to-Audio Retrieval
Andreea-Maria Oncescu Joao F. Henriques A. Sophia Koepke