@inproceedings{
du2024reversed,
title={Reversed in Time: A Novel Temporal-Emphasized Benchmark for Cross-Modal Video-Text Retrieval},
author={Yang Du and Yuqi Liu and Qin Jin},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=QPbeKwJNMb}
}
Video-text retrieval is an important task in the multimodal understanding field.Temporal understanding makes video-text retrieval more challenging than image-text retrieval. However, we find that the widely used video-text benchmarks have shortcomings in assessing model's retrieval ability, especially in temporal understanding, causing large-scale image-text pre-trained models can already achieve comparable zero-shot performance with video-text pre-trained models.In this paper, we introduce RTime, a novel temporal-emphasized video-text retrieval dataset, constructed through a top-down three-step process. We first obtain videos of actions or events with significant temporality, and then reverse these videos to create harder negative samples.  We recruit annotators to judge the significance and reversibility of candidate videos, and then write captions for qualified videos. We further adopt GPT-4 to extend more captions based on human-written captions. Our RTime dataset currently consists of 21k videos with 10 captions per video, totalling about 122 hours. Based on RTime, we propose three retrieval benchmark tasks: RTime-Origin, RTime-Hard, and RTime-Binary.We further enforce leveraging harder-negatives in model training, and benchmark a variety of video-text models on RTime. Extensive experiment analysis proves that RTime indeed poses new and higher challenges to video-text retrieval.We will release our RTime benchmarks to further advance  video-text retrieval and multimodal understanding research.
Reversed in Time: A Novel Temporal-Emphasized Benchmark for Cross-Modal Video-Text Retrieval
Yang Du Yuqi Liu Qin Jin