@inproceedings{
zhao2024does,
title={Does Video-Text Pretraining Help Open-Vocabulary Online Action Detection?},
author={Qingsong Zhao and Yi Wang and Jilan Xu and Yinan He and Zifan Song and Limin Wang and Yu Qiao and Cairong Zhao},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=PWzB2V2b6R}
}
Video understanding relies on accurate action detection for temporal analysis. However, existing mainstream methods have limitations in real-world applications due to their offline and closed-set evaluation approaches, as well as their dependence on manual annotations. To address these challenges and enable real-time action understanding in open-world scenarios, we propose OV-OAD, a zero-shot online action detector that leverages vision-language models and learns solely from text supervision. By introducing an object-centered decoder unit into a Transformer-based model, we aggregate frames with similar semantics using video-text correspondence. Extensive experiments on four action detection benchmarks demonstrate that OV-OAD outperforms other advanced zero-shot methods. Specifically, it achieves 37.5\% mean average precision on THUMOS’14 and 73.8\% calibrated average precision on TVSeries. This research establishes a robust baseline for zero-shot transfer in online action detection, enabling scalable solutions for open-world temporal understanding. The code will be available for download at \url{https://github.com/OpenGVLab/OV-OAD}.
Does Video-Text Pretraining Help Open-Vocabulary Online Action Detection?
Qingsong Zhao Yi Wang Jilan Xu Yinan He Zifan Song Limin Wang Yu Qiao Cairong Zhao