@inproceedings{
chen2024egocentric,
title={Egocentric Vehicle Dense Video Captioning},
author={Feiyu Chen and Cong Xu and Qi Jia and Yihua Wang and Yuhan Liu and Zhang Haotian and Endong Wang},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=QZCGSrjpJc}
}
Typical dense video captioning mostly concentrates on third-person videos, which are generally characterized by relatively delineated steps among events as seen in edited instructional videos. However, such videos do not genuinely reflect the way we perceive our real lives. Instead, we observe the world from an egocentric viewpoint and witness only continuous unedited footage. To facilitate further research, we introduce a new task, Egocentric Vehicle Dense Video Captioning, in classic first-person driving scenario. This is a multi-modal, multi-task project for a comprehensive understanding of untrimmed, egocentric driving videos. It consists of three sub-tasks that focus on event location, event captioning, and vehicle state estimation separately. For the purpose of accomplishing these tasks, it is necessary to deal with at least three challenges, those are extracting relevant ego-motion information, describing driving behavior and understanding the underlying rationale, as well as resolving the boundary ambiguity problem. In response, we devise corresponding solutions, encompassing a vehicle ego-motion learning strategy and a novel adjacent contrastive learning strategy, which effectively address the aforementioned issues to a certain extent. We validate our method by conducting extensive experiments on the BDD-X dataset, all of which show promising results and achieve new state-of-the-art performance on most metrics, which proves the effectiveness of our approach.
Egocentric Vehicle Dense Video Captioning
Feiyu Chen Cong Xu Qi Jia Yihua Wang Yuhan Liu Zhang Haotian Endong Wang