@inproceedings{
yang2024spatiotemporal,
title={Spatiotemporal Fine-grained Video Description for Short Videos},
author={Te Yang and Jian Jia and Bo Wang and Yanhua cheng and Yan Li and Dongze Hao and Xipeng Cao and Quan Chen and Han Li and Peng Jiang and Xiangyu Zhu and Zhen Lei},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=SWFpL4G2pf}
}
In the mobile internet era, short videos are inundating people's lives. However, research on visual language models specifically designed for short videos has yet to be fully explored. Short videos are not just videos of limited duration. The prominent visual details and high information density of short videos differentiate them to long videos. In this paper, we propose the SpatioTemporal Fine-grained Description (STFVD) emphasizing on the uniqueness of short videos, which entails capturing the intricate details of the main subject and fine-grained movements. To this end, we create a comprehensive Short Video Advertisement Description (SVAD) dataset, comprising 34,930 clips from 5,046 videos. The dataset covers a range of topics, including 191 sub-industries, 649 popular products, and 470 trending games. Various efforts have been made in the data annotation process to ensure the inclusion of fine-grained spatiotemporal information, resulting in 34,930 high-quality annotations. Compared to existing datasets, samples in SVAD exhibit a superior text information density, suggesting that SVAD is more appropriate for the analysis of short videos. Based on the SVAD dataset, we develop SVAD-VLM to generate spatiotemporal fine-grained description for short videos. We use a prompt-guided keyword generation task approach to efficiently learn key visual information. Moreover, we also utilize dual visual alignment to exploit the advantage of mixed-datasets training. Experiments on SVAD dataset demonstrate the challenge of STFVD and the competitive performance of proposed method compared to previous ones.
Spatiotemporal Fine-grained Video Description for Short Videos
Te Yang Jian Jia Bo Wang Yanhua cheng Yan Li Dongze Hao Xipeng Cao Quan Chen Han Li Peng Jiang Xiangyu Zhu Zhen Lei