@inproceedings{
jia2024generating,
title={Generating Action-conditioned Prompts for Open-vocabulary Video Action Recognition},
author={Chengyou Jia and Minnan Luo and Xiaojun Chang and Zhuohang Dang and Mingfei Han and Mengmeng Wang and Guang Dai and Sizhe Dang and Jingdong Wang},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=XiHC7aSQ56}
}
Exploring open-vocabulary video action recognition is a promising venture, which aims to recognize previously unseen actions within any arbitrary set of categories. 
Existing methods typically adapt pretrained image-text models to the video domain, capitalizing on their inherent strengths in generalization. A common thread among such methods is the augmentation of visual embeddings with temporal information to improve the recognition of seen actions. Yet, they compromise with standard less-informative action descriptions, thus faltering when confronted with novel actions. Drawing inspiration from human cognitive processes, we argue that augmenting text embeddings with human prior knowledge is pivotal for open-vocabulary video action recognition. To realize this, we innovatively blend video models with Large Language Models (LLMs) to devise Action-conditioned Prompts. Specifically, we propose the Action-Centric generation strategy to produce a set of descriptive sentences that contain distinctive features for identifying given actions. Building upon this foundation, we further introduce a multi-modal action knowledge alignment mechanism to align concepts in video and textual knowledge encapsulated within the prompts. Extensive experiments on various video benchmarks, including zero-shot, few-shot, and base-to-novel generalization settings, demonstrate that our method not only sets new SOTA performance but also possesses excellent interpretability.
Generating Action-conditioned Prompts for Open-vocabulary Video Action Recognition
Chengyou Jia Minnan Luo Xiaojun Chang Zhuohang Dang Mingfei Han Mengmeng Wang Guang Dai Sizhe Dang Jingdong Wang