@inproceedings{
liu2024openset,
title={Open-Set Video-based Facial Expression Recognition with Human Expression-sensitive Prompting},
author={Yuanyuan Liu and Yuxuan Huang and Shuyang Liu and Yibing Zhan and Zijing Chen and Zhe Chen},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=VkiacHTDs0}
}
In Video-based Facial Expression Recognition (V-FER), models are typically trained on closed-set datasets with a fixed number of known classes. However, these models struggle with unknown classes common in real-world scenarios. In this paper, we introduce a challenging Open-set Video-based Facial Expression Recognition (OV-FER) task, aiming to identify both known and new, unseen facial expressions. While existing approaches use large-scale vision-language models like CLIP to identify unseen classes, we argue that these methods may not adequately capture the subtle human expressions needed for OV-FER. To address this limitation, we propose a novel Human Expression-Sensitive Prompting (HESP) mechanism to significantly enhance CLIP's ability to model video-based facial expression details effectively. Our proposed HESP comprises three components: 1) a textual prompting module with learnable prompts to enhance CLIP's textual representation of both known and unknown emotions, 2) a visual prompting module that encodes temporal emotional information from video frames using expression-sensitive attention, equipping CLIP with a new visual modeling ability to extract emotion-rich information, and 3) an open-set multi-task learning scheme that promotes interaction between the textual and visual modules, improving the understanding of novel human emotions in video sequences. Extensive experiments conducted on four OV-FER task settings demonstrate that HESP can significantly boost CLIP's performance (a relative improvement of 17.93% on AUROC and 106.18% on OSCR) and outperform other state-of-the-art open-set video understanding methods by a large margin. Code is available at https://github.com/cosinehuang/HESP.
Open-Set Video-based Facial Expression Recognition with Human Expression-sensitive Prompting
Yuanyuan Liu Yuxuan Huang Shuyang Liu Yibing Zhan Zijing Chen Zhe Chen