@inproceedings{
pei2024emotion,
title={Emotion Recognition in {HMD}s: A Multi-task Approach Using Physiological Signals and Occluded Faces},
author={Yunqiang Pei and Jialei Tang and Qihang Tang and Mingfeng Zha and Dongyu Xie and Guoqing Wang and Zhitao Liu and Ning Xie and Peng Wang and Yang Yang and Heng Tao Shen},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=crJyt9Cign}
}
Prior research on emotion recognition in extended reality (XR) has faced challenges due to the occlusion of facial expressions by Head-Mounted Displays (HMDs). This limitation hinders accurate Facial Expression Recognition (FER), which is crucial for immersive user experiences. This study aims to overcome the occlusion challenge by integrating physiological signals with partially visible facial expressions to enhance emotion recognition in XR environments. We employed a multi-task approach, utilizing a feature-level fusion to fuse Electroencephalography (EEG) and Galvanic Skin Response (GSR) signals with occluded facial expressions. The model predicts valence and arousal simultaneously from both macro-and micro-expression. Our method demonstrated improved accuracy in emotion recognition under partial occlusion conditions. The integration of temporal physiological signals with other modalities significantly enhanced performance, particularly for half-face emotion recognition. The study presents a novel approach to emotion recognition in XR, addressing the limitations of facial occlusion by HMDs. The findings suggest that physiological signals are vital for interpreting emotions in occluded scenarios, offering potential for real-time applications and advancing social XR applications.
Emotion Recognition in HMDs: A Multi-task Approach Using Physiological Signals and Occluded Faces
Yunqiang Pei Jialei Tang Qihang Tang Mingfeng Zha Dongyu Xie Guoqing Wang Zhitao Liu Ning Xie Peng Wang Yang Yang Heng Tao Shen