@inproceedings{
tu2024multimodal,
title={Multimodal Emotion Recognition Calibration in Conversations},
author={Geng Tu and Feng Xiong and Bin Liang and Hui Wang and Xi Zeng and Ruifeng Xu},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=9KJsCWEJVe}
}
Multimodal Emotion Recognition in Conversations (MERC) aims to identify the emotions conveyed by each utterance in a conversational video. Current efforts focus on modeling speaker-sensitive context dependencies and multimodal fusion. Despite the progress, the reliability of MERC methods remains largely unexplored. Extensive empirical studies reveal that current methods suffer from unreliable predictive confidence. Specifically, in some cases, the confidence estimated by these models increases when a modality or specific contextual cues are corrupted, defining these as uncertain samples. This contradicts the foundational principle in informatics, namely, the elimination of uncertainty. Based on this, we propose a novel calibration framework CMERC to calibrate MERC models without altering the model structure. It integrates curriculum learning to guide the model in progressively learning more uncertain samples; hybrid supervised contrastive learning to refine utterance representations, by pulling uncertain samples and others apart; and confidence constraint to penalize the model on uncertain samples. Experimental results on two datasets show that the CMERC significantly enhances the reliability and generalization capabilities of various MERC models, surpassing the state-of-the-art methods.
Multimodal Emotion Recognition Calibration in Conversations
Geng Tu Feng Xiong Bin Liang Hui Wang Xi Zeng Ruifeng Xu