@inproceedings{
xia2024perceive,
title={Perceive before Respond: Improving Sticker Response Selection by Emotion Distillation and Hard Mining},
author={Wuyou Xia and Shengzhe Liu and Qin Rong and Guoli Jia and Eunil Park and Jufeng Yang},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=9KB9f8iOlo}
}
In online chatting, people are increasingly favoring the use of stickers to supplement or replace text for replies, as sticker images can express more vivid and varied emotions. The Sticker Response Selection (SRS) task aims to predict the sticker image that is most relevant to the history dialogue context. Previous researches explore the semantic similarity between context and stickers, while ignoring the role of both unimodal and cross-modal emotional information. In this paper, we propose a “Perceive before Respond” training paradigm. We perceive the emotion of stickers through a knowledge distillation module, which acquires emotion knowledge from the existing large-scale sticker emotion recognition dataset and distills it into our framework to enhance the understanding of sticker emotion. To further distinguish stickers with similar subject elements within the same topic, we perform contrastive learning at both inter-topic and intra-topic levels to obtain discriminative and diverse sticker representations. In addition, we improve the hard negative sampling method for image-text matching based on cross-modal sentiment association, conducting hard sample mining from both semantic similarity and sentiment polarity similarity for sticker-to-dialogue and dialogue-to-sticker. Extensive experiments verify the effectiveness of each proposed component. Ablation experiments on different backbone networks demonstrate the generality of our approach. The code is provided in the supplement material and will be released to the public.
Perceive before Respond: Improving Sticker Response Selection by Emotion Distillation and Hard Mining
Wuyou Xia Shengzhe Liu Qin Rong Guoli Jia Eunil Park Jufeng Yang