TITLE: MSCTD: A Multimodal Sentiment Chat Translation Dataset 

ABSTRACT: Multimodal machine translation and textual chat translation have received
considerable attention in recent years. Although the conversation in its
natural form is usually multimodal, there still lacks work on multimodal
machine translation in conversations. In this work, we introduce a new task
named Multimodal Chat Translation (MCT), aiming to generate more accurate
translations with the help of the associated dialogue history and visual
context. To this end, we firstly construct a Multimodal Sentiment Chat
Translation Dataset (MSCTD) containing 142,871 English-Chinese utterance pairs
in 14,762 bilingual dialogues and 30,370 English-German utterance pairs in
3,079 bilingual dialogues. Each utterance pair, corresponding to the visual
context that reflects the current conversational scene, is annotated with a
sentiment label. Then, we benchmark the task by establishing multiple baseline
systems that incorporate multimodal and sentiment features for MCT. Preliminary
experiments on four language directions (English-Chinese and English-German)
verify the potential of contextual and multimodal information fusion and the
positive impact of sentiment on the MCT task. Additionally, as a by-product of
the MSCTD, it also provides two new benchmarks on multimodal dialogue sentiment
analysis. Our work can facilitate research on both multimodal chat translation
and multimodal dialogue sentiment analysis.
Multimodal machine translation and textual chat translation have received
considerable attention in recent years. Although the conversation in its
natural form is usually multimodal, there still lacks work on multimodal
machine translation in conversations. In this work, we introduce a new task
named Multimodal Chat Translation (MCT), aiming to generate more accurate
translations with the help of the associated dialogue history and visual
context. To this end, we firstly construct a Multimodal Sentiment Chat
Translation Dataset (MSCTD) containing 142,871 English-Chinese utterance pairs
in 14,762 bilingual dialogues and 30,370 English-German utterance pairs in
3,079 bilingual dialogues. Each utterance pair, corresponding to the visual
context that reflects the current conversational scene, is annotated with a
sentiment label. Then, we benchmark the task by establishing multiple baseline
systems that incorporate multimodal and sentiment features for MCT. Preliminary
experiments on four language directions (English-Chinese and English-German)
verify the potential of contextual and multimodal information fusion and the
positive impact of sentiment on the MCT task. Additionally, as a by-product of
the MSCTD, it also provides two new benchmarks on multimodal dialogue sentiment
analysis. Our work can facilitate research on both multimodal chat translation
and multimodal dialogue sentiment analysis.
MSCTD: A Multimodal Sentiment Chat Translation Dataset
http://arxiv.org/abs/2202.13645v1