A Novel Graph based Multi modal Fusion Encoder for Neural Machine Translation
Multi modal neural machine translation (NMT) aims to translate source sentences into a target language paired with images. However, dominant multi modal NMT models do not fully exploit fine grained semantic correspondences between semantic units of different modalities, which have potential to refine multi modal representation learning. To deal with this issue, in this paper, we propose a novel graph based multi modal fusion encoder for NMT. Specifically, we first represent the input sentence and image using a unified multi modal graph, which captures various semantic relationships between multi modal semantic units (words and visual objects) We then stack multiple graph based multi modal fusion layers that iteratively perform semantic interactions to learn node representations. Finally, these representations provide an attention based context vector for the decoder. We evaluate our proposed encoder on the Multi30K datasets. Experimental results and in depth analysis show the superiority of our multi modal NMT model.
ACL