Dynamic Attention Aggregation with BERT for Neural Machine Translation
The recently proposed BERT has demonstrated great power in various natural language processing tasks. However, the model does not perform effectively on cross lingual tasks, especially on machine translation. In this work, we propose three methods to introduce pre trained BERT into neural machine translation without fine tuning. Our approach consists of a) a linear attention aggregation that leverages a parameter matrix to capture the key knowledge of BERT, b) a self attention aggregation which aims to learn what is vital for input and output, and c) a switch gate aggregation to dynamically control the balance of the information flowing from the pre trained BERT or the NMT model. We conduct experiments on several translation benchmarks and substantially improve over 2 BELU points on the IWSLT'14 English German task with switch gate aggregation method compared to a strong baseline, while our proposed model also performs remarkably on the other tasks.
2020 International Joint Conference on Neural Networks (IJCNN)