Document Level Neural Machine Translation Using BERT as Context Encoder
Large scale pre trained representations such as BERT have been widely used in many natural language understanding tasks. The methods of incorporating BERT into document level machine translation are still being explored. BERT is able to understand sentence relationships since BERT is pre trained using the next sentence prediction task. In our work, we leverage this property to improve document level machine translation. In our proposed model, BERT performs as a context encoder to achieve document level contextual information, which is then integrated into both the encoder and decoder. Experiment results show that our proposed method can significantly outperform strong document level machine translation baselines on BLEU score. Moreover, the ablation study shows our method can capture document level context information to boost translation performance.
AACL