We perform an in-depth investigation of the suitability of self-attention models for character-level neural machine translation. We test the standard transformer model, as well as a novel variant in which the encoder block combines information from nearby characters using convolution. We perform extensive experiments on WMT and UN datasets, testing both bilingual and multilingual translation to English using up to three input languages (French, Spanish, and Chinese). Our transformer variant consistently outperforms the standard transformer at the character-level and converges faster while learning more robust character-level alignments. Most existing Neural Machine Translation (NMT) models operate on the word or subword-level. Often, these models are memory inefficient because of large vocabulary size. Character-level models (Lee et al., 2017; Cherry et al., 2018) instead work directly on raw characters, resulting in a more compact language representation, while mitigating out-of-vocabulary (OOV) problems (Luong and Manning, 2016) . They are especially suitable for multilingual translation, where multiple languages can be modelled using the same character vocabulary. Multilingual training can lead to improvements in the overall performance without any increase in model complexity (Lee et al., 2017) . It also circumvents the need to train separate models for each language pair. Models based on self-attention have achieved excellent performance on a number of tasks including machine translation (Vaswani et al., 2017) and representation learning (Devlin et al., 2019; Yang et al., 2019) . Despite the success of these models, no previous work has considered their suitability for character-level translation, with the In this work, we perform an in-depth investigation of the suitability of self-attention models for character-level translation. We consider two models: the standard transformer from (Vaswani et al., 2017) ; as well as a novel variant, which we call the convtransformer (Figure 1 , Section 3). The latter uses convolution to facilitate interactions among nearby character representations. We evaluate these models on both bilingual and multilingual translation to English, using up to three input languages: French (FR), Spanish (ES), and Chinese (ZH). We compare their translation performance on close (e.g., FR and ES) and on distant (e.g., FR and ZH) input languages (Section 5.1), and we analyze their learned character alignments (Section 5.2). We find that self-attention models work surprisingly well for character-level translation, performing competitively with equivalent subword-level models while requiring up to 60% fewer parameters. At the character-level, the convtransformer performs better than the standard transformer, converging faster and producing more robust alignments. We performed a detailed investigation of the utility of self-attention models for character-level translation, testing the standard transformer architecture, as well as a novel variant augmented by convolution in the encoder to facilitate information propagation across characters. Our experiments show that self-attention performs very well on characterlevel translation, performing competitively with subword-level models, while requiring fewer parameters. Training on multiple input languages is also effective and leads to improvements across all languages when the source and target languages are similar. When the languages are different, we observe a drop in performance, in particular for the distant language. In future work, we will extend our analysis to include additional source and target languages from different language families, such as more Asian languages. We will also work towards improving the training efficiency of character-level models, which is one of their main bottlenecks. A Example model outputs Tables 3, 4 and 5 contain example translations produced by our different bilingual and multilingual models trained on the UN datasets.
We perform an in-depth investigation of the suitability of self-attention models for character-level neural machine translation.
{"src": "scitldr-AIC_train", "paper_id": "BWlCpme3TS"}