In this paper, we propose to extend the recently introduced model-agnostic meta-learning algorithm (MAML, Finn et al., 2017) for low resource neural machine translation (NMT). We frame low-resource translation as a meta-learning problem, and we learn to adapt to low-resource languages based on multilingual high-resource language tasks. We use the universal lexical representation (Gu et al., 2018b) to overcome the input-output mismatch across different languages. We evaluate the proposed meta-learning strategy using eighteen European languages (Bg, Cs, Da, De, El, Es, Et, Fr, Hu, It, Lt, Nl, Pl, Pt, Sk, Sl, Sv and Ru) as source tasks and five diverse languages (Ro, Lv, Fi, Tr, and Ko) as target tasks. We show that the proposed approach significantly outperforms the multilingual, transfer learning based approach (Zoph et al., 2016) and enables us to train a competitive NMT system with only a fraction of training examples. For instance, the proposed approach can achieve as high as 22.04 BLEU on Romanian-English WMT’16 by seeing only 16,000 translated words (~600 parallel sentences). Despite the massive success brought by neural machine translation (NMT, BID36 BID4 BID37 , it has been noticed that the vanilla NMT often lags behind conventional machine translation systems, such as statistical phrase-based translation systems (PBMT, BID24 , for low-resource language pairs (see, e.g., BID23 . In the past few years, various approaches have been proposed to address this issue. The first attempts at tackling this problem exploited the availability of monolingual corpora BID17 BID32 BID40 . It was later followed by approaches based on multilingual translation, in which the goal was to exploit knowledge from high-resource language pairs by training a single NMT system on a mix of high-resource and low-resource language pairs (Firat et al., 2016a,b; BID27 BID21 BID19 . Its variant, transfer learning, was also proposed by BID42 , in which an NMT system is pretrained on a high-resource language pair before being finetuned on a target low-resource language pair.In this paper, we follow up on these latest approaches based on multilingual NMT and propose a meta-learning algorithm for low-resource neural machine translation. We start by arguing that the recently proposed model-agnostic meta-learning algorithm (MAML, Finn et al., 2017) could be applied to low-resource machine translation by viewing language pairs as separate tasks. This view enables us to use MAML to find the initialization of model parameters that facilitate fast adaptation for a new language pair with a minimal amount of training examples ( §3). Furthermore, the vanilla MAML however cannot handle tasks with mismatched input and output. We overcome this limitation by incorporating the universal lexical representation BID15 and adapting it for the meta-learning scenario ( §3.3).We extensively evaluate the effectiveness and generalizing ability of the proposed meta-learning algorithm on low-resource neural machine translation. We utilize 17 languages from Europarl and Russian from WMT as the source tasks and test the meta-learned parameter initialization against five target languages (Ro, Lv, Fi, Tr and Ko), in all cases translating to English. Our experiments using only up to 160k tokens in each of the target task reveal that the proposed meta-learning approach outperforms the multilingual translation approach across all the target language pairs, and the gap grows as the number of training examples 2 Background Neural Machine Translation (NMT) Given a source sentence X = {x 1 , ..., x T }, a neural machine translation model factors the distribution over possible output sentences Y = {y 1 , ..., y T } into a chain of conditional probabilities with a leftto-right causal structure: DISPLAYFORM0 where special tokens y 0 ( bos ) and y T +1 ( eos ) are used to represent the beginning and the end of a target sentence. These conditional probabilities are parameterized using a neural network. Typically , an encoder-decoder architecture BID36 BID9 BID4 with a RNN-based decoder is used. More recently , architectures without any recurrent structures BID13 BID37 have been proposed and shown to speedup training while achieving state-of-the-art performance.Low Resource Translation NMT is known to easily over-fit and result in an inferior performance when the training data is limited BID23 . In general, there are two ways for handling the problem of low resource translation:(1) utilizing the resource of unlabeled monolingual data, and (2) sharing the knowledge between low-and high-resource language pairs. Many research efforts have been spent on incorporating the monolingual corpora into machine translation, such as multi-task learning BID17 Zong, 2016), back-translation (Sennrich et al., 2015) , dual learning BID20 and unsupervised machine translation with monolingual corpora only for both sides BID3 BID26 . For the second approach, prior researches have worked on methods to exploit the knowledge of auxiliary translations, or even auxiliary tasks. For instance, BID8 ; BID28 investigate the use of a pivot to build a translation path between two languages even without any directed resource. The pivot can be a third language or even an image in multimodal domains. When pivots are not easy to obtain, BID11 ; BID27 ; BID21 have shown that the structure of NMT is suitable for multilingual machine translation. BID15 also showed that such a multilingual NMT system could improve the performance of low resource translation by using a universal lexical representation to share embedding information across languages. All the previous work for multilingual NMT assume the joint training of multiple high-resource languages naturally results in a universal space (for both the input representation and the model) which, however, is not necessarily true, especially for very low resource cases.Meta Learning In the machine learning community, meta-learning, or learning-to-learn, has recently received interests. Meta-learning tries to solve the problem of "fast adaptation on new training data." One of the most successful applications of meta-learning has been on few-shot (or oneshot) learning BID25 , where a neural network is trained to readily learn to classify inputs based on only one or a few training examples. There are two categories of meta-learning:1. learning a meta-policy for updating model parameters (see, e.g., BID1 BID18 BID30 2. learning a good parameter initialization for fast adaptation (see, e.g., BID10 BID38 BID35 .In this paper, we propose to use a meta-learning algorithm for low-resource neural machine translation based on the second category. More specifically, we extend the idea of model-agnostic metalearning (MAML, Finn et al., 2017) in the multilingual scenario. In this paper, we proposed a meta-learning algorithm for low-resource neural machine translation that exploits the availability of high-resource languages pairs. We based the proposed algorithm on the recently proposed model-agnostic metalearning and adapted it to work with multiple languages that do not share a common vocabulary using the technique of universal lexcal representation, resulting in MetaNMT. Our extensive evaluation, using 18 high-resource source tasks and 5 low-resource target tasks, has shown that the proposed MetaNMT significantly outperforms the existing approach of multilingual, transfer learning in low-resource neural machine translation across all the language pairs considered. The proposed approach opens new opportunities for neural machine translation. First, it is a principled framework for incorporating various extra sources of data, such as source-and targetside monolingual corpora. Second, it is a generic framework that can easily accommodate existing and future neural machine translation systems.
we propose a meta-learning approach for low-resource neural machine translation that can rapidly learn to translate on a new language
{"src": "scitldr-AIC_train", "paper_id": "S1g5ylbm1Q"}