Multilingual Neural Machine Translation (NMT) systems are capable of translating between multiple source and target languages within a single system. An important indicator of generalization within these systems is the quality of zero-shot translation - translating between language pairs that the system has never seen during training. However, until now, the zero-shot performance of multilingual models has lagged far behind the quality that can be achieved by using a two step translation process that pivots through an intermediate language (usually English). In this work, we diagnose why multilingual models under-perform in zero shot settings. We propose explicit language invariance losses that guide an NMT encoder towards learning language agnostic representations. Our proposed strategies significantly improve zero-shot translation performance on WMT English-French-German and on the IWSLT 2017 shared task, and for the first time, match the performance of pivoting approaches while maintaining performance on supervised directions. In recent years, the emergence of sequence to sequence models has revolutionized machine translation. Neural models have reduced the need for pipelined components, in addition to significantly improving translation quality compared to their phrase based counterparts BID35 . These models naturally decompose into an encoder and a decoder with a presumed separation of roles: The encoder encodes text in the source language into an intermediate latent representation, and the decoder generates the target language text conditioned on the encoder representation. This framework allows us to easily extend translation to a multilingual setting, wherein a single system is able to translate between multiple languages BID11 BID28 .Multilingual NMT models have often been shown to improve translation quality over bilingual models, especially when evaluated on low resource language pairs BID14 BID20 . Most strategies for training multilingual NMT models rely on some form of parameter sharing, and often differ only in terms of the architecture and the specific weights that are tied. They allow specialization in either the encoder or the decoder, but tend to share parameters at their interface. An underlying assumption of these parameter sharing strategies is that the model will automatically learn some kind of shared universally useful representation, or interlingua, resulting in a single model that can translate between multiple languages.The existence of such a universal shared representation should naturally entail reasonable performance on zero-shot translation, where a model is evaluated on language pairs it has never seen together during training. Apart from potential practical benefits like reduced latency costs, zero-shot translation performance is a strong indicator of generalization. Enabling zero-shot translation with sufficient quality can significantly simplify translation systems, and pave the way towards a single multilingual model capable of translating between any two languages directly. However, despite being a problem of interest for a lot of recent research, the quality of zero-shot translation has lagged behind pivoting through a common language by 8-10 BLEU points BID15 BID24 BID21 BID27 . In this paper we ask the question , What is the missing ingredient that will allow us to bridge this gap? Figure 1 : The proposed multilingual NMT model along with the two training objectives. CE stands for the cross-entropy loss associated with maximum likelihood estimation for translation between English and other languages. Align represents the source language invariance loss that we impose on the representations of the encoder. While training on the translation objective , training samples (x, y) are drawn from the set of parallel sentences , D x,y . For the invariance losses, (x, y) could be drawn from D x,y for the cosine loss , or independent data distributions for the adversarial loss. Both losses are minimized simultaneously. Since we have supervised data only to and from English, one of x or y is always in English.In BID24 , it was hinted that the extent of separation between language representations was negatively correlated with zero-shot translation performance. This is supported by theoretical and empirical observations in domain adaptation literature, where the extent of subspace alignment between the source and target domains is strongly associated with transfer performance BID7 BID8 BID17 . Zero-shot translation is a special case of domain adaptation in multilingual models, where English is the source domain and other languages collectively form the target domain. Following this thread of domain adaptation and subspace alignment, we hypothesize that aligning encoder representations of different languages with that of English might be the missing ingredient to improving zero-shot translation performance.In this work, we develop auxiliary losses that can be applied to multilingual translation models during training, or as a fine-tuning step on a pre-trained model, to force encoder representations of different languages to align with English in a shared subspace. Our experiments demonstrate significant improvements on zero-shot translation performance and, for the first time, match the performance of pivoting approaches on WMT English-French-German (en-fr-de) and the IWSLT 2017 shared task, in all zero shot directions, without any meaningful regression in the supervised directions.We further analyze the model's representations in order to understand the effect of our explicit alignment losses. Our analysis reveals that tying weights in the encoder, by itself, is not sufficient to ensure shared representations. As a result, standard multilingual models overfit to the supervised directions, and enter a failure mode when translating between zero-shot languages. Explicit alignment losses incentivize the model to use shared representations, resulting in better generalization.2 ALIGNMENT OF LATENT REPRESENTATIONS 2.1 MULTILINGUAL NEURAL MACHINE TRANSLATION Let x = (x 1 , x 2 ...x m ) be a sentence in the source language and y = (y 1 , y 2 , ...y n ) be its translation in the target language. For machine translation, our objective is to learn a model, p(y|x; θ). In modern NMT, we use sequence-to-sequence models supplemented with an attention mechanism BID5 to learn this distribution. These sequence-to-sequence models consist of an encoder, Enc(x) = z = (z 1 , z 2 , ...z m ) parameterized with θ enc , and a decoder that learns to map from the latent representation z to y by modeling p(y|z; θ dec ), again parameterized with θ dec . This model is trained to maximize the likelihood of the available parallel data, D x,y . DISPLAYFORM0 In multilingual training we jointly train a single model BID26 to translate from many possible source languages to many potential target languages. When only the decoder is informed about the desired target language, a special token to indicate the target language, < tl >, is input to the first step of the decoder. In this case, D x,y is the union of all the parallel data for each of the supervised translation directions. Note that either the source or the target is always English. In this work we propose explicit alignment losses, as an additional constraint for multilingual NMT models, with the goal of improving zero-shot translation. We view the zero-shot NMT problem in the light of subspace alignment for domain adaptation, and propose simple approaches to achieve this. Our experiments demonstrate significantly improved zero-shot translation performance that are, for the first time, comparable to strong pivoting based approaches. Through careful analyses we show how our proposed alignment losses result in better representations, and thereby better zeroshot performance, while still maintaining performance on the supervised directions. Our proposed methods have been shown to work reliably on two public benchmarks datasets: WMT EnglishFrench-German and the IWSLT 2017 shared task.
Simple similarity constraints on top of multilingual NMT enables high quality translation between unseen language pairs for the first time.
{"src": "scitldr-AIC_train", "paper_id": "ByWMz305FQ"}