We introduce the masked translation model (MTM) which combines encoding and decoding of sequences within the same model component. The MTM is based on the idea of masked language modeling and supports both autoregressive and non-autoregressive decoding strategies by simply changing the order of masking. In experiments on the WMT 2016 Romanian-English task, the MTM shows strong constant-time translation performance, beating all related approaches with comparable complexity. We also extensively compare various decoding strategies supported by the MTM, as well as several length modeling techniques and training settings. Neural machine translation (NMT) has been developed under the encoder-decoder framework (Sutskever et al., 2014) with an intermediary attention mechanism (Bahdanau et al., 2015) . The encoder learns contextualized representations of source tokens, which are used by the decoder to predict target tokens. These two components have individual roles in the translation process, and they are connected via an encoder-decoder attention layer (Bahdanau et al., 2015) . Many advances in NMT modeling are based on changes in the internal layer structure (Gehring et al., 2017; Wang et al., 2017; Vaswani et al., 2017; Dehghani et al., 2019; , tweaking the connection between the layers (Zhou et al., 2016; Shen et al., 2018; Bahar et al., 2018; Li et al., 2019a) , or appending extra components or latent variables (Gu et al., 2016; Zhang et al., 2016; Shah & Barber, 2018; ) -all increasing the overall architectural complexity of the model while keeping the encoder and decoder separated. Our goal is to simplify the general architecture of machine translation models. For this purpose, we propose the masked translation model (MTM) -a unified model which fulfills the role of both the encoder and decoder within a single component. The MTM gets rid of the conventional decoder as well as the encoder-decoder attention mechanism. Its architecture is only a sequence encoder with self-attention layers, trained with an objective function similar to masked language modeling (Devlin et al., 2019) . In order to model the translation problem, the MTM is given the concatenation of the source and target side from a parallel sentence pair. This approach is similar to the translation language model presented by Lample & Conneau (2019) , but focuses on the target side, i.e. the masking is applied to some selected positions in the target sentence. The MTM is trained to predict the masked target words relying on self-attention layers which consider both the source sentence and a masked version of the target sentence. Trained in this way, the model is perfectly suitable for non-autoregressive decoding since the model learned to predict every position in parallel, removing the dependency on decisions at preceding target positions. Within its extremely simple architecture, one can realize various decoding strategies, e.g., using left-to-right, non-autoregressive, or iterative decoding by merely adjusting the masking schemes in search. We present a unified formulation of the MTM for different decoding concepts by factorizing the model probability over a set of masked positions. The MTM has several advantages over the conventional encoder-decoder framework: • A simpler architecture In this work we simplify the existing Transformer architecture by combining the traditional encoder and decoder elements into a single component. The resulting masked translation model is trained by concatenating source and target and applying BERT-style masking to the target sentence. The novel training strategy introduced with the MTM requires a rethinking of the search process and allows for various new decoding strategies to be applied in the theoretical framework we developed in this work. A detailed comparison shows that unmasking the sequence one-by-one gives the overall best performance, be it left-to-right, right-to-left, or confidence-based. Unveiling a constant number of tokens based on confidence in each decoding step, however, can achieve reasonable performance with a fixed, much smaller number of iterations. We show that there is a potential of at least 1.5 % BLEU improvement that can be achieved by more elaborate length models, which yields itself as a good start for further research. Furthermore, we plan to extend the decoding strategies to work with beam search and verify our observations on further language pairs.
We use a transformer encoder to do translation by training it in the style of a masked translation model.
{"src": "scitldr-AIC_train", "paper_id": "HygaSxHYvH"}