A Survey of Deep Learning Techniques for Neural Machine Translation <s> C. Other related work <s> The attentional mechanism has proven to be effective in improving end-to-end neural machine translation. However, due to the intricate structural divergence between natural languages, unidirectional attention-based models might only capture partial aspects of attentional regularities. We propose agreement-based joint training for bidirectional attention-based end-to-end neural machine translation. Instead of training source-to-target and target-to-source translation models independently, our approach encourages the two complementary models to agree on word alignment matrices on the same training data. Experiments on Chinese-English and English-French translation tasks show that agreement-based joint training significantly improves both alignment and translation quality over independent training. <s> BIB001 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> C. Other related work <s> Knowing which words have been attended to in previous time steps while generating a translation is a rich source of information for predicting what words will be attended to in the future. We improve upon the attention model of Bahdanau et al. (2014) by explicitly modeling the relationship between previous and subsequent attention levels for each word using one recurrent network per input word. This architecture easily captures informative features, such as fertility and regularities in relative distortion. In experiments, we show our parameterization of attention improves translation quality. <s> BIB002 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> C. Other related work <s> Neural encoder-decoder models of machine translation have achieved impressive results, rivalling traditional translation models. However their modelling formulation is overly simplistic, and omits several key inductive biases built into traditional models. In this paper we extend the attentional neural translation model to include structural biases from word based alignment models, including positional bias, Markov conditioning, fertility and agreement over translation directions. We show improvements over a baseline attentional model and standard phrase-based model over several language pairs, evaluating on difficult languages in a low resource setting. <s> BIB003 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> C. Other related work <s> Attention mechanism has enhanced stateof-the-art Neural Machine Translation (NMT) by jointly learning to align and translate. It tends to ignore past alignment information, however, which often leads to over-translation and under-translation. To address this problem, we propose coverage-based NMT in this paper. We maintain a coverage vector to keep track of the attention history. The coverage vector is fed to the attention model to help adjust future attention, which lets NMT system to consider more about untranslated source words. Experiments show that the proposed approach significantly improves both translation quality and alignment quality over standard attention-based NMT. 1 <s> BIB004 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> C. Other related work <s> In this paper, we improve the attention or alignment accuracy of neural machine translation by utilizing the alignments of training sentence pairs. We simply compute the distance between the machine attentions and the "true" alignments, and minimize this cost in the training procedure. Our experiments on large-scale Chinese-to-English task show that our model improves both translation and alignment qualities significantly over the large-vocabulary neural machine translation system, and even beats a state-of-the-art traditional syntax-based system. <s> BIB005
Besides the above description in significant progress, there are also some other refinements in a different perspective of attention mechanism. In perspective of attention structure, Yang et al. improved the traditional attention structure by providing a network to model the relationship of word with its previous and subsequent attention BIB002 . Feng et al. proposed a recurrent attention mechanism to improve the alignment accuracy, and it has been proved to outperformed vanilla models in largescale ChineseEnglish task  . Moreover, other researches focus on the training process. Cohn et al. extended the original attention structure by adding several structural biases, they including positional bias, Markov conditioning, fertility, and Bilingual Symmetry BIB003 , model that integrated with these refinements have got better translation performance over the basic attention-based model. More concretely, the above methods can be seen as a kind of inheritance from the alignment model in SMT research, with more experiential assumption and intuition in linguistics: P osition Bias : It assumed words in source and target sentence with the same meaning would also have a similar relative position, especially when both two-sentence have a similar word order. As an adjustment of the original attention mechanism, it helps to improve the alignment accuracy by encouraging words in similar relative position to be aligned. Fig-ure11111 demonstrated the phenomena strongly, where words in diagonal are tended to be aligned. M arkov Condition : Empirically, in one sentence, one word has higher Correlation with its nearby words rather than those far from it, this is also the basement in explaining context capture of n-gram LM. As for translation task, it's obvious that words are adjacent in source sentence would also map to the nearby position in target sentence, taking advantage of this property, this consideration thus improves the alignment accuracy by discouraging the huge jumps in finding the corresponding alignment of nearby words. In addition, the method with similar consideration but different implementation is local attention. F ertility : Fertility measures whether the word has been attended at the right level, it considers preventing both scenarios when the word hasn't got enough attention or has been paid too much attention. This design comes from the fact that the poor translation result is commonly due to repeatedly translated some word or lack coverage of other words, which refers to Under-translation and Over-translation. Bilingual Symmetry : In theory, word alignment should be a reversible result, which means the same word alignment should be got when translation processing form A to B with translation from B to A. This motivates the parallel training for the model in both directions and encouraging the similar alignment result. The refinement infertility was further extended by Tu et al. BIB004 , who proposed fertility prediction as a normalizer before decoding, this method adjusts the context vector in original NMT model by adding coverage information when calculating attention weights, thus can provide complementary information about the probability of source words have been translated in prior steps. Besides the intuition that heuristics from SMT, Cheng et al. applied the agreement-based learning method on NMT task, which encourages joint training in the agreement of word alignment with both translation directions BIB001 . In later, Mi et al. proposed a supervised method for attention component, and it utilized annotated data with additional alignment constraints in its objective function, experiments in Chinese-to-English task has proven to benefit for both translation performance and alignment accuracy BIB005 .