In this paper we show that a simple beam approximation of the joint distribution between attention and output is an easy , accurate , and efficient attention mechanism for sequence to sequence learning . The method combines the advantage of sharp focus in hard attention and the implementation ease of soft attention . On five translation and two morphological inflection tasks we show effortless and consistent gains in BLEU compared to existing attention mechanisms . In structured input-output models as used in tasks like translation and image captioning , the attention variable decides which part of the input aligns to the current output . Many attention mechanisms have been proposed ( Xu et al . , 2015 ; Bahdanau et al . , 2014 ; Luong et al . , 2015 ; Martins and Astudillo , 2016 ) but the de facto standard is a soft attention mechanism that first assigns attention weights to input encoder states , then computes an attention weighted ' soft ' aligned input state , which finally derives the output distribution . This method is end to end differentiable and easy to implement . Another less popular variant is hard attention that aligns each output to exactly one input state but requires intricate training to teach the network to choose that state . When successfully trained , hard attention is often found to be more accurate ( Xu et al . , 2015 ; Zaremba and Sutskever , 2015 ) . In NLP , a recent success has been in a monotonic hard attention setting in morphological inflection tasks ( Yu et al . , 2016 ; Aharoni and Goldberg , 2017 ) . For general seq2seq learning , methods like Sparse-Max ( Martins and Astudillo , 2016 ) and local attention ( Luong et al . , 2015 ) were proposed to bridge the gap between soft and hard attention . * Both authors contributed equally to this work In this paper we propose a surprisingly simpler alternative based on the original joint distribution between output and attention , of which existing soft and hard attention mechanisms are approximations . The joint model couples input states individually to the output like in hard attention , but it combines the advantage of end-to-end trainability of soft attention . When the number of input states is large , we propose to use a simple approximation of the full joint distribution called Beam-joint . This approximation is also easily trainable and does not suffer from the high variance of Monte-Carlo sampling gradients of hard attention . We evaluated our model on five translation tasks and increased BLEU by 0.8 to 1.7 over soft attention , which in turn was better than hard and the recent Sparsemax ( Martins and Astudillo , 2016 ) attention . More importantly , the training process was as easy as soft attention . For further support , we also evaluate on two morphological inflection tasks and got gains over soft and hard attention . In this paper we showed a simple yet effective approximation of the joint attention-output distribution in sequence to sequence learning . Our joint model consistently provides higher accuracy without significant running time overheads in five translation and two morphological inflection tasks . An interesting direction for future work is to extend beam-joint to multi-head attention architectures as in ( Vaswani et al . , 2017 ; Xu Chen , 2018 ) .
Softmax attention models are popular because of their differentiable and easy to implement nature while hard attention models outperform them when successfully trained.
They propose a method to approximate the joint attention-output distribution which provides sharp attention as hard attention and easy implementation as soft attention.
The proposed approach outperforms soft attention models and recent hard attention and Sparsemax models on five translation tasks and also on morphological inflection tasks.