The main limitation of [1]} is that the performance degrades drastically when the input sequence becomes longer. It is because of unavoidable information loss when compressing the whole input sequence to a fixed-dimension vector. [2]} and [3]} overcame this limitation by introducing attention mechanism to LSTM encoder-decoder. The model stores a contextualised vector for each time step in the input sequence. During decoding, the decoder computes the attention weights dynamically to focus on different contextualised vectors. Attention mechanism overtook the previous state-of-the-art in English-French and English-German translation and yields much more robust performance for longer input sequences.
