To further improve the sequence learning ability of the recurrent networks, attention mechanisms have been integrated with EnDec RNNs as in [1]}, [2]}, [3]}. The conventional EnDec RNNs work on a fixed length context vector, thus there is a chance for forgetting key information conveyed in long input sequences. However, the improved attention-based models generate the context vector representing the relationship between input and output sequences using three elements: encoder hidden states, decoder hidden states, and alignment information between input and target (cf. REF ). Through this approach, these models
make sure all past contextual information are not forgotten during the target prediction in every time step. Similar to the EnDec RNN models, Sutskever et al. [4]}, implemented a novel Sequence to Sequence (seq2seq) deep model using LSTM cells capable of taking arbitrary length input-output sequences. This model performs better than the RNN-based EnDec solutions, the authors advocate.
