It need to be noted that there is an essential difference between the LSTM systems and the attention mechanism. We can see that the LSTM systems are the coupling of homogeneous subsystems but the attention mechanism is in fact a coupling of two heterogeneous subsystems, i.e. one decoder RNN/LSTM network and one attention model with a different structure. Other similar systems, such as the recurrent models of visual attention[1]} and the neural Turing machines[2]}[3]}, also hold a similar picture of coupling heterogeneous subnetworks.
