
LSTM
The structure of the standard LSTM is shown in Fig. REF . It's straight forward to see that it's the evolution of two parallel but coupled strings, the hidden states \(h\)  and the memory \(m\) . Compared with normal RNNs where there is only a single information passage, it's possible that the existence of the parallel information passages in LSTM makes the LSTM system capable of modeling long range dependencies.

Stacked LSTM
The stacked LSTM is in fact a deep LSTM whose geometric picture is a cascaded double string as in Fig. REF .

Grid LSTM
The grid LSTM[1]} is an extension of the LSTM to higher dimensions. Accordingly the strings in the standard LSTM are replaced by higher dimensional membranes as given in [1]}, where a 2d deep grid LSTM can be understood as stacked multiple 2d membranes. So if the deep LSTM is a cascaded coupled double strings, then the deep 2d grid LSTM is a cascaded membranes (Fig. REF ). Similar geometric pictures can be constructed for higher dimensional and deep grid LSTM systems.

Attention mechanism
The attention mechanism is getting popular in the decoder of the machine translation and the image caption generation. The key feature is that during the text generation, an alignment or an attention model (a feed-forward network in [3]} or a multiple layer perceptron network in [4]}) is used to generate a time-varying attention vector to indicate which part of the input data should be paid more attention to. This structure can be understood as a coupling between the normal decoder network (a RNN or a LSTM) with the attention network as shown in Fig. REF .

