The modeling of hierarchical temporal information has long been an important topic in many research areas. The most notable model is LSTM proposed by Hochreiter and Schmidhuber [1]}. LSTM employs the multi-scale updating concept, where the hidden units' update can be controlled by gating such as input gates or forget gates. This mechanism enables the LSTM to deal long term dependencies in temporal domain. Despite the advantage, the maximum time steps are limited within a few hundreds because of the leaky integration which makes the memory for long-term gradually diluted [2]}. Actually, the maximum time steps in video processing is several dozens which makes the application of LSTM in video recognition very challenging.
