RNNs have been popular for speech recognition [1]}, image caption generation [2]}, and video description generation [3]}. There have also been efforts made for the application of LSTM RNNs in action recognition. For instance, [4]} proposed an end-to-end training system using CNN and RNNs deep both in space and time to recognize activities in video. [5]} also explicitly models the video as an ordered sequence of frames using LSTM. Most of the previous works treat image features extracted from CNNs as static inputs to RNNs to generate action labels at each frame. However, static features indiscriminating of important and irrelevant information will be detrimental to system performance. On the other hand, the interpretation of CNN features will be much easier if attention model can be applied for action recognition because attention mechanism automatically focuses on specific regions to facilitate the classification.
