HMDB51 is a more difficult and bigger dataset. The performance of baseline approach is reported in [1]}, with 41.3% accuracy. Our HM-AN model with soft attention improves the accuracy to 43.8%. We then apply the REINFORCE-Hard Attention approach on this dataset. The accuracy result turns out to be lower than the HM-AN with soft attention. Moreover, the model with REINFORCE-like algorithm converges slower than the Gumbel-softmax with adaptive temperature, also with more oscillations on the training cost, which is shown in Fig. REF . With constant value 0.3 of temperature for hard attention, the model achieves 44.0% accuracy. Again, the improvement by adding adaptive temperature is obvious, with 44.2% accuracy on HMDB51 dataset. The accuracy results are further summarized in Table REF . We also compare the performance of the proposed HM-AN with some published models related to ours. Firstly, the methods not from RNN family but only with the spatial image shows poor performance as shown in Table REF , even with CNN model fine-tuned. Specifically, the softmax regression results from [1]} directly extracted image feature of each frame and perform softmax regression on them, with 33.5% accuracy. The spatial convolutional net from the famous two-stream approach [3]} can be considered as a similar method with the difference that the two-stream approach performs fine-tuning on the CNN model, with an improved accuracy of 40.5%. The LSTM without attention also achieves 40.5% accuracy [1]}. When adding soft attention mechanism, an improved accuracy of 41.3% can be obtained. The Conv-Attention [5]} and ConvALSTM [6]} both use convolutional LSTM with attention. The differences are that Conv-Attention extracts features from Residual-152 Networks [7]} without fine-tuning whilst ConvALSTM extracts image features from a fine-tuned VGG16 model. The ConvALSTM leads Conv-Attention by a small margin, with 43.3% accuracy. As explained previously, CHAM [5]} has a hand-designed hierarchical architecture in contrast with ours in which the temporal hierarchy is formed through training. Our best setting (Adaptive-Gumbel-Hard Attention) reports the highest accuracy (44.2%) among methods from the RNN family and leads the CHAM results (43.4%) by 0.8 percent.
<TABLE><TABLE><FIGURE>