We firstly tested the performance of the LSTM with soft attention proposed in [1]} on the UCF sports dataset and obtained 70.0% accuracy. All the experimental settings are the same as these in [1]}. Then we evaluated the proposed four approaches mentioned previously. HM-AN with stochastic hard attention which is realized with REINFORCE-like algorithm improves the results to 82.0%. HM-AN with soft attention is similar to the REINFORCE-Hard Attention, with accuracy results of 81.1%. The hard attention mechanism realized by Gumbel-softmax with adaptive temperature achieves 82.0% accuracy, similar to our REINFORCE-Hard Attention model. However, the Constant-Gumbel-Hard Attention which uses Gumbel-softmax with constant temperature value of 0.3 only yields 76.0% accuracy, which indicates the significant role of adaptive temperature in maintaining the system performance. Fig. REF  shows the curves of training cost cross entropy for the Adaptive-Gumbel-Hard Attention approach and REINFORCE-Hard Attention approach, respectively. It can be seen from the figure that the REINFORCE-Hard Attention converges marginally slower than the approach of Adaptive-Gumbel-Hard Attention.
