For the experiments, we firstly extracted frame-level CNN features using MatConvNet [1]} based on Residue-152 Networks [2]} trained on the ImageNet [3]} dataset. The images were resized to 224\(\times \) 224, hence the dimension of each frame-level features is 7\(\times \) 7\(\times \) 2048. For the network training, we applied mini-batch size of 64 samples at each iteration. For each video clip, the baseline approach randomly selected 30 frames for training while the proposed approaches selected 60 frames for training as the proposed HM-AN is able to capture long-term dependencies. Actually, the optimal length for LSTM with attention is 30 and increasing the number will seriously deteriorate the performance. We applied the back propagation algorithm through time and Adam optimizer [4]} with a learning rate of 0.0001 to train the networks. The learning rate was changed to 0.00001 after 10,000 iterations. At test time, we compute class predictions for each time step and then average
those predictions over 60 frames. To obtain a prediction for the entire video clip, we average the predictions from all 60 frame blocks in the video. Also, we find normally the networks converge in several epoches.
