We use 210 videos for training, 30 videos for validation and the rest 70 videos for testing. We fix our sampling rate to 5fps to bring uniformity in the temporal representation of actions across all videos. These sampled frames are then converted into features using VGGNet [1]}. We initialize the decoder embedding with Google pre-trained word2vec word embeddings [2]}. We tune all hyperparameters using our validation data: learning rate, weight initializations, hidden states. We use a 1024-dimension RNN hidden state size. We use Adam optimizer [3]}. We apply a dropout of 0.5.
