Training. We train the models end-to-end using sgd with momentum 0.9, weight decay \(10^{-4}\)  and an initial learning rate of 0.01, for 60 epochs. We use a cosine learning rate schedule with warmup (10 epochs) for adjusting the learning rate. The batch size used is 32 and a dropout of 0.5 is applied to avoid overfitting. 8 or 16 frames sampled from each video is used for predicting the action present in the video. Random scaling, cropping and flipping along with temporal jittering as presented in [1]} are used as data augmentations during training. The backbone cnn are pretrained on ImageNet. Unless otherwise specified, we use the same setting for training gsf on all the datasets used in this work.
