Video Action Recognition. We evaluate the visual representations from \(f_{v}(\cdot )\)  on two widely-used benchmarks, UCF101 [1]} and HMDB51 [2]}.
UCF101 contains ~13K videos of 101 action classes, and HMDB51 contains ~7K videos of 51 activity classes.
During training, we randomly sample 10 clips of 32 frames for each video.
For visual augmentation, we follow [3]} and use random cropping, color jitter, and random horizontal flipping.
For optimization, we use SGD with \(lr=0.005\) , weight decay as \(5e-3\) , and momentum as 0.9.
We use a mini-batch of 16 and train for 12 epochs.
During testing, we uniformly sample 10 clips for each video, average their softmax scores, and report the class with the highest score.
The averaged top-1 accuracies across three split folds are reported for both UCF101 and HMDB51.
