HVU Experiments. For our HVU experiment, we choose the 3D-ResNet architecture as our feature extractor and run experiments on 3D-ResNet-18 and 3D-ResNet-50. Models are pretrained on the Kinetics-700 dataset [1]} before running on the HVU dataset. We train with stochastic gradient descent (SGD) and initialize our learning rate as \(1 \times 10^{-2}\) , dividing by a factor of 10 when validation action accuracy saturates. The activation after the fourth ResNet block is used as the feature representation \(f_\theta (x)\) , the output of the feature extractor \(f_\theta \) . In the action branch, we apply a fully connected layer onto the extracted features to attain the final class scores. In the scene and object branches, we use two sets of linear, batchnorm, ReLU layers before a final fully connected layer. The hidden layers have 2048 units. We choose weights of the scene and object branches, \(\lambda \)  and \(\gamma \) , via cross-validation for each respective model and split. We train our network until validation accuracy saturates.
