The Holistic Video Understanding (HVU) dataset [1]} contains 572k video clips of humans performing different actions. We use action labels where each video corresponds to one primary action label. Importantly, beyond action labels, the videos in the dataset contain additional labels in one of five categories: “Scene," “Object," “Event," “Attribute," and “Concept." In this work, we use the scene and object labels. We experiment on three different splits of the HVU dataset. One is the original split defined in the paper. It contains 572k videos, 882 action labels, 282 scene labels, and 1917 object labels. There are 476k videos in the training set and 31k videos in the validation set. The two other Scene-Invariant splits are defined in Section REF .
