Spatial Bias in Video Datasets. The main difference between video and image data is temporal modeling, and, hence, temporal information in videos plays an important role in classifying the actions happening in a sequence [1]}, [2]}. Related works show that state-of-the-art 3D convolutional neural networks do not perform well on recognizing the action a subject is performing without explicit scene detail [3]}, [4]}, [5]}. Huang et al.  [4]}
analyzes motion information in video datasets such as UCF101 and Kinetics. They found that around 40% of classes in these datasets do not require motion to match the average class accuracy. Through their approach they concluded that motion was not important for classifying an action in a video and that these video datasets are not built in a way that makes temporal information relatively important for making action predictions. One way to address this is to create datasets that actually need temporal information for classification [7]}.
Li et al.  [8]} formulates bias minimization as an optimization problem, such that the dataset is resampled and example-level weights penalize examples that are easy for the classifier if built on a given feature representation. In this work, we show that multi-task learning is one alternative promising direction to mitigate bias of irrelevant spatial scene features for action recognition.
