Video Action Recognition. Two-stream networks [1]} and 3D convolutional networks [2]}, [3]}, [4]} have improved human action recognition in recent years.
One of the most important aspects of video action recognition is capturing the long-term temporal structure of the action being performed across video timesteps [5]}. Current large-scale video datasets, however, provide many alternate cues over time that may influence the classifier's learned parameters [6]}, [7]}, [8]}, [9]}, [10]}. For example, the Holistic Video Understanding (HVU) dataset provides additional labels like scenes and objects [11]}. Here, we propose a generalized MTL framework to better utilize these additional cues to learn video models that capture action-related information.
