Instead of relying on intuition to determine what tasks are helpful or bias-inducing, we propose using Necessary Condition Analysis (NCA) [1]} as a data-driven method for differentiating between auxiliary and adversarial tasks. We choose the Holistic Video Understanding (HVU) dataset [2]} to showcase our approach with AMT, as indicated in Fig. REF . All branches share a 3D feature extraction backbone, ensuring that the extracted features are predictive of the goal task(s) while not doing well on the undesired tasks. Our key observation for the HVU dataset is that while some scenes are highly correlated with the actions being performed in them, they are not necessary in order for the actions to be performed. For example, being in a casino is not necessary for playing blackjack, but the two have high correlation. On the other hand, holding a violin is necessary for playing violin. Therefore, we use scene recognition as an adversarial task and object detection as an auxiliary task for action recognition.
