In this paper, we propose a dependent multi-task framework with the causal intervention (DMTCI) to eliminate the spurious correlations. Firstly, we formulate the image caption as a dependent multi-task (DMT) learning problem [1]}, which includes two tasks: bag-of-categories (BOC) generation and caption generation. The outputs of the first task are considered as the mediators, which are already annotated in certain datasets like MSCOCO and also utilized by recent work [2]}, [3]}. Secondly, in terms of the confounders, we are probably to have no access to the real confounders. For example, "female" is not annotated. Fortunately, an available approach is using so-called "proxy confounders" [4]}. Since the spurious correlations problem we observe is that the models over-generating high-frequency categories or predicates, we let the proxy confounders be sampled from a set of concepts including high-frequency predicates and fine-grained object categories. Then we apply the variational inference [5]} to estimate the real confounders and let it sampled from a prior distribution. As a side advantage, our model can also infer covariates from the captions. Finally, we apply a multi-agent reinforcement learning strategy (MARL) to enable end-to-end training and reduce the inter-task error accumulations. An extensive set of experiments demonstrates that our model outperforms the baseline models and achieves competitive performance with those meticulously designed Transformers on the MSCOCO dataset. The main contributions of this work can be summarized as follows:
