Compared Models. We compare our models with several models including SCST [1]}, Up-Down [2]}, AoANet [3]}, Up-Down+VC [4]}, WeakVRD [5]}, \(\mathcal {M}^{2}\)  Transformer [6]}, and XLAN [7]}. Among them, XLAN is the recent SOTA with an advanced architecture, and Up-Down+VC is related to our work which uses causal intervention to reduce spurious correlations between objects. Also, we compare with the baseline model, TransLSTM, consisting of a Transformer encoder and an up-down LSTM attention decoder. It is worth noting that one may find that a recent model OSCAR [8]} greatly outperforms the models mentioned above, but we want to clarify that OSCAR make two unfair settings: 1) using more vision-language datasets for pre-training, including Conceptual Captions, SBU captions, Flicker30k, and GQA [8]}; 2) utilizing the ground-truth rather than generated object categories in the inference phase.
