@inproceedings{
cao2024see,
title={See or Guess: Counterfactually Regularized Image Captioning},
author={Qian Cao and Xu Chen and Ruihua Song and Xiting Wang and Xinting Huang and Yuchen Ren},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=ugDKVMW62p}
}
Image captioning, which generates natural language descriptions of the visual information in an image, is a crucial task in vision-language research. Previous models have typically addressed this task by aligning the generative capabilities of machines with human intelligence through statistical fitting of existing datasets. While these models demonstrate proficiency in describing the content of normal images, they may struggle to accurately describe those where certain parts of the image are obscured or edited. Conversely, humans effortlessly excel at it in this case. The weaknesses these models exhibit, including hallucinations and limited interpretability, often result in performance declines when applied to scenarios involving shifted association patterns. In this paper, we present a generic image captioning framework that leverages causal inference to make existing models more capable of interventional tasks, and counterfactually explainable. Specifically, our approach consists of two variants that utilize either total effect or natural direct effect. We incorporate these concepts into the training process, enabling the models to handle counterfactual scenarios and thereby become more generalizable. Extensive experiments on various datasets have demonstrated that our method can effectively reduce hallucinations and increase the model's faithfulness to the images, with a high portability for both small-scale and large-scale image-to-text models.
See or Guess: Counterfactually Regularized Image Captioning
Qian Cao Xu Chen Ruihua Song Xiting Wang Xinting Huang Yuchen Ren