@inproceedings{
gao2024fact,
title={Fact: Teaching {MLLM}s with Faithful, Concise and Transferable Rationales},
author={Minghe Gao and Shuang Chen and Liang Pang and Yuan Yao and Jisheng Dang and Wenqiao Zhang and Juncheng Li and Siliang Tang and Yueting Zhuang and Tat-Seng Chua},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=vwqjDXmFss}
}
The remarkable performance of Multimodal Large Language Models (MLLMs) has unequivocally demonstrated their proficient understanding capabilities in handling a wide array of visual tasks. Nevertheless, the opaque nature of their black-box reasoning processes persists as an enigma, rendering them uninterpretable and struggling with hallucination. Their ability to execute intricate compositional reasoning tasks is also constrained, culminating in a stagnation of learning progression for these models. In this work, we introduce Fact, a novel paradigm designed to generate multimodal rationales that are faithful, concise, and transferable for teaching MLLMs.  This paradigm utilizes verifiable visual programming to generate executable code guaranteeing faithfulness and precision. 
Subsequently, through a series of operations including pruning, merging, and bridging, the rationale enhances its conciseness. 
Furthermore, we filter rationales that can be transferred to end-to-end paradigms from programming paradigms to guarantee transferability. Empirical evidence from experiments demonstrates the superiority of our method across models of varying parameter sizes, significantly enhancing their compositional reasoning and generalization ability.  Our approach also reduces hallucinations owing to its high correlation between images and text. The anonymous project is available at: https://anonymous.4open.science/r/Fact_program-216D/
Fact: Teaching MLLMs with Faithful, Concise and Transferable Rationales
Minghe Gao Shuang Chen Liang Pang Yuan Yao Jisheng Dang Wenqiao Zhang Juncheng Li Siliang Tang Yueting Zhuang Tat-Seng Chua