@inproceedings{
yang2023learning,
title={Learning Inner Monologue and Its Utilization in Vision-Language Challenges},
author={Diji Yang and Kezhen Chen and Jinmeng Rao and Xiaoyuan Guo and Yawen Zhang and Jie Yang and Yi Zhang},
booktitle={Socially Responsible Language Modelling Research},
year={2023},
url={https://openreview.net/forum?id=hCZGKGZ802}
}
Inner monologue is an essential phenomenon for reasoning and insight mining in human cognition. In this work, we propose a novel approach for AI systems to simulate inner monologue. Specifically, we consider the communications between components in an LLM-centric system as inner monologues, and demonstrate inner monologue reasoning ability can be learned by supervised learning and reinforcement learning, and then be utilized to solve different complex vision-language problems in different domains. Driven by the power of Large Language Models (LLMs), two prominent methods for vision-language tasks have emerged: (1) the hybrid integration between LLMs and Vision-Language Models (VLMs), where visual inputs are firstly converted into language descriptions by VLMs, serving as inputs for LLMs to generate final answer(s); (2) visual feature alignment in language space, where visual inputs are encoded as embeddings and projected to LLMs' language space via further supervised fine-tuning. The first approach provides light training costs and interpretability but is hard to be optimized in an end-to-end fashion. The second approach presents decent performance, but feature alignment usually requires large amounts of training data and lacks interpretability. With inner monologue simulation, our approach achieves competitive performance with less training data and promising interpretability when compared with state-of-the-art models on two popular tasks.
Learning Inner Monologue and Its Utilization in Vision-Language Challenges
Diji Yang Kezhen Chen Jinmeng Rao Xiaoyuan Guo Yawen Zhang Jie Yang Yi Zhang