@inproceedings{
zhao2024autograph,
title={AutoGraph: Enabling Visual Context via Graph Alignment in Open Domain Multi-Modal Dialogue Generation},
author={Deji Zhao and Donghong Han and Ye Yuan and Bo Ning and Li Mengxiang and Zhongjiang He and Shuangyong Song},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=hZYk17jJaf}
}
Open-domain multi-modal dialogue system heavily relies on visual information to generate contextually relevant responses. The existing open-domain multi-modal dialog generation methods ignore the complementary relationship between multiple modalities, and are difficult to integrate with LLMs. To address these issues, we propose an automatically constructed visual context graph method, called AutoGraph. We aim to structure complex information and seamlessly integrate it with large language models (LLMs), aligning information from multiple modalities at both semantic and structural levels. Specifically, we fully connect the text graphs and scene graphs, and then trim unnecessary edges via LLMs to automatically construct a visual context graph. Next, we design several graph sampling grammar for the first time to convert graph structures into sequence which is suitable for LLMs. Finally, we propose a two-stage fine-tuning method to allow LLMs to understand graph sampling grammar and generate responses. The AutoGraph method is a general approach that can enhance the visual capabilities of LLMs. We validate our proposed method on text-based LLMs, and visual-based LLMs, respectively. Experimental results show that our proposed method achieves state-of-the-art performance on multiple public datasets.
AutoGraph: Enabling Visual Context via Graph Alignment in Open Domain Multi-Modal Dialogue Generation
Deji Zhao Donghong Han Ye Yuan Bo Ning Li Mengxiang Zhongjiang He Shuangyong Song