@inproceedings{
huang2024chatscene,
title={Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers},
author={Haifeng Huang and Yilun Chen and Zehan Wang and Rongjie Huang and Runsen Xu and Tai Wang and Luping Liu and Xize Cheng and Yang Zhao and Jiangmiao Pang and Zhou Zhao},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=t3BhmwAzhv}
}
Recent advancements in 3D Large Language Models (LLMs) have demonstrated promising capabilities for 3D scene understanding. However, previous methods exhibit deficiencies in general referencing and grounding capabilities for intricate scene comprehension. In this paper, we introduce the use of object identifiers and object-centric representations to interact with scenes at the object level. Specifically, we decompose the input 3D scene into a set of object proposals, each assigned a unique identifier token, which enables efficient object referencing and grounding during user-assistant interactions. Given the scarcity of scene-language data, we model the scene embeddings as a sequence of explicit object-level embeddings, derived from semantic-rich 2D or 3D representations. By employing object identifiers, we transform diverse 3D scene-language tasks into a unified question-answering format, facilitating joint training without the need for additional task-specific heads. With minimal fine-tuning on all downstream tasks, our model significantly outperforms existing methods on benchmarks including ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, and SQA3D.
Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers
Haifeng Huang Yilun Chen Zehan Wang Rongjie Huang Runsen Xu Tai Wang Luping Liu Xize Cheng Yang Zhao Jiangmiao Pang Zhou Zhao