@inproceedings{
li2024imagebindd,
title={ImageBind3D: Image as Binding Step for Controllable 3D Generation},
author={Zhenqiang Li and Jie LI and Yangjie Cao and Jiayi Wang and Runfeng Lv},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=ucj1H91vzj}
}
Recent advancements in 3D generation have garnered considerable interest due to their potential applications. Despite these advancements, the field faces persistent challenges in multi-conditional control, primarily due to the lack of paired datasets and the inherent complexity of 3D structures. To address these challenges, we introduce ImageBind3D, a novel framework for controllable 3D generation that integrates text, hand-drawn sketches, and depth maps to enhance user controllability. Our innovative contribution is the adoption of an inversion-align strategy, facilitating controllable 3D generation without requiring paired datasets. Firstly, utilizing GET3D as a baseline, our method innovates a 3D inversion technique that synchronizes 2D images with 3D shapes within the latent space of 3D GAN.  Subsequently, we leverage images as intermediaries to facilitate pseudo-pairing between the shapes and various modalities. Moreover, our multi-modal diffusion model design strategically aligns external control signals with the generative model's latent knowledge, enabling precise and controllable 3D generation. Extensive experiments validate that ImageBind3D surpasses existing state-of-the-art methods in both fidelity and controllability. Additionally, our approach can offer composable guidance for any feed-forward 3D generative models, significantly enhancing their controllability.
ImageBind3D: Image as Binding Step for Controllable 3D Generation
Zhenqiang Li Jie LI Yangjie Cao Jiayi Wang Runfeng Lv