@inproceedings{
liu2024mvpbev,
title={{MVP}bev: Multi-view Perspective Image Generation from {BEV} with Test-time Controllability and Generalizability},
author={Buyu Liu and Kai Wang and Yansong Liu and Jun Bao and Tingting Han and Jun Yu},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=DkPInDljfM}
}
This work aims to address the multi-view perspective RGB generation from text prompts given Bird-Eye-View(BEV) semantics. Unlike prior methods that neglect layout consistency, lack the ability to handle detailed text prompts, or incapable of generalizing to unseen view points, MVPbev simultaneously generates cross-view consistent images of different perspective views with a two-stage design, allowing object-level control and novel view generation at test-time. Specifically, MVPbev firstly projects given BEV semantics to perspective view with camera parameters, empowering model to generalize to unseen view points. Then we introduce a multi-view attention module where special initialization and de-nosing processes are introduced to explicitly enforce local consistency among overlapping views w.r.t. cross-view homography. Last but not the least, MVPbev further allows test-time instance-level controllabity by refining a pre-trained text-to-image diffusion model. Our extensive experiments on NuScenes demonstrate that our method is capable of generating high-resolution photorealistic images from text descriptions with thousands of training samples, surpassing the state-of-the-art methods under various evaluation metrics. We further demonstrate the advances of our method in terms of generalizability and controllability with the help of novel evalution metrics and comprehensive human analysis. Our code and model will be made available.
MVPbev: Multi-view Perspective Image Generation from BEV with Test-time Controllability and Generalizability
Buyu Liu Kai Wang Yansong Liu Jun Bao Tingting Han Jun Yu