@inproceedings{
jiang2024in,
title={In Situ 3D Scene Synthesis for Ubiquitous Embodied Interfaces},
author={Haiyan Jiang and Song Leiyu and dongdong weng and Zhe Sun and Li Huiying and Xiaonuo Dongye and Zhenliang Zhang},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=hwieLHhkRr}
}
Virtual reality (VR) provides an interface to access virtual environments anytime and anywhere, allowing us to experience and interact with an immersive virtual world. It has been widely used in various fields, such as entertainment, training, and education. However, the user's body cannot be separated from the physical world. When users are immersed in virtual scenes, they encounter safety and immersion issues caused by physical objects in the surrounding environment. Although virtual scene synthesis has attracted widespread attention, many popular methods are limited to generating purely virtual scenes independent of the physical environment or simply mapping all physical objects as obstacles. To this end, we propose a scene agent that synthesizes situated 3D virtual scenes as a kind of ubiquitous embodied interface in VR for users.  The scene agent synthesizes scenes by perceiving the user's physical environment as well as inferring the user's demands.  The synthesized scenes maintain the affordances of the physical environment, enabling immersive users to interact with the physical environment and improving the user's sense of security. Meanwhile, the synthesized scenes maintain the style described by the user, improving the user's immersion. The comparison results show that the proposed scene agent can synthesize virtual scenes with better affordance maintenance, scene diversity, style maintenance, and 3D intersection over union (3D IoU) compared to state-of-the-art baseline methods. To the best of our knowledge, this is the first work that achieves in situ scene synthesis with virtual-real affordance consistency and user demand.
In Situ 3D Scene Synthesis for Ubiquitous Embodied Interfaces
Haiyan Jiang Song Leiyu dongdong weng Zhe Sun Li Huiying Xiaonuo Dongye Zhenliang Zhang