@inproceedings{
cao2024neuma,
title={Neu{MA}: Neural Material Adaptor for Visual Grounding of Intrinsic Dynamics},
author={Junyi Cao and Shanyan Guan and Yanhao Ge and Wei Li and Xiaokang Yang and Chao Ma},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=AvWB40qXZh}
}
While humans effortlessly discern intrinsic dynamics and adapt to new scenarios, modern AI systems often struggle. Current methods for visual grounding of dynamics either use pure neural-network-based simulators (black box), which may violate physical laws, or traditional physical simulators (white box), which rely on expert-defined equations that may not fully capture actual dynamics. We propose the Neural Material Adaptor (NeuMA), which integrates existing physical laws with learned corrections, facilitating accurate learning of actual dynamics while maintaining the generalizability and interpretability of physical priors. Additionally, we propose Particle-GS, a particle-driven 3D Gaussian Splatting variant that bridges simulation and observed images, allowing back-propagate image gradients to optimize the simulator. Comprehensive experiments on various dynamics in terms of grounded particle accuracy, dynamic rendering quality, and generalization ability demonstrate that NeuMA can accurately capture intrinsic dynamics. Project Page: https://xjay18.github.io/projects/neuma.html.
NeuMA: Neural Material Adaptor for Visual Grounding of Intrinsic Dynamics
Junyi Cao Shanyan Guan Yanhao Ge Wei Li Xiaokang Yang Chao Ma