SAMURAI: Shape And Material from Unconstrained Real-world Arbitrary Image collections
Inverse rendering of an object under entirely unknown capture conditions is a
fundamental challenge in computer vision and graphics. Neural approaches such
as NeRF have achieved photorealistic results on novel view synthesis, but they
require known camera poses. Solving this problem with unknown camera poses is
highly challenging as it requires joint optimization over shape, radiance, and
pose. This problem is exacerbated when the input images are captured in the
wild with varying backgrounds and illuminations. Standard pose estimation
techniques fail in such image collections in the wild due to very few estimated
correspondences across images. Furthermore, NeRF cannot relight a scene under
any illumination, as it operates on radiance (the product of reflectance and
illumination). We propose a joint optimization framework to estimate the shape,
BRDF, and per-image camera pose and illumination. Our method works on
in-the-wild online image collections of an object and produces relightable 3D
assets for several use-cases such as AR/VR. To our knowledge, our method is the
first to tackle this severely unconstrained task with minimal user interaction.
Project page: https://markboss.me/publication/2022-samurai/ Video:
https://youtu.be/LlYuGDjXp-8