The idea of learning a prior (2D CNN/ViT) on radiance fields in order to enhance the few-view setup of novel view synthesis is previously investigated by several works [1]}, [2]}, [3]}. However, we propose SuRFNet to directly learn from the 3D sparse radiance fields, by optimizing partial SRFs from the few images and training a generalizable network that converts these partial SRFs to complete SRFs in a supervised fashion. Such a 3D setup benefits from structured 3D learning, creating a 3D prior that guarantees multi-view consistency, especially when rendering from out-of-distribution views. Also, this 3D sparse voxel setup benefits from the advancements in fast volume rendering [4]}, [5]}, allowing for end-to-end deep learning pipelines that harness volume rendering.
To the best of our knowledge, our SURFNet is the first model that learns to generate 3D radiance fields for unseen objects at test time with only a few/single views by learning from the distribution of radiance fields in 3D.
