Neural rendering models are another family of works that learn to model 3D scene structure implicitly by exploiting view synthesis as a self-supervision signal.
The target of neural rendering task is to reconstruct a new scene from an unknown viewpoint.
The seminar work, Generative Query Network (GQN) [1]} learns to capture representation and render a new scene.
GQN consists of a representation network and a generation network: the representation network encodes the observations from reference views into a scene representation; the generation network which is based on recurrent model, reconstructs the scene from a new view conditioned on the scene representation and a stochastic latent variable. Taking inputs as the observed images from several viewpoints, and the camera pose of a new view, GQN predicts the physical scene of this new view. Intuitively, through end-to-end training, the representation network can capture the necessary and important factors of 3D environment for the scene reconstruction task via the generation network. GQN is extended by incorporating a geometric-aware attention mechanism to allow more complex environment modelling [2]}, as well as including multimodal data for scene inference [3]}. Scene representation network (SRN) [4]} tackles the scene rendering problem via a learned continuous scene representation that connects a camera pose and its corresponding observation. A differentiable Ray Marching algorithm is integrated into SRN to enforce the network to model 3D structure consistently.
However, these frameworks can only be applied to synthetic datasets due to the complexity of real-world environments.
