Traditional novel view synthesis, which is closely related to image-based rendering (IBR), relies on the basic principle of warping colors from one frame to another.
One approach is to use a triangle-mesh proxy to directly warp the image colors to a novel view [1]}, [2]}, [3]}.
However, imperfect geometry leads to ghosting artifacts around silhouettes.
This can be improved by replacing hand-crafted heuristics in the classic IBR pipeline with deep neural networks [4]}, [5]}, [6]}, [7]}.
If no proxy geometry is available, learning-based approaches have been developed that create a multi plane image representation [8]}, [9]}, [10]}, [11]} or directly estimate the required warp-field [12]}, [13]}, [14]}.
Novel view synthesis can also be performed by reconstructing a 3D model of the scene and rendering it from novel view points.
Thies et al. [15]} learn a neural texture on a triangle mesh which can be rendered using traditional rasterization.
The rasterized image is then converted to RGB by a deep neural network.
Other approaches use ray-casting to automatically learn a voxel grid [16]}, [17]}, [18]} or an implicit function [19]}, [20]}.
It has also been shown that point clouds are suitable geometric proxies for novel view synthesis [21]}, [22]}.
Neural Point-based Graphics (NPBG) [23]}, which is closely related to our method, renders a point cloud with learned neural descriptors in multiple resolutions.
These images are then used to reconstruct the final output image by a deep neural network.
NPBG is very flexible as it doesn't require a textured triangle mesh as proxy and shows impressive results on public datasets.
Similar to NPBG, Dai et al. show that rendering multiple depth layers of a point cloud can also be used to synthesize novel views [24]}.
