Our approach (see Fig. ) builds on Aliev et al's [1]} pipeline and improves it in various ways.
In particular, we add a physical, differentiable camera model and a differentiable tone mapper, and provide a formulation for better approximation of the spatial gradient of one-pixel point rasterization.
This differentiable pipeline allows us to not only optimize the neural point features, but also to correct imprecisions from the input during the training stage.
So our system adjusts the camera pose and camera model based on the visual loss from the neural rendering network and estimates per-image exposure and white-balance values paired with a vignetting model and sensor response curve per camera.
Based on this cleaned and corrected input, rendering quality is significantly improved. Further, our method is able to synthesize arbitrary HDR and LDR setups and to correct under- or overexposed views.
Finally, the number of parameters inside the deep neural network can be significantly decreased because brightness and color changes are now handled separately by a physically correct sensor model.
Rendering performance is therefore increased and overfitting artifacts are reduced.
In summary, the contributions of our work are:
