This paper addresses the task of estimating the light arriving from all directions to a 3D point observed at a selected pixel in an RGB image. This task is challenging because it requires predicting a mapping from a partial scene observation by a camera to a complete illumination map for a selected position, which depends on the 3D location of the selection, the distribution of unobserved light sources, the occlusions caused by scene geometry, etc. Previous methods attempt to learn this complex mapping directly using a single black-box neural network, which often fails to estimate high-frequency lighting details for scenes with complicated 3D geometry. Instead, we propose "Neural Illumination" a new approach that decomposes illumination prediction into several simpler differentiable sub-tasks: 1) geometry estimation, 2) scene completion, and 3) LDR-to-HDR estimation. The advantage of this approach is that the sub-tasks are relatively easy to learn and can be trained with direct supervision, while the whole pipeline is fully differentiable and can be fine-tuned with end-to-end supervision. Experiments show that our approach performs significantly better quantitatively and qualitatively than prior work.
Learning-based Methods Deep learning has recently shown promising results on a number of computer vision tasks, including depth estimation @cite_5 @cite_13 and intrinsic image decomposition @cite_0 @cite_20 . Recently Gardner al @cite_23 propose to formulate the illumination estimation function as an end-to-end neural network. However, since the input and output representation of their network architecture does not share any immediate notion of pixel-to-pixel spatial correspondence, their model tends to generate illumination maps that reflect general color statistics of the training dataset, as opposed to important high-frequency lighting details. In contrast, our model predicts the 3D geometric structure of the scene and uses it to warp the observed image into an intermediate representation that encodes the input information in a way that is spatially aligned to the output illumination map. This results in the ability to fully utilize input information and preserve high-frequency lighting details. Moreover, Gardner al's algorithm does not generate illumination conditioned on a selected target pixel (i.e., it produces only one solution for each input image). In contrast, our algorithm is able to recover the spatially varying illumination for any selected pixel (Figure ).
@cite_0 @cite_23 @cite_5 @cite_13 @cite_20
2896707699 1776042733 2745172755 2605060370
Most popular color constancy algorithms assume that the light source obeys a uniform distribution across the scene. However, in the real world, the illuminations can vary a lot according to their spatial distribution. To overcome this problem, in this paper, we adopt a method based on a full end-to-end deep neural model to directly learn a mapping from the original image to the corresponding well-colored image. With this formulation, the network is able to determine pixel-wise illumination and produce a final visually compelling image. The training and evaluation of the network were performed on a standard dataset of two-dominant-illuminants. In this dataset, this approach achieves state-of-the-art performance. Besides, the main architecture of the network simply consists of a stack of fully convolutional blocks which can take the input of arbitrary size and produce correspondingly-sized output with effective learning. The experimental result shows that our customized loss function can help to reach a better performance than simply using MSE. Deep networks have recently enjoyed enormous success when applied to recognition and classification problems in computer vision [22, 33], but their use in graphics problems has been limited ([23, 7] are notable recent exceptions). In this work, we present a novel deep architecture that performs new view synthesis directly from pixels, trained from a large number of posed image sets. In contrast to traditional approaches, which consist of multiple complex stages of processing, each of which requires careful tuning and can fail in unexpected ways, our system is trained end-to-end. The pixels from neighboring views of a scene are presented to the network, which then directly produces the pixels of the unseen view. The benefits of our approach include generality (we only require posed image sets and can easily apply our method to different domains), and high quality results on traditionally difficult scenes. We believe this is due to the end-to-end nature of our system, which is able to plausibly generate pixels according to color, depth, and texture priors learnt automatically from the training data. We show view interpolation results on imagery from the KITTI dataset [12], from data from [1] as well as on Google Street View images. To our knowledge, our work is the first to apply deep learning to the problem of new view synthesis from sets of real-world, natural imagery. We present a machine learning algorithm that takes as input a 2D RGB image and synthesizes a 4D RGBD light field (color and depth of the scene in each ray direction). For training, we introduce the largest public light field dataset, consisting of over 3300 plenoptic camera light fields of scenes containing flowers and plants. Our synthesis pipeline consists of a convolutional neural network (CNN) that estimates scene geometry, a stage that renders a Lambertian light field using that geometry, and a second CNN that predicts occluded rays and non-Lambertian effects. Our algorithm builds on recent view synthesis methods, but is unique in predicting RGBD for each light field ray and improving unsupervised single image depth estimation by enforcing consistency of ray depths that should intersect the same scene point. Please see our supplementary video at this https URL We propose an automatic method to infer high dynamic range illumination from a single, limited field-of-view, low dynamic range photograph of an indoor scene. In contrast to previous work that relies on specialized image capture, user input, and or simple scene models, we train an end-to-end deep neural network that directly regresses a limited field-of-view photo to HDR illumination, without strong assumptions on scene geometry, material properties, or lighting. We show that this can be accomplished in a three step process: 1) we train a robust lighting classifier to automatically annotate the location of light sources in a large dataset of LDR environment maps, 2) we use these annotations to train a deep neural network that predicts the location of lights in a scene from a single limited field-of-view photo, and 3) we fine-tune this network using a small dataset of HDR environment maps to predict light intensities. This allows us to automatically recover high-quality HDR illumination estimates that significantly outperform previous state-of-the-art methods. Consequently, using our illumination estimates for applications like 3D object insertion, produces photo-realistic results that we validate via a perceptual user study.