Illumination estimation is often used in mixed reality to re-render a scene from another point of view, to change the color texture of an object, or to insert a virtual object consistently lit into a real video or photograph. Specifically, the estimation of a point light source is required for the shadows cast by the inserted object to be consistent with the real scene. We tackle the problem of illumination retrieval given an RGBD image of the scene as an inverse problem: we aim to find the illumination that minimizes the photometric error between the rendered image and the observation. In particular we propose a novel differentiable renderer based on the Blinn-Phong model with cast shadows. We compare our differentiable renderer to state-of-the-art methods and demonstrate its robustness to an incorrect reflectance estimation.
A way to cope with cast shadows is to model illumination as a set of candidates, either point sources @cite_26 or directional sources @cite_2 . Sato al @cite_2 propose to estimate the illumination distribution by an adaptive sampling of the directional sources. They implicitly make use of shadows by incorporating a shadow term in their Bidirectional Reflectance Distribution Function (BRDF), computed from the geometry and the potential directional light candidates. In @cite_26 , Jiddi al approximate a set of point light sources equally distributed around the scene, and select the candidates whose cast shadows correlate with a binary mask of shadows preliminary detected in the image. Our method differs from @cite_26 and @cite_2 in that we perform a continuous optimization, and we do not approximate a set of candidates. Hence there is no need to compute a set of shadow maps prior to the optimization: a shadow map is rendered at each iteration given the current illumination estimate.
@cite_26 @cite_2
2104487652 2057552653 1998270967 2151159838
This paper describes a new method for estimating the illumination distribution of a real scene from a radiance distribution inside shadows cast by an object in the scene. First, the illumination distribution of the scene is approximated by discrete sampling of an extended light source. Then the illumination distribution of the scene is estimated from a radiance distribution inside shadows cast by an object of known shape onto another object in the scene. Instead of assuming any particular reflectance properties of the surface inside the shadows, both the illumination distribution of the scene and the reflectance properties of the surface are estimated simultaneously, based on iterative optimization framework. In addition, this paper introduces an adaptive sampling of the illumination distribution of a scene. Rather than using a uniform discretization of the overall illumination distribution, we adaptively increase sampling directions of the illumination distribution based on the estimation at the previous iteration. Using the adaptive sampling framework, we are able to estimate overall illumination more efficiently by using fewer sampling directions. The proposed method is effective for estimating an illumination distribution even under a complex illumination environment. We introduce a method to compute intrinsic images for a multiview set of outdoor photos with cast shadows, taken under the same lighting. We use an automatic 3D reconstruction from these photos and the sun direction as input and decompose each image into reflectance and shading layers, despite the inaccuracies and missing data of the 3D model. Our approach is based on two key ideas. First, we progressively improve the accuracy of the parameters of our image formation model by performing iterative estimation and combining 3D lighting simulation with 2D image optimization methods. Second, we use the image formation model to express reflectance as a function of discrete visibility values for shadow and light, which allows to introduce a robust visibility classifier for pairs of points in a scene. This classifier is used for shadow labeling, allowing to compute high-quality reflectance and shading layers. Our multiview intrinsic decomposition is of sufficient quality to allow relighting of the input images. We create shadow-caster geometry which preserves shadow silhouettes and, using the intrinsic layers, we can perform multiview relighting with moving cast shadows. We present results on several multiview datasets, and show how it is now possible to perform image-based rendering with changing illumination conditions. Recently, a method for removing shadows from colour images was developed ( in IEEE Trans. Pattern Anal. Mach. Intell. 28:59---68, 2006) that relies upon finding a special direction in a 2D chromaticity feature space. This "invariant direction" is that for which particular colour features, when projected into 1D, produce a greyscale image which is approximately invariant to intensity and colour of scene illumination. Thus shadows, which are in essence a particular type of lighting, are greatly attenuated. The main approach to finding this special angle is a camera calibration: a colour target is imaged under many different lights, and the direction that best makes colour patch images equal across illuminants is the invariant direction. Here, we take a different approach. In this work, instead of a camera calibration we aim at finding the invariant direction from evidence in the colour image itself. Specifically, we recognize that producing a 1D projection in the correct invariant direction will result in a 1D distribution of pixel values that have smaller entropy than projecting in the wrong direction. The reason is that the correct projection results in a probability distribution spike, for pixels all the same except differing by the lighting that produced their observed RGB values and therefore lying along a line with orientation equal to the invariant direction. Hence we seek that projection which produces a type of intrinsic, independent of lighting reflectance-information only image by minimizing entropy, and from there go on to remove shadows as previously. To be able to develop an effective description of the entropy-minimization task, we go over to the quadratic entropy, rather than Shannon's definition. Replacing the observed pixels with a kernel density probability distribution, the quadratic entropy can be written as a very simple formulation, and can be evaluated using the efficient Fast Gauss Transform. The entropy, written in this embodiment, has the advantage that it is more insensitive to quantization than is the usual definition. The resulting algorithm is quite reliable, and the shadow removal step produces good shadow-free colour image results whenever strong shadow edges are present in the image. In most cases studied, entropy has a strong minimum for the invariant direction, revealing a new property of image formation. We present an algorithm for performing Lambertian photometric stereo in the presence of shadows. The algorithm has three novel features. First, a fast graph cuts based method is used to estimate per pixel light source visibility. Second, it allows images to be acquired with multiple illuminants, and there can be fewer images than light sources. This leads to better surface coverage and improves the reconstruction accuracy by enhancing the signal to noise ratio and the condition number of the light source matrix. The ability to use fewer images than light sources means that the imaging effort grows sublinearly with the number of light sources. Finally, the recovered shadow maps are combined with shading information to perform constrained surface normal integration. This reduces the low frequency bias inherent to the normal integration process and ensures that the recovered surface is consistent with the shadowing configuration The algorithm works with as few as four light sources and four images. We report results for light source visibility detection and high quality surface reconstructions for synthetic and real datasets.