This paper addresses the problem of photometric stereo for non-Lambertian surfaces. Existing approaches often adopt simplified reflectance models to make the problem more tractable, but this greatly hinders their applications on real-world objects. In this paper, we propose a deep fully convolutional network, called PS-FCN, that takes an arbitrary number of images of a static object captured under different light directions with a fixed camera as input, and predicts a normal map of the object in a fast feed-forward pass. Unlike the recently proposed learning based method, PS-FCN does not require a pre-defined set of light directions during training and testing, and can handle multiple images and light directions in an order-agnostic manner. Although we train PS-FCN on synthetic data, it can generalize well on real datasets. We further show that PS-FCN can be easily extended to handle the problem of uncalibrated photometric stereo.Extensive experiments on public real datasets show that PS-FCN outperforms existing approaches in calibrated photometric stereo, and promising results are achieved in uncalibrated scenario, clearly demonstrating its effectiveness.
Many sophisticated reflectance models have been proposed to approximate the non-Lambertian model, including analytical models like Torrance-Sparrow model @cite_8 , Ward model @cite_34 , Cook-Torrance model @cite_33 , etc. Instead of rejecting specular observations as outliers, sophisticated reflectance model based methods fit an analytical model to all observations. These methods require solving complex optimization problems, and can only handle limited classes of materials. Recently, bivariate BRDF representations @cite_20 @cite_39 were adopted to approximate isotropic BRDF, and a symmetry-based approach @cite_2 was proposed to handle anisotropic reflectance without explicitly estimating a reflectance model.
@cite_33 @cite_8 @cite_39 @cite_2 @cite_34 @cite_20
2165477640 1996352068 2962741401 2051218308
Under the Lambertian reflectance model, uncalibrated photometric stereo with unknown light sources is inherently ambiguous. In this paper, we consider the use of a more general reflectance model, namely the Torrance and Sparrow model, in uncalibrated photometric stereo. We demonstrate that this can not only resolve the ambiguity when the light sources are unknown, but can also result in more accurate surface reconstructions and can capture the reflectance properties of a large number of nonLambertian surfaces. Our method uses single light source images with unknown lighting and no knowledge about the parameters of the reflectance model. It can recover the 3D shape of surfaces (up to the binary convex concave ambiguity) together with their reflectance properties. We have successfully tested our algorithm on a variety of nonLambertian surfaces demonstrating the effectiveness of our approach. In the case of human faces, the estimated skin reflectance has been shown to closely resemble the measured skin reflectance reported in the literature. We also demonstrate improved recognition results on 4050 images of 10 faces with variable lighting and viewpoint when the synthetic image-based representations of the faces are generated using the surface reconstructions and reflectance properties recovered while assuming the extended reflectance model. This paper provides an introduction to photometric methods for image-based 3D shape reconstruction and a survey of photometric stereo techniques. We begin with taxonomy of active and passive shape acquisition techniques. Then we describe the methodical background of photometric 3D reconstruction, define the canonical setting of photometric stereo (Lambertian surface reflectance, parallel incident light, known illumination direction, known surface albedo, absence of cast shadows), discuss the 3D reconstruction of surfaces from local gradients, summarize the concept of the bidirectional reflectance distribution function (BRDF), and outline several important empirically and physically motivated reflectance models. We provide a detailed treatment of several generalizations of the canonical setting of photometric stereo, namely non-distant light sources, unknown illumination directions, and, in some detail, non-Lambertian surface reflectance functions. An important special case is purely specular reflections, where an extended light source allows capturing a surface that consists of perfectly specular surface patches. Linear combinations of purely Lambertian and purely specular reflectance components are favorably used for reconstructing smooth surfaces and also human skin. Nonuniform surface reflectance properties are estimated based on a simultaneous 3D reconstruction and determination of the locally variable parameters of the reflectance function based on a multitude of images. Assuming faceted surfaces, the effective resolution of the 3D reconstruction result can be increased to some extent beyond that of the underlying images. Other approaches separate specular and diffuse reflectance components based on polarization data or color information. The specular reflections can be used additionally to estimate the direction from which the surface is illuminated. Finally, we describe methods to combine photometric 3D reconstruction techniques with active and passive triangulation-based approaches. Estimating surface reflectance (BRDF) is one key component for complete 3D scene capture, with wide applications in virtual reality, augmented reality, and human computer interaction. Prior work is either limited to controlled environments (e.g., gonioreflectometers, light stages, or multi-camera domes), or requires the joint optimization of shape, illumination, and reflectance, which is often computationally too expensive (e.g., hours of running time) for real-time applications. Moreover, most prior work requires HDR images as input which further complicates the capture process. In this paper, we propose a lightweight approach for surface reflectance estimation directly from 8-bit RGB images in real-time, which can be easily plugged into any 3D scanning-and-fusion system with a commodity RGBD sensor. Our method is learning-based, with an inference time of less than 90ms per scene and a model size of less than 340K bytes. We propose two novel network architectures, HemiCNN and Grouplet, to deal with the unstructured input data from multiple viewpoints under unknown illumination. We further design a loss function to resolve the color-constancy and scale ambiguity. In addition, we have created a large synthetic dataset, SynBRDF, which comprises a total of 500K RGBD images rendered with a physically-based ray tracer under a variety of natural illumination, covering 5000 materials and 5000 shapes. SynBRDF is the first large-scale benchmark dataset for reflectance estimation. Experiments on both synthetic data and real data show that the proposed method effectively recovers surface reflectance, and outperforms prior work for reflectance estimation in uncontrolled environments. This paper presents a technique for acquiring the shape of real-world objects with complex isotropic and anisotropic reflectance. Our method estimates the local normal and tangent vectors at each pixel in a reference view from a sequence of images taken under varying point lighting. We show that for many real-world materials and a restricted set of light positions, the 2D slice of the BRDF obtained by fixing the local view direction is symmetric under reflections of the halfway vector across the normal-tangent and normal-binormal planes. Based on this analysis, we develop an optimization that estimates the local surface frame by identifying these planes of symmetry in the measured BRDF. As with other photometric methods, a key benefit of our approach is that the input is easy to acquire and is less sensitive to calibration errors than stereo or multi-view techniques. Unlike prior work, our approach allows estimating the surface tangent in the case of anisotropic reflectance. We confirm the accuracy and reliability of our approach with analytic and measured data, present several normal and tangent fields acquired with our technique, and demonstrate applications to appearance editing.