FIELD

The present invention relates to image processing, in particular to estimating three-dimensional shape and spatially-varying reflectance of an object from a set of images of the object.

BACKGROUND

Accurately acquiring the shape and appearance of real-world objects and materials has so been an active area of research in vision and graphics with a wide range of applications including, for example, analysis/recognition, and digitization for visual effects, games, virtual reality, cultural heritage, advertising and design. Advances in digital imaging over the last two decades has resulted in image-based acquisition techniques becoming an integral component of appearance modelling and three-dimensional (3D) reconstruction.

J. Riviere et al.: “Polarization imaging reflectometry in the wild”, ACM Transactions on Graphics, volume 36, no. 6, Article 206 (2017) describes on-site acquisition of surface reflectance for planar, spatially varying, isotropic samples in uncontrolled outdoor environment. It employs linear-polarization imaging from two, near-orthogonal views, close to the Brewster angle of incidence, to maximize polarization cues for surface reflectance estimation.

Z. Li et al.: “Learning to reconstruct shape and spatially-varying reflectance from a single image”, ACM Transactions on Graphics, volume 37, no. 6, Article 269 (2018) (herein referred to as “Li et al.”) describes recovering spatially-varying bidirectional reflectance distribution function (SVBRDFs) and complex geometry from a single RGB image captured under a combination of unknown environment illumination and flash lighting by training a deep neural network to regress shape and reflectance from the image.

V. Deschaintre et al.: “Single-Image SVBRDF Capture with a Rendering-Aware Deep Network”, ACM Transactions on Graphics, volume 37, no. 4, Article 128 (2018) (herein referred to as “Deschaintre et al.”) describes using a neural network to reconstruct complex SVBRDFs of planar samples given a single input photograph under flash illumination, based on training using only synthetic data.

A. Kadambi et al.: “Polarized 3D: High-quality depth sensing with polarization cues”, Proceedings of the IEEE International Conference on Computer Vision, pages 3370-3378 (2015) (herein referred to as “Kadambi et al.”) describes using polarization enhance depth maps obtained using a Microsoft® Kinect depth sensor. Y. Ba et al.: “Deep shape from polarization”, European Conference on Computer Vision (ECCV), 2020 (herein referred to as “Ba et al.”) describes a deep learning-based approach to inferring the shape of a surface under uncontrolled environment illumination using polarization imaging. Both Kadambi et al. and Ba et al. only estimate shape.

M. Boss et al.: “Two-shot spatially-varying brdf and shape estimation”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020 (herein referred to as “Boss et al.”) describes a cascaded network and guided prediction networks for SVBRDF and shape estimation from two-shot images, under flash and ambient environmental illumination respectively.

SUMMARY

According to a first aspect of the present invention there is provided a method comprising receiving a set of at least three images of an object including at least two linearly-polarized images (for example, at least two linearly-polarized color images) and at least one color image (which may or may not be linearly-polarized), wherein the three images have the same view of the object and are acquired under the same illumination condition (in other words, for each of the at least three images, the object is illuminated in the same way, e.g., from the same, single fixed point, from the same, multiple fixed points, or from the same fixed range or extent of illumination) in which either diffuse polarization or specular polarization dominates in surface reflectance, and wherein a set of Stokes parameters s0, s1and s2is determinable from the at least three images. The method further comprises generating three-dimensional shape and spatially-varying reflectance of the object from the set of at least three images using a deep neural network trained with a plurality of sets of training images, each of the plurality of sets of training images including at least three training images including at least two linearly-polarized training images and at least one color image from which a respective set of Stokes parameters s0, s1and s2is determinable and storing said three-dimensional shape and spatially-varying reflectance generated by the deep neural network.

The three-dimensional shape and spatially-varying reflectance can be used to render a high-quality image of the object under new lighting conditions.

The images are preferably acquired under controlled illumination, for example, indoors or, if outdoors, under cloudy conditions or other suitably shaded conditions providing uniform illumination, whereby polarized illumination is minimised or minimal such that it is not dominant.

The illumination may be flash illumination such that diffuse polarization dominates and, thus, the Stokes map may be based on diffuse polarization. The illumination may be unpolarized. The flash illumination may, however, be linearly polarized or circularly polarized. The flash illumination may include a mixture of polarized light (linearly- and/or circularly-polarized light) and/or unpolarized light.

The illumination may be uniform and surround the object (e.g., spherical or hemispherical illumination) such that specular polarization dominates. The uniform illumination may be unpolarized or circularly-polarized for non-planar 3D objects. The uniform illumination may include a mixture of circularly-polarized light and unpolarized light for non-planar 3D objects. For a planar object, the uniform illumination may be from an extended or a sufficiently large area light source or light panel or display panel, or even locally uniform environmental illumination. For planar objects, the uniform illumination from an extended area-light may be unpolarized, linearly-polarized or circularly-polarized.

The set of at least three images may comprise at least three color images. The at least two linearly-polarized images and at least one color image may comprise at least two linearly-polarized colour images.

A set of Stokes parameters s0, s1and s2is determinable from the at least three images, for example, if the at least two linearly-polarized images include first and second linearly-polarized images in which the angle of polarization between the first and second images are separated by 45°. A set of Stokes parameters s0, s1and s2is determinable from the at least three images, for example, if the at least two linearly-polarized images include first, second and third linearly-polarized images in which the angle of polarization are 0°, 45° and 90° respectively. The set of linear Stokes parameters may be determined by a different combination of angles of polarization, such as, for example, 0°, 60° and 120° respectively.

At least the unpolarised Stokes parameter s0has color. The horizontally polarized reflectance Stokes parameter s1and/or the polarization reflectance Stokes parameter s1may have color.

The object may be a three-dimensional object, that is, an object which is not substantially flat or planar, and/or includes one or more convex surface(s). The three-dimensional object may include whole or part of a human subject (e.g., face or full-body), an animal or a plant. The object, however, may be a planar object.

The method may further comprise receiving a polarization shape map generated from the Stokes parameters s1and s2for the object and/or a colour map and/or a degree of polarization (DOP) map, or generating a polarization shape map from the Stokes parameters s1and s2for the object and/or a colour map and/or a DOP map using the set of at least three images. The three-dimensional shape and spatially-varying reflectance may be generated from the set of at least three images and the polarization shape map and/or the colour map and/or the DOP map.

The color map may be a diffuse color map. The polarization shape map may be a normalised Stokes map or an angle of polarization map. The degree of polarization (DOP) map may be a diffuse DOP map or a specular DOP map.

The plurality of sets of training images may comprise a plurality of sets of synthesized training images. For example, the plurality of sets of synthesized training images may so be generated using a plurality of meshes of objects and a plurality of different spatially-varying bidirectional reflectance distribution function (SVBRDs) corresponding to different materials. Generation of a training image may include selecting a mesh and a material and randomly rotating the mesh and material.

Additionally or alternatively, the plurality of sets of training images may comprise a plurality of sets of measured training images.

The at least three images of the object may comprise three or four linearly-polarized images, for example, three or four linearly-polarized color images.

The three-dimensional shape may comprise a surface normal map and a depth map. The spatially-varying reflectance may comprise a diffuse albedo map, and a specular albedo map, and/or a specular roughness map.

The deep neural network may comprise a convolutional neural network having an encoder and a decoder and skip connections between the encoder and decoder. The decoder may be a branched decoder comprising at least two branches. The skip connections may include at least one residual block or a series of at least two residual blocks. The deep neural network trained by considering rendering losses for each linearly-polarized image. The deep network may include a parallel arrangement of a U-Net image-to-image network and a global features network.

The set of at least three images may be acquired using frontal flash illumination (which may be unpolarized, or linearly or circularly polarized) incident on the object so as to cause diffuse polarization to dominate in the surface reflectance. The frontal illumination can be from a flash or a projector. Alternatively, the set of at least three images may be acquired using uniform illumination (which may be unpolarized or circularly polarized) disposed around and directed at the object so as to cause specular polarization to dominate in the surface reflectance. The uniform illumination may comprise a plurality of light sources arranged in a hemisphere or sphere around the object, or surrounding the object, to provide uniform illumination on the object.

If the object is a planar object, uniform illumination can be achieved using an extended or a sufficiently large area-light source or light panel or display panel, or locally-uniform environmental illumination incident on the object at near normal incidence or so obliquely incident at near Brewster angle of incidence. For a planar object, the uniform illumination may be unpolarized, linearly polarized or circularly polarized.

According to a second aspect of the present invention there is provided a method comprising receiving a set of linearly-polarized color images of an object, each linearly-polarized image having a different angle of polarization, the linearly-polarized color images having the same view of the object and acquired using unpolarized, frontal, flash illumination of the object. The method may optionally include receiving a reflectance map and a shape map for the object generated from the set of linearly-polarized images. The method comprises generating three-dimensional shape and spatially-varying reflectance of the object from the set of linearly-polarized images, and optionally the reflectance map and the shape map, using a deep neural network trained with a synthetic or measured dataset, wherein the synthetic or measured dataset includes a plurality of sets of data, each set of data including a set of linearly-polarized images having different polarizations, and optionally a reflectance map and a shape map generated from the linearly-polarized images, and ground truth three-dimensional shape and spatially-varying reflectance and storing said three-dimensional shape and spatially-varying reflectance generated by the deep neural network

According to a third aspect of the present invention is provided a computer program comprising instructions for performing the method of the first or second aspect.

According to a fourth aspect of the present invention is provided a computer program product comprising a computer readable medium (which may be non-transitory) storing the computer program of the third aspect.

According to a fifth aspect of the present invention there is provided a device comprising at least one processor and storage. The at least one processor is configured, in response to receiving a set of at least three images of an object including at least two linearly-polarized images and at least one color image, wherein the three images have the same view of the object and are acquired under the same illumination condition in which either diffuse polarization or specular polarization dominates, wherein a set of Stokes parameters s0, s1and s2is determinable from the at least three images, to generate three-dimensional shape and spatially-varying reflectance of the object from the set of at least three images using a deep neural network trained with a plurality of sets of training images, each of the plurality of sets of training images including at least three training images including at least two linearly-polarized training images and at least one color image from which a respective set of Stokes parameters s0, s1and s2is determinable and to store said three-dimensional shape and spatially-varying reflectance generated by the deep neural network in the storage.

The at least one processor may receive a polarization shape map generated from the Stokes parameters s1and s2for the object and/or a colour map and/or a degree of polarization (DOP) map. The at least one processor may further be configured to generate a polarization shape map from the Stokes parameters s1and s2for the object and/or a colour map and/or a DOP map using the set of at least three color images. The at least one processor may be configured to generate three-dimensional shape and spatially-varying reflectance from the set of at least three color images and the polarization shape map and/or the colour map and/or the DOP map.

The device may further comprise a color digital camera and a linear polarizing filter for acquiring the at least three color images.

The device may further comprise or be provided with a flash or a projector for providing directional illumination on the object, preferably from a frontal direction. The device may further comprise or be provided with a one or more light sources (for example, light emitting diodes, light panels or display panels) and, optionally, one or more reflecting surfaces arranged around the object to provide uniform illumination on the object. Light from the one or more light sources may be bounced from the one or more reflecting surfaces(s).

The one or more light sources may comprise a plurality of light sources arranged in a hemisphere or sphere around the object. The one or more reflecting surfaces may comprise plurality of reflecting surfaces arranged in a hemisphere or sphere around the object. The reflecting surface(s) may be concave. The reflecting surface(s) may provide diffuse reflection.

The at least one processor may include one or more central processing units (CPUs). The at least one processor may include one or more graphical processing units (GPUs).

According to a sixth aspect of the present invention there is provided a method of training a deep neural network. The method comprises providing a plurality of sets of training images and corresponding ground truth three-dimensional shape and spatially-varying reflectance of objects to a deep neural network, each set of training images including at least three training images including at least two linearly-polarized training images (for example, at least two linearly-polarized color images) and at least one color image (which may or may not be linearly-polarized) from which a respective set of Stokes parameters s0, s1and s2is determinable; and storing the trained deep neural network.

The method may further comprise providing a polarization shape map generated from the Stokes parameters s1and s2and/or a colour map and/or a DOP map.

The set of training images may comprise a plurality of sets of synthesized training images and/or measured training images.

According to a seventh aspect of the present invention is provided a computer program comprising instructions for performing the method of the sixth aspect.

According to an eighth aspect of the present invention is provided a computer program product comprising a computer readable medium (which may be non-transitory) storing the computer program of the seventh aspect.

According to a ninth aspect of the present invention there is provided apparatus for comprising at least one processor and storage for training a deep neural network.

