We address the problem of recovering the 3D geometry of a human face from a set of facial images in multiple views. While recent studies have shown impressive progress in 3D Morphable Model (3DMM) based facial reconstruction, the settings are mostly restricted to a single view. There is an inherent drawback in the single-view setting: the lack of reliable 3D constraints can cause unresolvable ambiguities. We in this paper explore 3DMM-based shape recovery in a different setting, where a set of multi-view facial images are given as input. A novel approach is proposed to regress 3DMM parameters from multi-view inputs with an end-to-end trainable Convolutional Neural Network (CNN). Multiview geometric constraints are incorporated into the network by establishing dense correspondences between different views leveraging a novel self-supervised view alignment loss. The main ingredient of the view alignment loss is a differentiable dense optical flow estimator that can backpropagate the alignment errors between an input view and a synthetic rendering from another input view, which is projected to the target view through the 3D shape to be inferred. Through minimizing the view alignment loss, better 3D shapes can be recovered such that the synthetic projections from one view to another can better align with the observed image. Extensive experiments demonstrate the superiority of the proposed method over other 3DMM methods.
Blanz and Vetter @cite_20 introduced the 3D morphable model to represent textured 3D faces using linear combinations of a set of shape and texture bases, which is derived from collections of real 3D face scans. The model is later extended to include facial expressions by FaceWarehouse @cite_35 . In this paper, we focus on recovering the underlying 3D shapes of human faces, hence we are only interested in regressing 3DMM parameters for shapes and expressions. We argue that more realistic textures for 3D meshes can be obtained with more advanced texture synthesis techniques @cite_6 instead of the 3DMM texture representations.
@cite_35 @cite_6 @cite_20
2804621595 2156119076 2237250383 1516424378
3D Morphable Models (3DMMs) are powerful statistical models of 3D facial shape and texture, and are among the state-of-the-art methods for reconstructing facial shape from single images. With the advent of new 3D sensors, many 3D facial datasets have been collected containing both neutral as well as expressive faces. However, all datasets are captured under controlled conditions. Thus, even though powerful 3D facial shape models can be learnt from such data, it is difficult to build statistical texture models that are sufficient to reconstruct faces captured in unconstrained conditions (“in-the-wild”). In this paper, we propose the first “in-the-wild” 3DMM by combining a statistical model of facial identity and expression shape with an “in-the-wild” texture model. We show that such an approach allows for the development of a greatly simplified fitting procedure for images and videos, as there is no need to optimise with regards to the illumination parameters. We have collected three new benchmarks that combine “in-the-wild” images and video with ground truth 3D facial geometry, the first of their kind, and report extensive quantitative evaluations using them that demonstrate our method is state-of-the-art. This paper presents a fully automated algorithm for reconstructing a textured 3D model of a face from a single photograph or a raw video stream. The algorithm is based on a combination of Support Vector Machines (SVMs) and a Morphable Model of 3D faces. After SVM face detection, individual facial features are detected using a novel regression- and classification-based approach, and probabilistically plausible configurations of features are selected to produce a list of candidates for several facial feature positions. In the next step, the configurations of feature points are evaluated using a novel criterion that is based on a Morphable Model and a combination of linear projections. To make the algorithm robust with respect to head orientation, this process is iterated while the estimate of pose is refined. Finally, the feature points initialize a model-fitting procedure of the Morphable Model. The result is a high resolution 3D surface model. In this paper, a new technique for modeling textured 3D faces is introduced. 3D faces can either be generated automatically from one or more photographs, or modeled directly through an intuitive user interface. Users are assisted in two key problems of computer aided face modeling. First, new face images or new 3D face models can be registered automatically by computing dense one-to-one correspondence to an internal face model. Second, the approach regulates the naturalness of modeled faces avoiding faces with an “unlikely” appearance. Starting from an example set of 3D face models, we derive a morphable face model by transforming the shape and texture of the examples into a vector space representation. New faces and expressions can be modeled by forming linear combinations of the prototypes. Shape and texture constraints derived from the statistics of our example faces are used to guide manual modeling or automated matching algorithms. We show 3D face reconstructions from single images and their applications for photo-realistic image manipulations. We also demonstrate face manipulations according to complex parameters such as gender, fullness of a face or its distinctiveness. This thesis presents an approach to the modeling of facial aging using and extending the Morphable Model technique. For modeling the face variation across individuals, facial expressions, and physical attributes, we collected 3D face scans of 298 persons. The 3D face scans where acquired with a structured light 3D scanner, which we improved in collaboration with the manufacturer to achieve superior geometry and texture quality. Moreover, we developed an efficient way to measure fine skin structure and reflection properties with the scanner. The collected face scans have been used to build the Basel Face Model, a new publicly available Morphable Model. Using the 3D scans we learn the correlation between physical attributes such as weight, height, and especially age and faces. With the learned correlation, we present a novel way to simultaneously manipulate different attributes and demonstrate the capability to model changes caused by aging. Using the attributes of the face model in conjunction with a skull model developed in the same research group, we present a method to reconstruct faces from skull shapes which considers physical attributes, as the body weight, age etc. The most important aspect of facial aging that can not be simulated with the Morphable Model is the appearance of facial wrinkles. In this work we present a novel approach to synthesize age wrinkles based on statistics. Our wrinkle synthesis consists of two main parts: The learning of a generative model of wrinkle constellations, and the modeling of their visual appearance. For learning the constellations we use kernel density estimation of manually labeled wrinkles to estimate the wrinkle occurrence probability. To learn the visual appearance of wrinkles we use the fine scale skin structure captured with our improved scanning method. Our results show that the combination of the attribute fitting based aging and the wrinkle synthesis, facilitate a simulation of visually convincing progressive aging. The method is without restrictions applicable to any face that can be represented by the Morphable Model.