We consider the problem of face swapping in images, where an input identity is transformed into a target identity while preserving pose, facial expression, and lighting. To perform this mapping, we use convolutional neural networks trained to capture the appearance of the target identity from an unstructured collection of his her photographs.This approach is enabled by framing the face swapping problem in terms of style transfer, where the goal is to render an image in the style of another one. Building on recent advances in this area, we devise a new loss function that enables the network to produce highly photorealistic results. By combining neural networks with simple pre- and post-processing steps, we aim at making face swap work in real-time with no input from the user.
One notable approach trying to solve the related problem of pupeteering -- that is, controlling the expression of one face with another face -- was presented by Suwajanakorn al @cite_26 . The core idea is to build a 3D model of both the input and the replacement face from a large number of images. That is, it only works well where a few hundred images are available but cannot be applied to single images.
@cite_26
2951461827 2134724684 2276844532 2107813907
We present a novel approach for synthesizing photo-realistic images of people in arbitrary poses using generative adversarial learning. Given an input image of a person and a desired pose represented by a 2D skeleton, our model renders the image of the same person under the new pose, synthesizing novel views of the parts visible in the input image and hallucinating those that are not seen. This problem has recently been addressed in a supervised manner, i.e., during training the ground truth images under the new poses are given to the network. We go beyond these approaches by proposing a fully unsupervised strategy. We tackle this challenging scenario by splitting the problem into two principal subtasks. First, we consider a pose conditioned bidirectional generator that maps back the initially rendered image to the original pose, hence being directly comparable to the input image without the need to resort to any training image. Second, we devise a novel loss function that incorporates content and style terms, and aims at producing images of high perceptual quality. Extensive experiments conducted on the DeepFashion dataset demonstrate that the images rendered by our model are very close in appearance to those obtained by fully supervised approaches. We address the problem of reconstructing 3D face models from large unstructured photo collections, e.g., obtained by Google image search or from personal photo collections in iPhoto. This problem is extremely challenging due to the high degree of variability in pose, illumination, facial expression, non-rigid changes in face shape and reflectance over time and occlusions. In light of this extreme variability, no single reconstruction can be consistent with all of the images. Instead, we define as the goal of reconstruction to recover a model that is locally consistent with the image set. I.e., each local region of the model is consistent with a large set of photos, resulting in a model that captures the dominant trends in the input data for different parts of the face. Our approach leverages multi-image shading, but unlike traditional photometric stereo approaches, allows for changes in viewpoint and shape. We optimize over pose, shape, and lighting in an iterative approach that seeks to minimize the rank of the transformed images. This approach produces high quality shape models for a wide range of celebrities from photos available on the Internet. State-of-the-art methods reconstruct three-dimensional (3D) face shapes from a single image by fitting 3D face models to input images or by directly learning mapping functions between two-dimensional (2D) images and 3D faces. However, they are often difficult to use in real-world applications due to expensive online optimization or to the requirement of frontal face images. This paper approaches the 3D face reconstruction problem as a regression problem rather than a model fitting problem. Given an input face image along with some pre-defined facial landmarks on it, a series of shape adjustments to the initial 3D face shape are computed through cascaded regressors based on the deviations between the input landmarks and the landmarks obtained from the reconstructed 3D faces. The cascaded regressors are offline learned from a set of 3D faces and their corresponding 2D face images in various views. By treating the landmarks that are invisible in large view angles as missing data, the proposed method can handle arbitrary view face images in a unified way with the same regressors. Experiments on the BFM and Bosphorus databases demonstrate that the proposed method can reconstruct 3D faces from arbitrary view images more efficiently and more accurately than existing methods. This paper presents a method for photo-realistic animation of any face shown in a single image or a video. The technique does not require example data of the person’s mouth movements, and the image to be animated is not restricted in pose and illumination. Video reanimation allows for head rotations and speech in the original sequence, yet neither of these motions is required. In order to animate novel faces, the system transfers mouth movements and expressions across individuals, based a common representation of different identities and facial expressions in a vector space of 3D shapes and textures. This space is computed from 3D scans of different neutral faces, and scans of facial expressions. The 3D model’s versatility with respect to pose and illumination is conveyed to photo-realistic image and video processing by a framework of analysis and synthesis algorithms: The system automatically estimates 3D shape, pose and other rendering parameters from single images, and tracks head pose and mouth movements in video. Reanimated with new mouth movements, the 3D face is rendered into the original images.