Sketch portrait generation benefits a wide range of applications such as digital entertainment and law enforcement. Although plenty of efforts have been dedicated to this task, several issues still remain unsolved for generating vivid and detail-preserving personal sketch portraits. For example, quite a few artifacts may exist in synthesizing hairpins and glasses, and textural details may be lost in the regions of hair or mustache. Moreover, the generalization ability of current systems is somewhat limited since they usually require elaborately collecting a dictionary of examples or carefully tuning features components. In this paper, we present a novel representation learning framework that generates an end-to-end photo-sketch mapping through structure and texture decomposition. In the training stage, we first decompose the input face photo into different components according to their representational contents (i.e., structural and textural parts) by using a pre-trained convolutional neural network (CNN). Then, we utilize a branched fully CNN for learning structural and textural representations, respectively. In addition, we design a sorted matching mean square error metric to measure texture patterns in the loss function. In the stage of sketch rendering, our approach automatically generates structural and textural representations for the input photo and produces the final result via a probabilistic fusion scheme. Extensive experiments on several challenging benchmarks suggest that our approach outperforms example-based synthesis algorithms in terms of both perceptual and objective metrics. In addition, the proposed method also has better generalization ability across data set without additional training.
Most works in sketch portrait generation focus on two kinds of sketches, namely profile sketches @cite_17 and shading sketches @cite_18 . Compared with the former, the shading sketches can not only use lines to reflect the overall profiles, but also capture the textural parts via shading. Thus, the shading sketches are more challenge to be modeled. We mainly study the automatic generation of shading sketches in this paper.
@cite_18 @cite_17
2097902422 2771349609 1993935870 2797225766
Face sketches are being used in eyewitness testimonies for about a century. These sketches are crucial in finding suspects when no photo is available, but a mental image in the eyewitness's mind. However, research shows that current procedures used for eyewitness testimonies have two main problems. First, they can significantly disturb the memories of the eyewitness. Second, in many cases, these procedures result in face images far from their target faces. These two problems are related to the plasticity of the human visual system and the differences between face perception in humans (holistic) and current methods of sketch production (piecemeal). In this paper, we present some insights for more realistic sketch to photo matching. We describe how to retrieve identity specific information from crude sketches, directly drawn by the non-artistic eyewitnesses. The sketches we used merely contain facial component outlines and facial marks (e.g. wrinkles and moles). We compare results of automatically matching two types sketches (trace-over and user-provided, 25 each) to four types of faces (original, locally exaggerated, configurally exaggerated, and globally exaggerated, 249 each), using two methods (PDM distance comparison and PCA classification). Based on our results, we argue that for automatic non-artistic sketch to photo matching, the algorithms should compare the user-provided sketches with globally exaggerated faces, with a soft constraint on facial marks, to achieve the best matching rates. This is because the user-provided sketch from the user's mental image, seems to be caricatured both locally and configurally. Sketch portrait generation is of wide applications including digital entertainment and law enforcement. Despite the great progress achieved by existing face sketch generation methods, they mostly yield blurred effects and great deformation over various facial parts. In order to tackle this challenge, we propose a novel composition-aided generative adversarial network (CA-GAN) for sketch portrait generation. First, we utilize paired inputs including a face photo and the corresponding pixel-wise face labels for generating the portrait. Second, we propose an improved pixel loss, termed compositional loss, to focus training on hard-generated components and delicate facial structures. Moreover, we use stacked CA-GANs (stack-CA-GAN) to further rectify defects and add compelling details. Experimental results show that our method is capable of generating identity-preserving, sketch-realistic, and visually comfortable sketch portraits over a wide range of challenging data, and outperforms existing methods. Besides, our methods show considerable generalization ability. We use a data-driven approach to study both style and abstraction in sketching of a human face. We gather and analyze data from a number of artists as they sketch a human face from a reference photograph. To achieve different levels of abstraction in the sketches, decreasing time limits were imposed -- from four and a half minutes to fifteen seconds. We analyzed the data at two levels: strokes and geometric shape. In each, we create a model that captures both the style of the different artists and the process of abstraction. These models are then used for a portrait sketch synthesis application. Starting from a novel face photograph, we can synthesize a sketch in the various artistic styles and in different levels of abstraction. Human free-hand sketches have been studied in various contexts including sketch recognition, synthesis and fine-grained sketch-based image retrieval (FG-SBIR). A fundamental challenge for sketch analysis is to deal with drastically different human drawing styles, particularly in terms of abstraction level. In this work, we propose the first stroke-level sketch abstraction model based on the insight of sketch abstraction as a process of trading off between the recognizability of a sketch and the number of strokes used to draw it. Concretely, we train a model for abstract sketch generation through reinforcement learning of a stroke removal policy that learns to predict which strokes can be safely removed without affecting recognizability. We show that our abstraction model can be used for various sketch analysis tasks including: (1) modeling stroke saliency and understanding the decision of sketch recognition models, (2) synthesizing sketches of variable abstraction for a given category, or reference object instance in a photo, and (3) training a FG-SBIR model with photos only, bypassing the expensive photo-sketch pair collection step.