Deep Learning for Free-Hand Sketch: A Survey <s> Sketch-3D Generation <s> Procedural modeling techniques can produce high quality visual content through complex rule sets. However, controlling the outputs of these techniques for design purposes is often notoriously difficult for users due to the large number of parameters involved in these rule sets and also their non-linear relationship to the resulting content. To circumvent this problem, we present a sketch-based approach to procedural modeling. Given an approximate and abstract hand-drawn 2D sketch provided by a user, our algorithm automatically computes a set of procedural model parameters, which in turn yield multiple, detailed output shapes that resemble the user's input sketch. The user can then select an output shape, or further modify the sketch to explore alternative ones. At the heart of our approach is a deep Convolutional Neural Network (CNN) that is trained to map sketches to procedural model parameters. The network is trained by large amounts of automatically generated synthetic line drawings. By using an intuitive medium, i.e., freehand sketching as input, users are set free from manually adjusting procedural model parameters, yet they are still able to create high quality content. We demonstrate the accuracy and efficacy of our method in a variety of procedural modeling scenarios including design of man-made and organic shapes. <s> BIB001 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch-3D Generation <s> Face modeling has been paid much attention in the field of visual computing. There exist many scenarios, including cartoon characters, avatars for social media, 3D face caricatures as well as face-related art and design, where low-cost interactive face modeling is a popular approach especially among amateur users. In this paper, we propose a deep learning based sketching system for 3D face and caricature modeling. This system has a labor-efficient sketching interface, that allows the user to draw freehand imprecise yet expressive 2D lines representing the contours of facial features. A novel CNN based deep regression network is designed for inferring 3D face models from 2D sketches. Our network fuses both CNN and shape based features of the input sketch, and has two independent branches of fully connected layers generating independent subsets of coefficients for a bilinear face representation. Our system also supports gesture based interactions for users to further manipulate initial face models. Both user studies and numerical results indicate that our sketching system can help users create face models quickly and effectively. A significantly expanded face database with diverse identities, expressions and levels of exaggeration is constructed to promote further research and evaluation of face modeling techniques. <s> BIB002 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch-3D Generation <s> 3D objects modeling has gained considerable attention in the visual computing community. We propose a low-cost unsupervised learning model for 3D objects reconstruction from hand-drawn sketches. Recent advancements in deep learning opened new opportunities to learn high-quality 3D objects from 2D sketches via supervised networks. However, the limited availability of labeled 2D hand-drawn sketches data (i.e. sketches and its corresponding 3D ground truth models) hinders the training process of supervised methods. In this paper, driven by a novel design of combination of retrieval and reconstruction process, we developed a learning paradigm to reconstruct 3D objects from hand-drawn sketches, without the use of well-labeled hand-drawn sketch data during the entire training process. Specifically, the paradigm begins with the training of an adaption network via autoencoder with adversarial loss, embedding the unpaired 2D rendered image domain with the hand-drawn sketch domain to a shared latent vector space. Then from the embedding latent space, for each testing sketch image, we retrieve a few (e.g. five) nearest neighbors from the training 3D data set as prior knowledge for a 3D Generative Adversarial Network. Our experiments verify our network's robust and superior performance in handling 3D volumetric object generation from single hand-drawn sketch without requiring any 3D ground truth labels. <s> BIB003 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch-3D Generation <s> We present sketchhair, a deep learning based tool for interactive modeling of 3D hair from 2D sketches. Given a 3D bust model as reference, our sketching system takes as input a user-drawn sketch (consisting of hair contour and a few strokes indicating the hair growing direction within a hair region), and automatically generates a 3D hair model, which matches the input sketch both globally and locally. The key enablers of our system are two carefully designed neural networks, namely, S2ONet, which converts an input sketch to a dense 2D hair orientation field; and O2VNet, which maps the 2D orientation field to a 3D vector field. Our system also supports hair editing with additional sketches in new views. This is enabled by another deep neural network, V2VNet, which updates the 3D vector field with respect to the new sketches. All the three networks are trained with synthetic data generated from a 3D hairstyle database. We demonstrate the effectiveness and expressiveness of our tool using a variety of hairstyles and also compare our method with prior art. <s> BIB004 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch-3D Generation <s> Freehand sketches are an important medium for expressing and communicating ideas. However creating a meaningful and understandable sketch drawing is not always an easy task especially for unskillful users. Existing methods for rendering 3D shape into line drawings such as Suggestive Contours, only consider the geometry-dependent and view-dependent information thus leads to over-regular or over-perfect results which doesn’t look like a human freehand drawing. For this challenge we address the problem of producing freehand line drawing sketches from a 3D object under a given viewpoint automatically. The core solution here is a recurrent generative deep neural network, which learns a functional mapping from the suggestive contours of a 3D shape to a more abstract sketch representation. We drop the encoder of the generator, i.e., use only a decoder to achieve better stability of the sketch structure. Users can tune the level of freehand style of the generated sketches by changing a single parameter. Experiments show that our results are expressive enough to faithfully describe the input shape and at the same time be with the style of freehand drawings created by a real human. We also perform a comparative user study to verify the quality and style of generated sketch results over existing methods. We also retrain our network using several different mingled dataset to test the extendibility of our method for this particular application. As far as our knowledge this work is the first research effort to automate the generation of human-like freehand sketches directly from 3D shapes. <s> BIB005
The generation across sketch and 3D model is also an interesting cross-modal research topic. Using sketch to generate 3D models/shapes BIB003 is an extremely challenging and application-driven task, which has some application scenarios, e.g., sketch-based product design BIB004 . Due to the task-unique challenges, this task is under-studied to date. The existing deep learning based sketch-to-3D generation models are engineered for the highly well-drawn even or professional pencil sketches BIB001 , BIB002 . Recently, 3D-to-sketch BIB005 generation also has been studied in deep learning manner.