Deep Learning for Free-Hand Sketch: A Survey <s> Generation <s> In the first part of this paper, a regular recurrent neural network (RNN) is extended to a bidirectional recurrent neural network (BRNN). The BRNN can be trained without the limitation of using input information just up to a preset future frame. This is accomplished by training it simultaneously in positive and negative time direction. Structure and training procedure of the proposed network are explained. In regression and classification experiments on artificial data, the proposed structure gives better results than other approaches. For real data, classification experiments for phonemes from the TIMIT database show the same tendency. In the second part of this paper, it is shown how the proposed bidirectional structure can be easily modified to allow efficient estimation of the conditional posterior probability of complete symbol sequences without making any explicit assumption about the shape of the distribution. For this part, experiments on real data are reported. <s> BIB001 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Generation <s> We present sketch-rnn, a recurrent neural network (RNN) able to construct stroke-based drawings of common objects. The model is trained on thousands of crude human-drawn images representing hundreds of classes. We outline a framework for conditional and unconditional sketch generation, and describe new robust training methods for generating coherent sketch drawings in a vector format. <s> BIB002 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Generation <s> In the quest towards general artificial intelligence (AI), researchers have explored developing loss functions that act as intrinsic motivators in the absence of external rewards. This paper argues that such research has overlooked an important and useful intrinsic motivator: social interaction. We posit that making an AI agent aware of implicit social feedback from humans can allow for faster learning of more generalizable and useful representations, and could potentially impact AI safety. We collect social feedback in the form of facial expression reactions to samples from Sketch RNN, an LSTM-based variational autoencoder (VAE) designed to produce sketch drawings. We use a Latent Constraints GAN (LC-GAN) to learn from the facial feedback of a small group of viewers, by optimizing the model to produce sketches that it predicts will lead to more positive facial expressions. We show in multiple independent evaluations that the model trained with facial feedback produced sketches that are more highly rated, and induce significantly more positive facial expressions. Thus, we establish that implicit social feedback can improve the output of a deep learning model. <s> BIB003 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Generation <s> Doodling is a useful and common intelligent skill that people can learn and master. In this work, we propose a two-stage learning framework to teach a machine to doodle in a simulated painting environment via Stroke Demonstration and deep Q-learning (SDQ). The developed system, Doodle-SDQ, generates a sequence of pen actions to reproduce a reference drawing and mimics the behavior of human painters. In the first stage, it learns to draw simple strokes by imitating in supervised fashion from a set of strokeaction pairs collected from artist paintings. In the second stage, it is challenged to draw real and more complex doodles without ground truth actions; thus, it is trained with Qlearning. Our experiments confirm that (1) doodling can be learned without direct stepby- step action supervision and (2) pretraining with stroke demonstration via supervised learning is important to improve performance. We further show that Doodle-SDQ is effective at producing plausible drawings in different media types, including sketch and watercolor. <s> BIB004 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Generation <s> Deep generative models have shown great promise when it comes to synthesising novel images. While they can generate images that look convincing on a higher-level, generating fine-grained details is still a challenge. In order to foster research on more powerful generative approaches, this paper proposes a novel task: generative modelling of 2D tree skeletons. Trees are an interesting shape class because they exhibit complexity and variations that are well-suited to measure the ability of a generative model to generated detailed structures. We propose a new dataset for this task and demonstrate that state-of-the-art generative models fail to synthesise realistic images on our benchmark, even though they perform well on current datasets like MNIST digits. Motivated by these results, we propose a novel network architecture based on combining a variational autoencoder using Recurrent Neural Networks and a convolutional discriminator. The network, error metrics and training procedure are adapted to the task of fine-grained sketching. Through quantitative and perceptual experiments, we show that our model outperforms previous work and that our dataset is a valuable benchmark for generative models. We will make our dataset publicly available. <s> BIB005 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Generation <s> Sketching is more fundamental to human cognition than speech. Deep Neural Networks (DNNs) have achieved the state-of-the-art in speech-related tasks but have not made significant development in generating stroke-based sketches a.k.a sketches in vector format. Though there are Variational Auto Encoders (VAEs) for generating sketches in vector format, there is no Generative Adversarial Network (GAN) architecture for the same. In this paper, we propose a standalone GAN architecture SkeGAN and a VAE-GAN architecture VASkeGAN, for sketch generation in vector format. SkeGAN is a stochastic policy in Reinforcement Learning (RL), capable of generating both multidimensional continuous and discrete outputs. VASkeGAN hybridizes a VAE and a GAN, in order to couple the efficient representation of data by VAE with the powerful generating capabilities of a GAN, to produce visually appealing sketches. We also propose a new metric called the Ske-score which quantifies the quality of vector sketches. We have validated that SkeGAN and VASkeGAN generate visually appealing sketches by using Human Turing Test and Ske-score. <s> BIB006
Sketch generation is a booming topic in deep learning era, which has several practical application scenarios, e.g., synthesising novel pictures, assisting artist design, finishing incomplete sketches. Sketch generation can be solved in various frameworks, e.g., variational autoencoder (VAE) BIB002 , BIB005 , BIB003 , Generative Adversarial Network (GAN) BIB006 , VAE-GAN BIB006 , reinforcement learning (RL) BIB006 , BIB004 . OK The seminal model SketchRNN BIB002 is a sequence-tosequence VAE model for conditional and unconditional sketch generation in the vector format. Its encoder and decoder are implemented by bidirectional RNN BIB001 and unidirectional RNN, respectively. Specifically, as stated earlier, free-hand sketching is a dynamic process of connecting key points into lines, so that the main idea of SketchRNN is to simulate the human sketching in a point-by-point manner. This is to say that SketchRNN produces sketch strokes in a point-by-point manner, and provides both the coordinates and pen state for each point. Pen state enables machine to know when to stop a stroke. As shown in Figure 8 , the VAE encoder of SketchRNN takes vector format sketches as input, and encodes each sketch as a hidden vector h, which is the RNN hidden state of the last time step. This hidden vector will be further encoded as two parameters µ and σ to model a Gaussian distribution N (µ, σ) , where a latent vecto z will be sampled out. Then, the LSTM based VAE decoder will generate the pen coordinates and pen states of the key stroke points, conditionally on z . In particular, the pen coordinates and state for each key stroke point is sampled from a Gaussian mixture model, also as the input for the next time step for decoder. SketchRNN is able to work in conditional and unconditional modes, corresponding to two practical scenarios: (i) Conditional Reconstruction: Neural network follows the human sketching styles to imitate the inputted sketches. (ii) Unconditional Generation: Neural network draws a sketch of the given class label, without any references. In unconditional generation scenario, it only needs to train the decoder of VEA, without any input or latent vectors. The unconditional generation has various practical applications, e.g., helping human to design novel graphic patterns. To improve SketchRNN to deal with multi-class generation, Cao et al.  propose a generative model named as "AI-Sketcher", which is also a VAE based network. There are several development trends of sketch generation: (i) Fine-grained sketching BIB005 is proposed by researchers. (ii) Recently, a novel evaluation metric termed as "Ske-score" is proposed in BIB006 , which quantifies the goodness of generated vector sketches.