Deep Learning for Free-Hand Sketch: A Survey <s> Sketch Data Augmentation: <s> Humans have used sketching to depict our visual world since prehistoric times. Even today, sketching is possibly the only rendering technique readily available to all humans. This paper is the first large scale exploration of human sketches. We analyze the distribution of non-expert sketches of everyday objects such as 'teapot' or 'car'. We ask humans to sketch objects of a given category and gather 20,000 unique sketches evenly distributed over 250 object categories. With this dataset we perform a perceptual study and find that humans can correctly identify the object category of a sketch 73% of the time. We compare human performance against computational recognition methods. We develop a bag-of-features sketch representation and use multi-class support vector machines, trained on our sketch dataset, to classify sketches. The resulting recognition method is able to identify unknown sketches with 56% accuracy (chance is 0.4%). Based on the computational model, we demonstrate an interactive sketch recognition system. We release the complete crowd-sourced dataset of sketches to the community. <s> BIB001 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch Data Augmentation: <s> Convolutional neural networks (CNNs) perform well on problems such as handwriting recognition and image classification. However, the performance of the networks is often limited by budget and time constraints, particularly when trying to train deep networks. ::: Motivated by the problem of online handwriting recognition, we developed a CNN for processing spatially-sparse inputs; a character drawn with a one-pixel wide pen on a high resolution grid looks like a sparse matrix. Taking advantage of the sparsity allowed us more efficiently to train and test large, deep CNNs. On the CASIA-OLHWDB1.1 dataset containing 3755 character classes we get a test error of 3.82%. ::: Although pictures are not sparse, they can be thought of as sparse by adding padding. Applying a deep convolutional network using sparsity has resulted in a substantial reduction in test error on the CIFAR small picture datasets: 6.28% on CIFAR-10 and 24.30% for CIFAR-100. <s> BIB002 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch Data Augmentation: <s> We propose a multi-scale multi-channel deep neural network framework that, for the first time, yields sketch recognition performance surpassing that of humans. Our superior performance is a result of explicitly embedding the unique characteristics of sketches in our model: (i) a network architecture designed for sketch rather than natural photo statistics, (ii) a multi-channel generalisation that encodes sequential ordering in the sketching process, and (iii) a multi-scale network ensemble with joint Bayesian fusion that accounts for the different levels of abstraction exhibited in free-hand sketches. We show that state-of-the-art deep networks specifically engineered for photos of natural objects fail to perform well on sketch recognition, regardless whether they are trained using photo or sketch. Our network on the other hand not only delivers the best performance on the largest human sketch dataset to date, but also is small in size making efficient training possible using just CPUs. <s> BIB003 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch Data Augmentation: <s> Freehand sketches often contain sparse visual detail. In spite of the sparsity, they are easily and consistently recognized by humans across cultures, languages and age groups. Therefore, analyzing such sparse sketches can aid our understanding of the neuro-cognitive processes involved in visual representation and recognition. In the recent past, Convolutional Neural Networks (CNNs) have emerged as a powerful framework for feature representation and recognition for a variety of image domains. However, the domain of sketch images has not been explored. This paper introduces a freehand sketch recognition framework based on "deep" features extracted from CNNs. We use two popular CNNs for our experiments -- Imagenet CNN and a modified version of LeNet CNN. We evaluate our recognition framework on a publicly available benchmark database containing thousands of freehand sketches depicting everyday objects. Our results are an improvement over the existing state-of-the-art accuracies by 3% - 11%. The effectiveness and relative compactness of our deep features also make them an ideal candidate for related problems such as sketch-based image retrieval. In addition, we provide a preliminary glimpse of how such features can help identify crucial attributes (e.g. object-parts) of the sketched objects. <s> BIB004 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch Data Augmentation: <s> Sketch recognition remains a significant challenge due to the limited training data and the substantial intra-class variance of freehand sketches for the same object. Conventional methods for this task often rely on the availability of the temporal order of sketch strokes, additional cues acquired from different modalities and supervised augmentation of sketch datasets with real images, which also limit the applicability and feasibility of these methods in real scenarios. ::: In this paper, we propose a novel sketch-specific data augmentation (SSDA) method that leverages the quantity and quality of the sketches automatically. From the aspect of quantity, we introduce a Bezier pivot based deformation (BPD) strategy to enrich the training data. Towards quality improvement, we present a mean stroke reconstruction (MSR) approach to generate a set of novel types of sketches with smaller intra-class variances. Both of these solutions are unrestricted from any multi-source data and temporal cues of sketches. Furthermore, we show that some recent deep convolutional neural network models that are trained on generic classes of real images can be better choices than most of the elaborate architectures that are designed explicitly for sketch recognition. As SSDA can be integrated with any convolutional neural networks, it has a distinct advantage over the existing methods. Our extensive experimental evaluations demonstrate that the proposed method achieves state-of-the-art results (84.27%) on the TU-Berlin dataset, outperforming the human performance by a remarkable 11.17% increase. We also present a new benchmark named Sketchy-R to facilitate future research in sketch recognition. Finally, more experiments show the practical value of our approach to the task of sketch-based image retrieval. <s> BIB005 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch Data Augmentation: <s> Sketch-based image synthesis aims to generate a photo image given a sketch. It is a challenging task; because sketches are drawn by non-professionals and only consist of strokes, they usually exhibit shape deformation and lack visual cues, i.e., colors and textures. Thus translation from sketch to photo involves two aspects: shape and color (texture). Existing methods cannot handle this task well, as they mostly focus on solving one translation. In this work, we show that the key to this task lies in decomposing the translation into two sub-tasks, shape translation and colorization. Correspondingly, we propose a model consisting of two sub-networks, with each one tackling one sub-task. We also find that, when translating shapes, specific drawing styles affect the generated results significantly and may even lead to failure. To make our model more robust to drawing style variations, we design a data augmentation strategy and re-purpose an attention module, aiming to make our model pay less attention to distracted regions of a sketch. Besides, a conditional module is adapted for color translation to improve diversity and increase users' control over the generated results. Both quantitative and qualitative comparisons are presented to show the superiority of our approach. In addition, as a side benefit, our model can synthesize high-quality sketches from photos inversely. We also demonstrate how these generated photos and sketches can benefit other applications, such as sketch-based image retrieval. <s> BIB006
Please note that all the sketch-specific data augmentation methods discussed in this subsection can be applied to both sketch recognition and all the other sketch involved tasks, e.g., sketch-based image retrieval, sketch-related generation. (i) Sketch can be formatted in visual domain as picture, so that most of the common data augmentations designed for natural photos (e.g., ImageNet [84] photos) can be applied to sketch, e.g., horizontal reflection/mirroring, rotation, horizontal shift, vertical shift, central zoom. These augmentations have already been evaluated by the early sketch-oriented deep learning works BIB003 , BIB004 . However, the random cropping is unsuitable to sketch due to that partial sketches are too sparse and abstract to recognize even for humans. (ii) Stroke thickening/dilation is another kind of augmentation for free-hand sketch. As discussed in some pre-vious works on the spatially-sparse convolutional neural networks BIB002 , the subtle details of sparse strokes of sketches will be lost during the layer-by-layer convolutions. Thus, the stroke thickening/dilation is useful for the neural networks to learn the details of sketches. (iii) Yu et al.  propose to remove the strokes to obtain more diverse sketches, which rely on the temporal order information of human sketching. Note that this stroke remove is not a random remove. Its main idea comes from a general rule BIB001 that during sketching human like draw main outline first and then draw details. Thus, Yu et al. progressively remove detail from each training sketch to generate new sketches at multiple abstraction levels. (iv) Zheng et al. BIB005 propose a Bezier pivot based deformation (BPD) strategy and a mean stroke reconstruction (MSR) approach. Both of these two augmentations do not need any temporal cues of sketches. The main idea of MSR is to generate novel sketches with smaller intra-class variance. (v) Some sketch-related generation works BIB006 also propose some sketch data augmentation strategies. Liu et al. propose two sketch-specific data augmentation strategies: (a) Manually extract some strokes from sketch SVG files to construct the noise stroke masks. Then, randomly apply the noise stroke masks to the original sketches to synthesize augmented sketches. (b) Randomly extract a patch from another sketch, and attach it upon a given sketch. Comments: Compared with augmentations on the full picture basis (e.g., rotation, shift), making full use of the stroke information to design sketch-specific data augmentations is a significant progress. This benefits from that researchers gradually analyse sketches from global to local details. However, all the aforementioned stroke modification based augmentations are merely random perturbations, without the considering how to mimic the real drawing styles of humans.