Deep Learning for Free-Hand Sketch: A Survey <s> Loss Function: <s> Thesupport-vector network is a new learning machine for two-group classification problems. The machine conceptually implements the following idea: input vectors are non-linearly mapped to a very high-dimension feature space. In this feature space a linear decision surface is constructed. Special properties of the decision surface ensures high generalization ability of the learning machine. The idea behind the support-vector network was previously implemented for the restricted case where the training data can be separated without errors. We here extend this result to non-separable training data.High generalization ability of support-vector networks utilizing polynomial input transformations is demonstrated. We also compare the performance of the support-vector network to various classical learning algorithms that all took part in a benchmark study of Optical Character Recognition. <s> BIB001 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Loss Function: <s> Freehand sketches are an interesting universal form of visual representation. Sketching has become easily accessible with many of the devices that we use on a daily basis. In this paper, we propose a system for real-time sketch recognition and similarity search. Our system is able to recognize partial sketches from 250 object categories. It is then able to retrieve similar sketches but also images/photographs. In this work, we propose to use deep convolutional neural networks (ConvNets) for partial sketch recognition and feature extraction. Features are extracted from sketches and image contours in order to be used as a basis for similarity search using k-Nearest Neighbors (kNN). Our system demonstrates promising results in identifying similar images, and could be integrated in larger content-based search engines. <s> BIB002 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Loss Function: <s> We propose a deep hashing framework for sketch retrieval that, for the first time, works on a multi-million scale human sketch dataset. Leveraging on this large dataset, we explore a few sketch-specific traits that were otherwise under-studied in prior literature. Instead of following the conventional sketch recognition task, we introduce the novel problem of sketch hashing retrieval which is not only more challenging, but also offers a better testbed for large-scale sketch analysis, since: (i) more fine-grained sketch feature learning is required to accommodate the large variations in style and abstraction, and (ii) a compact binary code needs to be learned at the same time to enable efficient retrieval. Key to our network design is the embedding of unique characteristics of human sketch, where (i) a two-branch CNN-RNN architecture is adapted to explore the temporal ordering of strokes, and (ii) a novel hashing loss is specifically designed to accommodate both the temporal and abstract traits of sketches. By working with a 3.8M sketch dataset, we show that state-of-the-art hashing models specifically engineered for static images fail to perform well on temporal sketch data. Our network on the other hand not only offers the best retrieval performance on various code sizes, but also yields the best generalization performance under a zero-shot setting and when re-purposed for sketch recognition. Such superior performances effectively demonstrate the benefit of our sketch-specific design. <s> BIB003 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Loss Function: <s> In this paper, we address the problem of hand-drawn sketch recognition. Inspired by the Bayesian decision theory, we present a deep metric learning loss with the objective to minimize the Bayesian risk of misclassification. We estimate this risk for every mini-batch during training, and learn robust deep embeddings by backpropagating it to a deep neural network in an end-to-end trainable paradigm. Our learnt embeddings are discriminative and robust despite of intra-class variations and inter-class similarities naturally present in hand-drawn sketch images. Outperforming the state of the art on sketch recognition, our method achieves 82.2% and 88.7% on TU-Berlin-250 and TU-Berlin-160 benchmarks respectively. <s> BIB004 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Loss Function: <s> We introduce a novel problem of scene sketch zero-shot learning (SSZSL), which is a challenging task, since (i) different from photo, the gap between common semantic domain (e.g., word vector) and sketch is too huge to exploit common semantic knowledge as the bridge for knowledge transfer, and (ii) compared with single-object sketch, more expressive feature representation for scene sketch is required to accommodate its high-level of abstraction and complexity. To overcome these challenges, we propose a deep embedding model for scene sketch zero-shot learning. In particular, we propose the augmented semantic vector to conduct domain alignment by fusing multi-modal semantic knowledge (e.g., cartoon image, natural image, text description), and adopt attention-based network for scene sketch feature learning. Moreover, we propose a novel distance metric to improve the similarity measure during testing. Extensive experiments and ablation studies demonstrate the benefit of our sketch-specific design. <s> BIB005 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Loss Function: <s> Hand-drawn sketch recognition is a fundamental problem in computer vision, widely used in sketch-based image and video retrieval, editing, and reorganization. Previous methods often assume that a complete sketch is used as input; however, hand-drawn sketches in common application scenarios are often incomplete, which makes sketch recognition a challenging problem. In this paper, we propose SketchGAN, a new generative adversarial network (GAN) based approach that jointly completes and recognizes a sketch, boosting the performance of both tasks. Specifically, we use a cascade Encode-Decoder network to complete the input sketch in an iterative manner, and employ an auxiliary sketch recognition task to recognize the completed sketch. Experiments on the Sketchy database benchmark demonstrate that our joint learning approach achieves competitive sketch completion and recognition performance compared with the state-of-the-art methods. Further experiments using several sketch-based applications also validate the performance of our method. <s> BIB006
Most of the previous deep sketch recognition methods use cross-entropy softmax loss to train the deep neural networks. However, as stated in Section 2.1, free-hand sketch has its intrinsic traits and domain-unique challenges, so that it's better to study the sketch-specific loss function to drive the recognition models to learn more fine-grained sketch patterns. Xu et al. proposed the sketch-specific center loss BIB003 for million-scale sketches, based on the staged-training strategy. The basis is that the image entropy distribution of each sketch category is a truncated Gaussian distribution. Inspired by the classical Bayesian decision theory  and Bayesian risk BIB004 , Mishra et al. proposed a deep metric loss to drive the pretained deep neural network to minimize the Bayesian risk R(F Θ ) of misclassifying the sketch pairs that were randomly selected within each mini-batch. Assume F Θ (X i ) and F Θ (X j ) are the normalized feature embeddings of the random selected sketches X i and X j . c + and c − denote all the possible positive and negative class pairs that could be predicted, respectively. where D(·, ·) (∈ [−1, 1]) denotes cosine distance function. This risk is the probability integral of classifying positive sketches as negative and negative sketches as positive. Based on this Bayesian risk loss, sketch recognition needs a two-stage training. After obtained the features, it needs to train a linear SVM BIB001 as the classifier. Comments: The practical human-computer interaction motivates a new trend of sketch online recognition. Moreover, the previous deep learning models for sketch recognition work in the supervised manner, leaving the semisupervised and unsupervised method under-studied. Considering from the practical application scenarios, zeroshot BIB005 and few-shot sketch recognition methods are also worth studying in the future. Another interesting topic that proposed by recent works BIB002 , BIB006 is using multi-task learning framework to simultaneously solve sketch recognition with other tasks.