Deep Learning for Free-Hand Sketch: A Survey <s> Recognition <s> Humans have used sketching to depict our visual world since prehistoric times. Even today, sketching is possibly the only rendering technique readily available to all humans. This paper is the first large scale exploration of human sketches. We analyze the distribution of non-expert sketches of everyday objects such as 'teapot' or 'car'. We ask humans to sketch objects of a given category and gather 20,000 unique sketches evenly distributed over 250 object categories. With this dataset we perform a perceptual study and find that humans can correctly identify the object category of a sketch 73% of the time. We compare human performance against computational recognition methods. We develop a bag-of-features sketch representation and use multi-class support vector machines, trained on our sketch dataset, to classify sketches. The resulting recognition method is able to identify unknown sketches with 56% accuracy (chance is 0.4%). Based on the computational model, we demonstrate an interactive sketch recognition system. We release the complete crowd-sourced dataset of sketches to the community. <s> BIB001 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Recognition <s> We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overriding in the fully-connected layers we employed a recently-developed regularization method called "dropout" that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry. <s> BIB002 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Recognition <s> In this paper, we revisit the classical Bayesian face recognition method by Baback Moghaddam et al. and propose a new joint formulation. The classical Bayesian method models the appearance difference between two faces. We observe that this "difference" formulation may reduce the separability between classes. Instead, we model two faces jointly with an appropriate prior on the face representation. Our joint formulation leads to an EM-like model learning at the training time and an efficient, closed-formed computation at the test time. On extensive experimental evaluations, our method is superior to the classical Bayesian face and many other supervised approaches. Our method achieved 92.4% test accuracy on the challenging Labeled Face in Wild (LFW) dataset. Comparing with current best commercial system, we reduced the error rate by 10%. <s> BIB003 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Recognition <s> In this paper, we propose a novel neural network model called RNN Encoder‐ Decoder that consists of two recurrent neural networks (RNN). One RNN encodes a sequence of symbols into a fixedlength vector representation, and the other decodes the representation into another sequence of symbols. The encoder and decoder of the proposed model are jointly trained to maximize the conditional probability of a target sequence given a source sequence. The performance of a statistical machine translation system is empirically found to improve by using the conditional probabilities of phrase pairs computed by the RNN Encoder‐Decoder as an additional feature in the existing log-linear model. Qualitatively, we show that the proposed model learns a semantically and syntactically meaningful representation of linguistic phrases. <s> BIB004 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Recognition <s> In this paper, we present a system for sketch classification and similarity search. We used deep convolution neural networks (ConvNets), state of the art in the field of image recognition. They enable both classification and medium/highlevel features extraction. We make use of ConvNets features as a basis for similarity search using k-Nearest Neighbors (kNN). Evaluation are performed on the TU-Berlin benchmark. Our main contributions are threefold: first, we use ConvNets in contrast to most previous approaches based essentially on hand crafted features. Secondly, we propose a ConvNet that is both more accurate and lighter/faster than the two only previous attempts at making use of ConvNets for handsketch recognition. We reached an accuracy of 75.42%. Third, we shown that similarly to their application on natural images, ConvNets allow the extraction of medium-level and high-level features (depending on the depth) which can be used for similarity search.1 <s> BIB005 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Recognition <s> We propose a multi-scale multi-channel deep neural network framework that, for the first time, yields sketch recognition performance surpassing that of humans. Our superior performance is a result of explicitly embedding the unique characteristics of sketches in our model: (i) a network architecture designed for sketch rather than natural photo statistics, (ii) a multi-channel generalisation that encodes sequential ordering in the sketching process, and (iii) a multi-scale network ensemble with joint Bayesian fusion that accounts for the different levels of abstraction exhibited in free-hand sketches. We show that state-of-the-art deep networks specifically engineered for photos of natural objects fail to perform well on sketch recognition, regardless whether they are trained using photo or sketch. Our network on the other hand not only delivers the best performance on the largest human sketch dataset to date, but also is small in size making efficient training possible using just CPUs. <s> BIB006 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Recognition <s> Human's sketch understanding is important. It has many applications in human computer interaction, multimedia, and computer vision. Recognizing human sketches is also challenging. Previous methods focus on single-object sketch recognition. Understanding human's scene sketch that involves multiple objects and their complex interactions has not been explored. In this paper, we tackle this new problem. We create the first scene sketch dataset "Scene250" and propose a deep learning method to understand human scene sketches. We propose "Scene-Net", a new deep convolutional neural network (CNN) structure, based on which we build a novel scene sketch recognition system. Our system has been tested on the collected scene sketch dataset and compared with other state-of-the-art CNNs and sketch recognition approaches. Our experimental results demonstrate that our method achieves the state of art. <s> BIB007 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Recognition <s> In this study, we present a weakly supervised approach that discovers the discriminative structures of sketch images, given pairs of sketch images and web images. In contrast to traditional approaches that use global appearance features or relay on keypoint features, our aim is to automatically learn the shared latent structures that exist between sketch images and real images, even when there are significant appearance differences across its relevant real images. To accomplish this, we propose a deep convolutional neural network, named SketchNet. We firstly develop a triplet composed of sketch, positive and negative real image as the input of our neural network. To discover the coherent visual structures between the sketch and its positive pairs, we introduce the softmax as the loss function. Then a ranking mechanism is introduced to make the positive pairs obtain a higher score comparing over negative ones to achieve robust representation. Finally, we formalize above-mentioned constrains into the unified objective function, and create an ensemble feature representation to describe the sketch images. Experiments on the TUBerlin sketch benchmark demonstrate the effectiveness of our model and show that deep feature representation brings substantial improvements over other state-of-the-art methods on sketch classification. <s> BIB008 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Recognition <s> In this paper, we study oracle character recognition and general sketch recognition. First, a data set of oracle characters, which are the oldest hieroglyphs in China yet remain a part of modern Chinese characters, is collected for analysis. Second, typical visual representations in shape- and sketch-related works are evaluated. We analyze the problems suffered when addressing these representations and determine several representation design criteria. Based on the analysis, we propose a novel hierarchical representation that combines a Gabor-related low-level representation and a sparse-encoder-related mid-level representation. Extensive experiments show the effectiveness of the proposed representation in both oracle character recognition and general sketch recognition. The proposed representation is also complementary to convolutional neural network (CNN)-based models. We introduce a solution to combine the proposed representation with CNN-based models, and achieve better performances over both approaches. This solution has beaten humans at recognizing general sketches. <s> BIB009 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Recognition <s> This work provides a study on how Convolutional Neural Networks, trained to identify objects primarily in photos, perform when applied to more abstract representations of the same objects. Our main goal is to better understand the generalization abilities of these networks and their learned inner representations. We show that both GoogLeNet and AlexNet networks are largely unable to recognize abstract sketches that are easily recognizable by humans. Moreover, we show that the measured efficacy vary considerably across different classes and we discuss possible reasons for this. <s> BIB010 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Recognition <s> Freehand sketches are a simple and powerful tool for communication. They are easily recognized across cultures and suitable for various applications. In this paper, we propose a new approach for partial sketch recognition. This could be used to design applications using real-time sketch recognition. We use deep convolutional neural networks (ConvNets), state-of-the-art in the field of sketch recognition. To the best of our knowledge, it is the first ConvNet for partial sketch classification. Our first aim is to build a ConvNet capable of recognizing partial sketches without compromising the accuracy reached for complete sketch recognition. Therefore, we evaluate different approaches and propose an efficient way for partial sketch recognition. Our second aim is improving complete sketch recognition using information about sketching progression. We obtained a ConvNet that outperforms state-of-the-art results in the TU-Berlin sketch benchmark. We reached an accuracy of 77.69%.1 <s> BIB011 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Recognition <s> Freehand sketches are an interesting universal form of visual representation. Sketching has become easily accessible with many of the devices that we use on a daily basis. In this paper, we propose a system for real-time sketch recognition and similarity search. Our system is able to recognize partial sketches from 250 object categories. It is then able to retrieve similar sketches but also images/photographs. In this work, we propose to use deep convolutional neural networks (ConvNets) for partial sketch recognition and feature extraction. Features are extracted from sketches and image contours in order to be used as a basis for similarity search using k-Nearest Neighbors (kNN). Our system demonstrates promising results in identifying similar images, and could be integrated in larger content-based search engines. <s> BIB012 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Recognition <s> Freehand sketching is an inherently sequential process. Yet, most approaches for hand-drawn sketch recognition either ignore this sequential aspect or exploit it in an ad-hoc manner. In our work, we propose a recurrent neural network architecture for sketch object recognition which exploits the long-term sequential and structural regularities in stroke data in a scalable manner. Specifically, we introduce a Gated Recurrent Unit based framework which leverages deep sketch features and weighted per-timestep loss to achieve state-of-the-art results on a large database of freehand object sketches across a large number of object categories. The inherently online nature of our framework is especially suited for on-the-fly recognition of objects as they are being drawn. Thus, our framework can enable interesting applications such as camera-equipped robots playing the popular party game Pictionary with human players and generating sparsified yet recognizable sketches of objects. <s> BIB013 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Recognition <s> Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation. <s> BIB014 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Recognition <s> Modeling the distribution of natural images is a landmark problem in unsupervised learning. This task requires an image model that is at once expressive, tractable and scalable. We present a deep neural network that sequentially predicts the pixels in an image along the two spatial dimensions. Our method models the discrete probability of the raw pixel values and encodes the complete set of dependencies in the image. Architectural novelties include fast two-dimensional recurrent layers and an effective use of residual connections in deep recurrent networks. We achieve log-likelihood scores on natural images that are considerably better than the previous state of the art. Our main results also provide benchmarks on the diverse ImageNet dataset. Samples generated from the model appear crisp, varied and globally coherent. <s> BIB015 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Recognition <s> We present DrawFromDrawings , an interactive drawing system that provides users with visual feedback for assistance in 2D drawing using a database of sketch images. Following the traditional imitation and emulation training from art education, DrawFromDrawings enables users to retrieve and refer to a sketch image stored in a database and provides them with various novel strokes as suggestive or deformation feedback. Given regions of interest (ROIs) in the user and reference sketches, DrawFromDrawings detects as-long-as-possible (ALAP) stroke segments and the correspondences between user and reference sketches that are the key to computing seamless interpolations. The stroke-level interpolations are parametrized with the user strokes, the reference strokes, and new strokes created by warping the reference strokes based on the user and reference ROI shapes, and the user study indicated that the interpolation could produce various reasonable strokes varying in shapes and complexity. DrawFromDrawings allows users to either replace their strokes with interpolated strokes (deformation feedback) or overlays interpolated strokes onto their strokes (suggestive feedback). The other user studies on the feedback modes indicated that the suggestive feedback enabled drawers to develop and render their ideas using their own stroke style, whereas the deformation feedback enabled them to finish the sketch composition quickly. <s> BIB016 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Recognition <s> We present sketch-rnn, a recurrent neural network (RNN) able to construct stroke-based drawings of common objects. The model is trained on thousands of crude human-drawn images representing hundreds of classes. We outline a framework for conditional and unconditional sketch generation, and describe new robust training methods for generating coherent sketch drawings in a vector format. <s> BIB017 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Recognition <s> Human sketches are unique in being able to capture both the spatial topology of a visual object, as well as its subtle appearance details. Fine-grained sketch-based image retrieval (FG-SBIR) importantly leverages on such fine-grained characteristics of sketches to conduct instance-level retrieval of photos. Nevertheless, human sketches are often highly abstract and iconic, resulting in severe misalignments with candidate photos which in turn make subtle visual detail matching difficult. Existing FG-SBIR approaches focus only on coarse holistic matching via deep cross-domain representation learning, yet ignore explicitly accounting for fine-grained details and their spatial context. In this paper, a novel deep FG-SBIR model is proposed which differs significantly from the existing models in that: (1) It is spatially aware, achieved by introducing an attention module that is sensitive to the spatial position of visual details: (2) It combines coarse and fine semantic information via a shortcut connection fusion block: and (3) It models feature correlation and is robust to misalignments between the extracted features across the two domains by introducing a novel higher-order learnable energy function (HOLEF) based loss. Extensive experiments show that the proposed deep spatial-semantic attention model significantly outperforms the state-of-the-art. <s> BIB018 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Recognition <s> In this paper, a deep end-to-end network for sketch recognition, named Deep Visual-Sequential Fusion model (DVSF) is proposed to model the visual and sequential patterns of the strokes. To capture the intermediate states of sketches, a three-way representation learner is first utilized to extract the visual features. These deep features are simultaneously fed into the visual and sequential networks to capture spatial and temporal properties, respectively. More specifically, visual networks are novelly proposed to learn the stroke patterns by stacking the Residual Fully-Connected (R-FC) layers, which integrate ReLU and Tanh activation functions to achieve the sparsity and generalization ability. To learn the patterns of stroke order, sequential networks are constructed by Residual Long Short-Term Memory (R-LSTM) units, which optimize the network architecture by skip connection. Finally, the visual and sequential representations of the sketches are seamlessly integrated with a fusion layer to obtain the final results. Experiments conducted on the benchmark sketch dataset TU-Berlin demonstrate the effectiveness of the proposed method, which outperforms the state-of-the-art approaches. <s> BIB019 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Recognition <s> We propose a deep hashing framework for sketch retrieval that, for the first time, works on a multi-million scale human sketch dataset. Leveraging on this large dataset, we explore a few sketch-specific traits that were otherwise under-studied in prior literature. Instead of following the conventional sketch recognition task, we introduce the novel problem of sketch hashing retrieval which is not only more challenging, but also offers a better testbed for large-scale sketch analysis, since: (i) more fine-grained sketch feature learning is required to accommodate the large variations in style and abstraction, and (ii) a compact binary code needs to be learned at the same time to enable efficient retrieval. Key to our network design is the embedding of unique characteristics of human sketch, where (i) a two-branch CNN-RNN architecture is adapted to explore the temporal ordering of strokes, and (ii) a novel hashing loss is specifically designed to accommodate both the temporal and abstract traits of sketches. By working with a 3.8M sketch dataset, we show that state-of-the-art hashing models specifically engineered for static images fail to perform well on temporal sketch data. Our network on the other hand not only offers the best retrieval performance on various code sizes, but also yields the best generalization performance under a zero-shot setting and when re-purposed for sketch recognition. Such superior performances effectively demonstrate the benefit of our sketch-specific design. <s> BIB020 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Recognition <s> We introduce a novel problem of scene sketch zero-shot learning (SSZSL), which is a challenging task, since (i) different from photo, the gap between common semantic domain (e.g., word vector) and sketch is too huge to exploit common semantic knowledge as the bridge for knowledge transfer, and (ii) compared with single-object sketch, more expressive feature representation for scene sketch is required to accommodate its high-level of abstraction and complexity. To overcome these challenges, we propose a deep embedding model for scene sketch zero-shot learning. In particular, we propose the augmented semantic vector to conduct domain alignment by fusing multi-modal semantic knowledge (e.g., cartoon image, natural image, text description), and adopt attention-based network for scene sketch feature learning. Moreover, we propose a novel distance metric to improve the similarity measure during testing. Extensive experiments and ablation studies demonstrate the benefit of our sketch-specific design. <s> BIB021 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Recognition <s> Text-based retrieval systems have been popular, but content-based retrieval systems have gained widespread acceptance in recent years to directly retrieve diverse media based on their visual content, such as color, texture, and shape. Among many content-based retrieval systems, sketch-based media retrieval systems have attracted attention recently with the proliferation of tablet PCs and smart mobile devices. Sketch-based retrieval requires the user to draw a freehand sketch query, but freehand drawing can be challenging for those with limited drawing skills. This degrades retrieval performance, since successful retrieval depends on the quality of the sketch query image drawn by the user. To address this issue, we propose a real-time stroke guidance for freehand sketch retrieval that continuously displays next-stroke shadow sketches on the canvas based on the user's step-by-step partial strokes. We train a stroke guidance network that learns the mapping between the step-wise stroke relations to predict the user's next stroke. The proposed stroke guidance for freehand sketch retrieval system runs on a five step next-stroke prediction model that identifies candidate next-stroke sketches from a database of millions of sketches. The system retrieves variable number of sketch object classes at different drawing stages. During the initial sketching stage, diverse drawing possibilities are covered by retrieving multiple sketch classes; as the sketching progresses, the intended sketch class is narrowed down to one. Deep binary hashing is employed for efficient similarity matching of relevant next-stroke sketches. We extend the Google QuickDraw dataset to create a five step sketch stroke database. Qualitative and quantitative experiments are conducted to verify the effectiveness of the proposed system, which can be utilized for drawing guidance, tracing, and sketch retrieval. Tracing refers to the act of copying the shadowed line of a guiding image by drawing over its lines. <s> BIB022 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Recognition <s> State-of-the-art methods on sketch classification and retrieval are based on deep convolutional neural network to learn representations. Although deep neural networks have the ability to model images with hierarchical representations by convolution kernels, they cannot automatically extract the structural representations of object categories in a human-perceptible way. Furthermore, sketch images usually have large-scale visual variations caused by the styles of drawing or viewpoints, which make it difficult to develop generalized representations using the fixed computational mode of convolutional kernel. In this paper, our aim is to address the problem of fixed computational mode in feature extraction process without extra supervision. We propose a novel architecture to dynamically discover the object landmarks and learn the discriminative structural representations. Our model is composed of two components: a representative landmark discovering module that localizes the key points on the object and a category-aware representation learning module that develops the category-specific features. Specifically, we develop a structure-aware offset layer to dynamically localize the representative landmarks, which is optimized based on the category labels without extra supervision. After that, a diversity branch is introduced to extract the global discriminative features for each category. Finally, we employ a multi-task loss function to develop an end-to-end trainable architecture. At testing time, we fuse all the predictions with different number of landmarks to achieve the final results. Through extensive experiments, we compare our model with several state-of-the-art methods on two challenging datasets, TU-Berlin and Sketchy, for sketch classification and retrieval, and the experimental results demonstrate the effectiveness of our proposed model. <s> BIB023 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Recognition <s> We study the problem of sketch image recognition. This problem is plagued with two major challenges: 1) sketch images are often scarce in contrast to the abundance of natural images, rendering the training task difficult, and 2) the significant domain gap between sketch image and its natural image counterpart makes the task of bridging the two domains challenging. In order to overcome these challenges, in this paper we propose to transfer the knowledge of a network learned from natural images to a sketch network - a new deep net architecture which we term as cousin network. This network guides a sketch-recognition network to extract more relevant features that are close to those of natural images, via adversarial training. Moreover, to enhance the transfer ability of the classification model, a sketch-to-image attribute warehouse is constructed to approximate the transformation between the sketch domain and the real image domain. Extensive experiments conducted on the TU-Berlin dataset show that the proposed model is able to efficiently distill knowledge from natural images and achieves superior performance than the current state of the art. <s> BIB024
Sketch recognition BIB001 is to predict the class label for the given sketch, which is one of the most fundamental tasks in free-hand sketch community, with a lot of practical applications, e.g., interactive drawing system that provides some feedback to users BIB016 , sketch-based science education system  . Both of object sketches BIB001 , BIB017 and scene sketches BIB007 , BIB021 have been studied to date. The data augmentations, network architectures, loss functions that are engineered for sketch recognition can be used for other sketch tasks. Moreover, the deep learning based sketch recognition techniques have been already widely used in some sketch-based human-computer interaction applications, e.g., online drawing game QuickDraw, WeChat miniapp Caihua Xiaoge that was released by Google. Sketch recognition can be categorised as (i) offline recognition and (ii) online recognition. Offline recognition systems take the whole sketch as input and predict a class label, after the sketching was done. Online recognition systems take the accumulated sketch strokes and continuously predict the class label, during sketching. Most of the existing deep learning based sketch recognition works are offline recognition models. Online sketch recognition can be used for real-time drawing guidance BIB022 , tracing, and sketch retrieval, providing better interactive experience. According to the sketching content, sketch recognition can also be categorised as object recognition and scene recognition/understanding. Comments: With the development of deep learning, the understanding of researchers on sketch is gradually deepening, the main progressive trends including: (i) from CNN-based static representation to RNN-based dynamic representation, (ii) from explicit picture space to implicit sequence space; (iii) from global representation to local understanding (i.e., semantic spatial attention BIB018 ), from Euclidean space (CNN, RNN based) to geometric space (GNN based), (iv) from conventional supervised training to adversarial training. Benefit from the booming progress of deep learning, various deep models BIB008 , BIB005 ,  , BIB009 , BIB010 , BIB011 , BIB023 , BIB012 , BIB024 are proposed for free-hand sketch recognition. In the following, the deep learning based sketch recognition works will be introduced from three perspectives, i.e., (i) network architectures, (ii) data augmentations, (iii) loss functions. Network Architecture: Figure 7 demonstrates the evolution of the existing deep learning-based sketch representations. Moreover, Table 3 summarized the existing network architectures that are engineered for free-hand sketches and capable of sketch recognition. The following will introduce some representative architectures. (i) Sketch-a-Net BIB006 ,  is the first deep CNN designed for free-hand sketch. Compared with classic photo-oriented CNN architecture BIB002 , the sketch-specific aspects of its architecture mainly include: (a) Considered the highly sparsity of sketch, larger size of 15 × 15 is used for the first layer filters. This is because sketches generally lack texture information. Larger convolutional filters are more likely to capture more patterns from the structured/sparse context. (b) Local response normalization (LRN) BIB002 layers are removed for faster learning without sacrificing performance, since LRN works for "pixel brightness normalization". The brightness of sketches is 0 or 1. Moreover, Yu et al. also proposed two novel sketch-specific data augmentation strategies to synthesises new sketches to enable Sketch-a-Net to learn the stroke ordering information. Given an ensemble of Sketch-a-Net trained by varying the mini-batch orders, joint Bayesian fusion BIB003 is adopted to improve the recognition performance. (ii) Sarvadevabhatla et al. BIB013 also propose a sketch recognition network to explore the sequential process of sketching, where each training sketch needs to be plotted as a continuous sequence of cumulative stroke pictures and the corresponding AlexNet BIB002 based deep features will be sent into Gated Recurrent Unit (GRU) BIB004 network in sequence. This network is able to work in online recognition mode, since it involves the intermediate status of the sketch. (iii) Similarly, He et al. BIB019 proposed the Deep Visual-Sequential Fusion (DVSF) net to capture spatial and temporal patterns of sketches simultaneously. For each training sketches, its three accumulation sub-pictures (with 60%, 80%, 100% of strokes) go through three-way CNNs (ResNet-18 BIB014 ) to produce deep features, which are parallelly fed into the visual and sequential networks. In particular, the visual and sequential networks are implemented by residual fully-connected (R-FC) layer and Residual Long Short Term Memory (R-LSTM) BIB015 layer, respectively. The visual and sequential representations are integrated into the fusion layer to conduct sketch recognition. (iv) In 2017, Ha and Eck proposed a groundbreaking work, i.e., SketchRNN BIB017 , which contributes a novel paradigm to model the sketching ordering patterns. SketchRNN brings significant insights to the sketch repre- sent learning, although it is a Variational Inference (VI)  based sketch generation model. Distinctively different to the stroke accumulated sub-picture based RNN inputs, RNN backbone of SketchRNN is directly fed into the key point coordinates of sketching stroke. In particular, as illustrated in Figure 2 , each key point is denoted as a vector consisting of two coordinate bits (i.e., horizontal and vertical coordinates) and the corresponding flag bits. Flag bits of the key stroke points depict the pen states during sketching, telling when to stop a stroke. Specifically, within each stroke, its ending point will have a distinctive flag bit representation to those of other key points. (v) Xu et al. proposed a sketch hashing network termed as SketchMate BIB020 , where the backbone is a CNN-RNN dual branch architecture, utilizing CNN to extract abstract visual concepts and RNN to model human sketching temporal orderings. The CNN branch takes in the raster pixel sketch pictures; and the RNN branch takes in the vector sketch (i.e., 