Deep Learning for Free-Hand Sketch: A Survey <s> TABLE 3 <s> Freehand sketching is an inherently sequential process. Yet, most approaches for hand-drawn sketch recognition either ignore this sequential aspect or exploit it in an ad-hoc manner. In our work, we propose a recurrent neural network architecture for sketch object recognition which exploits the long-term sequential and structural regularities in stroke data in a scalable manner. Specifically, we introduce a Gated Recurrent Unit based framework which leverages deep sketch features and weighted per-timestep loss to achieve state-of-the-art results on a large database of freehand object sketches across a large number of object categories. The inherently online nature of our framework is especially suited for on-the-fly recognition of objects as they are being drawn. Thus, our framework can enable interesting applications such as camera-equipped robots playing the popular party game Pictionary with human players and generating sparsified yet recognizable sketches of objects. <s> BIB001 </s> Deep Learning for Free-Hand Sketch: A Survey <s> TABLE 3 <s> Freehand sketching is a dynamic process where points are sequentially sampled and grouped as strokes for sketch acquisition on electronic devices. To recognize a sketched object, most existing methods discard such important temporal ordering and grouping information from human and simply rasterize sketches into binary images for classification. In this paper, we propose a novel single-branch attentive network architecture RNN-Rasterization-CNN (Sketch-R2CNN for short) to fully leverage the dynamics in sketches for recognition. Sketch-R2CNN takes as input only a vector sketch with grouped sequences of points, and uses an RNN for stroke attention estimation in the vector space and a CNN for 2D feature extraction in the pixel space respectively. To bridge the gap between these two spaces in neural networks, we propose a neural line rasterization module to convert the vector sketch along with the attention estimated by RNN into a bitmap image, which is subsequently consumed by CNN. The neural line rasterization module is designed in a differentiable way to yield a unified pipeline for end-to-end learning. We perform experiments on existing large-scale sketch recognition benchmarks and show that by exploiting the sketch dynamics with the attention mechanism, our method is more robust and achieves better performance than the state-of-the-art methods. <s> BIB002 </s> Deep Learning for Free-Hand Sketch: A Survey <s> TABLE 3 <s> The sketch segmentation problem remains largely unsolved because conventional methods are greatly challenged by the highly abstract appearances of freehand sketches and their numerous shape variations. In this work, we tackle such challenges by exploiting different modes of sketch data in a unified framework. Specifically, we propose a deep neural network SPFusionNet to capture the characteristic of sketch by fusing from its image and point set modes. The image modal component SketchNet learns hierarchically abstract ro-bust features and utilizes multi-level representations to produce pixel-wise feature maps, while the point set-modal component SPointNet captures local and global contexts of the sampled point set to produce point-wise feature maps. Then our framework aggregates these feature maps by a fusion network component to generate the sketch segmentation result. The extensive experimental evaluation and comparison with peer methods on our large SketchSeg dataset verify the effectiveness of the proposed framework. <s> BIB003
Comparison for the representative sketch-oriented network architectures. "-" means that it is vague or not mentioned in the original paper. The reported performances are the recognition accuracy of "acc.@top1". Abbreviations in this table are defined as: "para.": parameters; "ens.": ensemble; "stroke accu. pic.": stroke accumulated pictures; "R-FC": residual fully-connected layer; "cat.": category; "pad.": padding; "tru.": truncation; "augm.": specific augmentations; "tran.": transformer. key stroke point coordinates), respectively. The dual-branch interaction is implemented by a concatenation-based latefusion layer. This architecture demonstrates the complementarity of visual and temporal embedding spaces on sketch representation learning. This CNN-RNN dual-branch modeling idea has been widely applied to other sketch tasks, e.g., SPFusionNet BIB003 for sketch semantic segmentation. Addition to the parallel architectures of CNN and RNN, some cascaded architectures (e.g., CNN-to-RNN BIB001 , RNNto-CNN BIB002 ) also have been studied.