Deep Learning for Free-Hand Sketch: A Survey <s> Sketch Semantic Segmentation (SSS): <s> In this paper, we study the problem of how to segment a freehand sketch at the object level. By carefully considering the basic principles of human perceptual organization, a real-time solution is presented to automatically segment a user's sketch during his/her drawing. First, a graph-based sketch segmentation algorithm is proposed to segment a cluttered sketch into multiple parts based on the factor of proximity. Then, to improve the ability of detecting semantically meaningful objects, a semantic-based approach is introduced to simulate the past experience in the perceptual system by leveraging a web-scale clipart database. Finally, other important factors learnt from past experience, such as similarity, symmetry, direction, and closure, are also taken into account to make the approach more robust and practical. The proposed sketch segmentation framework has ability to handle complex sketches with overlapped objects. Extensive experimental results show the effectiveness of the proposed framework and algorithms. <s> BIB001 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch Semantic Segmentation (SSS): <s> We introduce a new approach for segmentation and label transfer in sketches that substantially improves the state of the art. We build on successful techniques to find how likely each segment is to belong to a label, and use a Conditional Random Field to find the most probable global configuration. Our method is trained fully on the sketch domain, such that it can handle abstract sketches that are very far from 3D meshes. It also requires a small quantity of annotated data, which makes it easily adaptable to new datasets. The testing phase is completely automatic, and our performance is comparable to state-of-the-art methods that require manual tuning and a considerable amount of previous annotation [Huang et al. 2014]. <s> BIB002 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch Semantic Segmentation (SSS): <s> Large repositories of 3D shapes provide valuable input for data-driven analysis and modeling tools. They are especially powerful once annotated with semantic information such as salient regions and functional parts. We propose a novel active learning method capable of enriching massive geometric datasets with accurate semantic region annotations. Given a shape collection and a user-specified region label our goal is to correctly demarcate the corresponding regions with minimal manual work. Our active framework achieves this goal by cycling between manually annotating the regions, automatically propagating these annotations across the rest of the shapes, manually verifying both human and automatic annotations, and learning from the verification results to improve the automatic propagation algorithm. We use a unified utility function that explicitly models the time cost of human input across all steps of our method. This allows us to jointly optimize for the set of models to annotate and for the set of models to verify based on the predicted impact of these actions on the human efficiency. We demonstrate that incorporating verification of all produced labelings within this unified objective improves both accuracy and efficiency of the active learning procedure. We automatically propagate human labels across a dynamic shape network using a conditional random field (CRF) framework, taking advantage of global shape-to-shape similarities, local feature similarities, and point-to-point correspondences. By combining these diverse cues we achieve higher accuracy than existing alternatives. We validate our framework on existing benchmarks demonstrating it to be significantly more efficient at using human input compared to previous techniques. We further validate its efficiency and robustness by annotating a massive shape dataset, labeling over 93,000 shape parts, across multiple model classes, and providing a labeled part collection more than one order of magnitude larger than existing ones. <s> BIB003 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch Semantic Segmentation (SSS): <s> Conventional unsupervised image segmentation methods use color and geometric information and apply clustering algorithms over pixels. They preserve object boundaries well but often suffer from over-segmentation due to noise and artifacts in the images. In this paper, we contribute on a preprocessing step for image smoothing, which alleviates the burden of conventional unsupervised image segmentation and enhance their performance. Our approach relies on a convolutional autoencoder (CAE) with the total variation loss (TVL) for unsupervised learning. We show that, after our CAE-TVL preprocessing step, the over-segmentation effect is significantly reduced using the same unsupervised image segmentation methods. We evaluate our approach using the BSDS500 image segmentation benchmark dataset and show the performance enhancement introduced by our approach in terms of both increased segmentation accuracy and reduced computation time. We examine the robustness of the trained CAE and show that it is directly applicable to other natural scene images. <s> BIB004 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch Semantic Segmentation (SSS): <s> In this work we aim to develop a universal sketch grouper. That is, a grouper that can be applied to sketches of any category in any domain to group constituent strokes/segments into semantically meaningful object parts. The first obstacle to this goal is the lack of large-scale datasets with grouping annotation. To overcome this, we contribute the largest sketch perceptual grouping (SPG) dataset to date, consisting of 20, 000 unique sketches evenly distributed over 25 object categories. Furthermore, we propose a novel deep universal perceptual grouping model. The model is learned with both generative and discriminative losses. The generative losses improve the generalisation ability of the model to unseen object categories and datasets. The discriminative losses include a local grouping loss and a novel global grouping loss to enforce global grouping consistency. We show that the proposed model significantly outperforms the state-of-the-art groupers. Further, we show that our grouper is useful for a number of sketch analysis tasks including sketch synthesis and fine-grained sketch-based image retrieval (FG-SBIR). <s> BIB005 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch Semantic Segmentation (SSS): <s> In this work we address the task of semantic image segmentation with Deep Learning and make three main contributions that are experimentally shown to have substantial practical merit. First , we highlight convolution with upsampled filters, or ‘atrous convolution’, as a powerful tool in dense prediction tasks. Atrous convolution allows us to explicitly control the resolution at which feature responses are computed within Deep Convolutional Neural Networks. It also allows us to effectively enlarge the field of view of filters to incorporate larger context without increasing the number of parameters or the amount of computation. Second , we propose atrous spatial pyramid pooling (ASPP) to robustly segment objects at multiple scales. ASPP probes an incoming convolutional feature layer with filters at multiple sampling rates and effective fields-of-views, thus capturing objects as well as image context at multiple scales. Third , we improve the localization of object boundaries by combining methods from DCNNs and probabilistic graphical models. The commonly deployed combination of max-pooling and downsampling in DCNNs achieves invariance but has a toll on localization accuracy. We overcome this by combining the responses at the final DCNN layer with a fully connected Conditional Random Field (CRF), which is shown both qualitatively and quantitatively to improve localization performance. Our proposed “DeepLab” system sets the new state-of-art at the PASCAL VOC-2012 semantic image segmentation task, reaching 79.7 percent mIOU in the test set, and advances the results on three other datasets: PASCAL-Context, PASCAL-Person-Part, and Cityscapes. All of our code is made publicly available online. <s> BIB006 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch Semantic Segmentation (SSS): <s> We investigate the problem of stroke-level sketch segmentation, which is to train machines to assign strokes with semantic part labels given a input sketch. Solving the problem of sketch segmentation opens the door for fine-grained sketch interpretation, which can benefit many novel sketch-based applications, including sketch recognition and sketch-based image retrieval. In this paper, we treat the problem of stroke-level sketch segmentation as a seqence-to-sequence generation problem, and a reccurent nueral networks (RNN)-based model SketchSegNet is presented to translate sequence of strokes into thier semantic part labels. In addition, for the first time a large-scale stroke-level sketch segmentation dataset is proposed, which is composed of 57K annotated free-hand human sketch selected from QuickDraw. Experimental results of stroke-level sketch segmentation on this novel dataset shows that our approach offers an average accuracy over 90% for stroke labeling. <s> BIB007 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch Semantic Segmentation (SSS): <s> We present a simple and efficient method based on deep learning to automatically decompose sketched objects into semantically valid parts. We train a deep neural network to transfer existing segmentations and labelings from 3D models to freehand sketches without requiring numerous well-annotated sketches as training data. The network takes the binary image of a sketched object as input and produces a corresponding segmentation map with per-pixel labelings as output. A subsequent post-process procedure with multi-label graph cuts further refines the segmentation and labeling result. We validate our proposed method on two sketch datasets. Experiments show that our method outperforms the state-of-the-art method in terms of segmentation and labeling accuracy and is significantly faster, enabling further integration in interactive drawing systems. We demonstrate the efficiency of our method in a sketch-based modeling application that automatically transforms input sketches into 3D models by part assembly. <s> BIB008 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch Semantic Segmentation (SSS): <s> Hand-drawn objects usually consist of multiple semantically meaningful parts. For example, a stick figure consists of a head, a torso, and pairs of legs and arms. Efficient and accurate identification of these subparts promises to significantly improve algorithms for stylization, deformation, morphing and animation of 2D drawings. In this paper, we propose a neural network model that segments symbols into stroke-level components. Our segmentation framework has two main elements: a fixed feature extractor and a Multilayer Perceptron (MLP) network that identifies a component based on the feature. As the feature extractor we utilize an encoder of a stroke-rnn, which is our newly proposed generative Variational Auto-Encoder (VAE) model that reconstructs symbols on a stroke by stroke basis. Experiments show that a single encoder could be reused for segmenting multiple categories of sketched symbols with negligible effects on segmentation accuracies. Our segmentation scores surpass existing methodologies on an available small state of the art dataset. Moreover, extensive evaluations on our newly annotated big dataset demonstrate that our framework obtains significantly better accuracies as compared to baseline models. We release the dataset to the community. <s> BIB009 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch Semantic Segmentation (SSS): <s> We investigate the problem of stroke-level sketch segmentation, which is to automatically assign strokes of a given sketch with semantic labels. Solving the problem of sketch segmentation opens the door for fine-grained sketch interpretation, which can benefit many novel sketch-based applications, including sketch recognition and sketch-based image retrieval. In this paper, we propose an approach for multi-class sketch semantic segmentation by considering it as a sequence-to-sequence generation problem. Specifically, an end-to-end learned network SketchSegNet+, built on recurrent neural networks (RNN), is presented to translate a sequence of strokes into a sequence of semantic labels. In addition, a large-scale stroke-level sketch segmentation dataset is constructed for the first time, which is composed of 150K annotated free-hand human sketch selected from QuickDraw. The dataset will be released publicly. The experimental results of stroke-level sketch semantic segmentation on this novel dataset and the SPG dataset demonstrate the effectiveness of our approach. <s> BIB010 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch Semantic Segmentation (SSS): <s> The sketch segmentation problem remains largely unsolved because conventional methods are greatly challenged by the highly abstract appearances of freehand sketches and their numerous shape variations. In this work, we tackle such challenges by exploiting different modes of sketch data in a unified framework. Specifically, we propose a deep neural network SPFusionNet to capture the characteristic of sketch by fusing from its image and point set modes. The image modal component SketchNet learns hierarchically abstract ro-bust features and utilizes multi-level representations to produce pixel-wise feature maps, while the point set-modal component SPointNet captures local and global contexts of the sampled point set to produce point-wise feature maps. Then our framework aggregates these feature maps by a fusion network component to generate the sketch segmentation result. The extensive experimental evaluation and comparison with peer methods on our large SketchSeg dataset verify the effectiveness of the proposed framework. <s> BIB011
Sketch semantic segmentation has drawn a lot of attention BIB001 ,  , BIB002 in free-hand sketch community as a classic topic before the eruption of deep learning. The goal of sketch semantic segmentation is to predict the semantic labels for sketch strokes.The major difference between sketch semantic segmentation and sketch grouping is that the semantic segmentation results explicitly provide the part labels (i.e., the concrete semantic category names) for each stroke, while the grouping results only provide the aggregation relationships for the strokes. As argued in BIB005 , the similarity between sketch perceptual grouping and sketch semantic segmentation is analogous to that between unsupervised image segmentation BIB004 and semantic segmentation BIB006 . According to the network architectures, the existing deep learning models for sketch semantic segmentation can be mainly divided into several manners, CNN, RNN, CNN-RNN dual-branch, VAE based models, etc. The representative models mainly contain Stroke-RNN BIB009 , SketchSeg-Net BIB007 , SketchSegNet+ BIB010 , SPFusionNet BIB011 , etc. Li et al. BIB008 train a CNN-based network to transfer the well-annotated segmentations and labellings from 3D dataset to sketch domain. They use annotated 3D data  , BIB003 to produce edge-map with partial annotations as the fake sketch to train the segmentation networks. Qi et al. propose SketchSegNet BIB007 and SketchSegNet+ BIB010 . SketchSegNet+ BIB010 considers the sketch stroke orderings and is able to work over multiple object categories. In particular, SketchSegNet and SketchSegNet+ work in RNNbased VEA framework, where the Gaussian mixture model (GMM) layers of SketchRNN are replaced with the fullyconnected layers combining with softmax layer to predict the part labels. The architecture of Stroke-RNN BIB009 equals to the encoder of SketchRNN. SPFusionNet BIB011 uses the late fusion of CNN-RNN dual-branch to represent sketch to conduct segmentation training. Comments: Essentially, sketch perceptual grouping (SPG) is a stroke-level clustering, however, sketch semantic segmentation (SSS) is a explicitly semantic based stroke-level classification. Considering the supervision during training, SSS needs more stronger supervision than SPG. Specifically, during training, SSS system needs to know not only which strokes are connected together to denote a semantic part but also the meaningful name of this part. Comments: Even though sketch semantic segmentation can be modeled by CNNs. However, CNNs lose the stroke orderings and temporal patterns of sketch, learning sketch representations on the patch basis. This CNN-based representation makes SSS fall into the same workflow as that of natural photo semantic segmentation. Comments: In some cases, the non-deep grouping methods mainly rely on thresholding low-level geometric properties among the strokes. This often results in that strokes with equal geometric properties but different semantics will be grouped. Thanks to RNN-based representation, deep SPG methods consider more high-level semantic and temporal information, from the perspective of sketching.