Deep Learning for Free-Hand Sketch: A Survey <s> FREE-HAND SKETCH DATASETS <s> Gives a basic idea and its fundamental algorithms of the visual interface for image database systems. The QVE (Query by Visual Example) accepts a sketch roughly drawn by a user to retrieve the original image and the similar images. The system evaluates the similarity between the rough sketch, i.e. a visual example, and each of the image data in the database automatically. The QVE interface is implemented and examined on an experimental electronic art gallery called ART MUSEUM. This paper also gives some experimental results and a current evaluation. The algorithms are quite effective for content based image retrieval. > <s> BIB001 </s> Deep Learning for Free-Hand Sketch: A Survey <s> FREE-HAND SKETCH DATASETS <s> Humans have used sketching to depict our visual world since prehistoric times. Even today, sketching is possibly the only rendering technique readily available to all humans. This paper is the first large scale exploration of human sketches. We analyze the distribution of non-expert sketches of everyday objects such as 'teapot' or 'car'. We ask humans to sketch objects of a given category and gather 20,000 unique sketches evenly distributed over 250 object categories. With this dataset we perform a perceptual study and find that humans can correctly identify the object category of a sketch 73% of the time. We compare human performance against computational recognition methods. We develop a bag-of-features sketch representation and use multi-class support vector machines, trained on our sketch dataset, to classify sketches. The resulting recognition method is able to identify unknown sketches with 56% accuracy (chance is 0.4%). Based on the computational model, we demonstrate an interactive sketch recognition system. We release the complete crowd-sourced dataset of sketches to the community. <s> BIB002 </s> Deep Learning for Free-Hand Sketch: A Survey <s> FREE-HAND SKETCH DATASETS <s> We investigate the problem of fine-grained sketch-based image retrieval (SBIR), where free-hand human sketches are used as queries to perform instance-level retrieval of images. This is an extremely challenging task because (i) visual comparisons not only need to be fine-grained but also executed cross-domain, (ii) free-hand (finger) sketches are highly abstract, making fine-grained matching harder, and most importantly (iii) annotated cross-domain sketch-photo datasets required for training are scarce, challenging many state-of-the-art machine learning techniques. In this paper, for the first time, we address all these challenges, providing a step towards the capabilities that would underpin a commercial sketch-based image retrieval application. We introduce a new database of 1,432 sketchphoto pairs from two categories with 32,000 fine-grained triplet ranking annotations. We then develop a deep tripletranking model for instance-level SBIR with a novel data augmentation and staged pre-training strategy to alleviate the issue of insufficient fine-grained training data. Extensive experiments are carried out to contribute a variety of insights into the challenges of data sufficiency and over-fitting avoidance when training deep networks for finegrained cross-domain ranking tasks. <s> BIB003 </s> Deep Learning for Free-Hand Sketch: A Survey <s> FREE-HAND SKETCH DATASETS <s> We present the Sketchy database, the first large-scale collection of sketch-photo pairs. We ask crowd workers to sketch particular photographic objects sampled from 125 categories and acquire 75,471 sketches of 12,500 objects. The Sketchy database gives us fine-grained associations between particular photos and sketches, and we use this to train cross-domain convolutional networks which embed sketches and photographs in a common feature space. We use our database as a benchmark for fine-grained retrieval and show that our learned representation significantly outperforms both hand-crafted features as well as deep features trained for sketch or photo classification. Beyond image retrieval, we believe the Sketchy database opens up new opportunities for sketch and image understanding and synthesis. <s> BIB004 </s> Deep Learning for Free-Hand Sketch: A Survey <s> FREE-HAND SKETCH DATASETS <s> We present sketch-rnn, a recurrent neural network (RNN) able to construct stroke-based drawings of common objects. The model is trained on thousands of crude human-drawn images representing hundreds of classes. We outline a framework for conditional and unconditional sketch generation, and describe new robust training methods for generating coherent sketch drawings in a vector format. <s> BIB005 </s> Deep Learning for Free-Hand Sketch: A Survey <s> FREE-HAND SKETCH DATASETS <s> In this work we aim to develop a universal sketch grouper. That is, a grouper that can be applied to sketches of any category in any domain to group constituent strokes/segments into semantically meaningful object parts. The first obstacle to this goal is the lack of large-scale datasets with grouping annotation. To overcome this, we contribute the largest sketch perceptual grouping (SPG) dataset to date, consisting of 20, 000 unique sketches evenly distributed over 25 object categories. Furthermore, we propose a novel deep universal perceptual grouping model. The model is learned with both generative and discriminative losses. The generative losses improve the generalisation ability of the model to unseen object categories and datasets. The discriminative losses include a local grouping loss and a novel global grouping loss to enforce global grouping consistency. We show that the proposed model significantly outperforms the state-of-the-art groupers. Further, we show that our grouper is useful for a number of sketch analysis tasks including sketch synthesis and fine-grained sketch-based image retrieval (FG-SBIR). <s> BIB006 </s> Deep Learning for Free-Hand Sketch: A Survey <s> FREE-HAND SKETCH DATASETS <s> We investigate the problem of stroke-level sketch segmentation, which is to automatically assign strokes of a given sketch with semantic labels. Solving the problem of sketch segmentation opens the door for fine-grained sketch interpretation, which can benefit many novel sketch-based applications, including sketch recognition and sketch-based image retrieval. In this paper, we propose an approach for multi-class sketch semantic segmentation by considering it as a sequence-to-sequence generation problem. Specifically, an end-to-end learned network SketchSegNet+, built on recurrent neural networks (RNN), is presented to translate a sequence of strokes into a sequence of semantic labels. In addition, a large-scale stroke-level sketch segmentation dataset is constructed for the first time, which is composed of 150K annotated free-hand human sketch selected from QuickDraw. The dataset will be released publicly. The experimental results of stroke-level sketch semantic segmentation on this novel dataset and the SPG dataset demonstrate the effectiveness of our approach. <s> BIB007 </s> Deep Learning for Free-Hand Sketch: A Survey <s> FREE-HAND SKETCH DATASETS <s> In this paper, we investigate the problem of zero-shot sketch-based image retrieval (ZS-SBIR), where human sketches are used as queries to conduct retrieval of photos from unseen categories. We importantly advance prior arts by proposing a novel ZS-SBIR scenario that represents a firm step forward in its practical application. The new setting uniquely recognizes two important yet often neglected challenges of practical ZS-SBIR, (i) the large domain gap between amateur sketch and photo, and (ii) the necessity for moving towards large-scale retrieval. We first contribute to the community a novel ZS-SBIR dataset, QuickDraw-Extended, that consists of 330,000 sketches and 204,000 photos spanning across 110 categories. Highly abstract amateur human sketches are purposefully sourced to maximize the domain gap, instead of ones included in existing datasets that can often be semi-photorealistic. We then formulate a ZS-SBIR framework to jointly model sketches and photos into a common embedding space. A novel strategy to mine the mutual information among domains is specifically engineered to alleviate the domain gap. External semantic knowledge is further embedded to aid semantic transfer. We show that, rather surprisingly, retrieval performance significantly outperforms that of state-of-the-art on existing datasets that can already be achieved using a reduced version of our model. We further demonstrate the superior performance of our full model by comparing with a number of alternatives on the newly proposed dataset. The new dataset, plus all training and testing code of our model, will be publicly released to facilitate future research <s> BIB008
This section will discuss the unique-challenges of free-hand sketch collection, and introduce the recently representative deep learning oriented datasets by a detailed taxonomy. As shown in Figure 5 , according to the data modalities involved, free-hand sketch based datasets can be categorized as two kinds: (i) single-modal and (ii) multimodal datasets. Single-modal sketch datasets only consist of sketches, which are mainly used for sketch recognition, retrieval/fast retrieval, perceptual grouping, segmentation, generation, etc. Multi-modal sketch datasets consist of both sketches and the samples from other modalities (e.g., natural photo, 3D shape, text, cartoon, video), which are mainly used for the retrieval/matching, generation across sketch and other modalities. Moreover, both single-modal and multi-modal sketch datasets can be divided in two kinds: (a) coarse-grained and (b) fine-grained. The coarse-grained single-modal sketch datasets (e.g., TU-Berlin BIB002 , QuickDraw BIB005 ) are usually used for sketch recognition, sketch retrieval. The fine-grained single-modal sketch datasets BIB007 , BIB006 have fine-grained annotations, which can serve for the fine-grained sketch tasks, e.g., perceptual grouping, segmentation, parse. The fined-grained multi-modal sketch datasets (e.g., SketchX Shoe BIB003 ) provide the instance-level pairing information, usually with fine-grained annotations, i.e., local attributes. Hence, the fined-grained multi-modal sketch datasets can be used for fine-grained sketch based crossmodal retrieval and generation tasks. Existing collection approaches mainly include: (i) crowdsourcing selecting and matching on existing datasets, e.g., Doodle2Sketch QuickDraw-Extended BIB008 , (ii) crowdsourcing drawing from scratch, e.g., SketchX Shoe BIB003 , Sketchy BIB004 . (iii) collecting via online drawing game, e.g., Google QuickDraw BIB005 . In particular, for fine-grained multi-modal sketch datasets, crowd-sourcing drawing is more suitable, since fine-grained selecting and matching takes more time. In addition, some novel concepts and tasks on learning fine-grained sketch patterns are widely studied in recent years, leading to some new sketch datasets, e.g., SketchX Shoe BIB003 , Sketchy BIB004 , QuickDraw BIB005 . This section will brief the most representative datasets in current stage, which are also the most frequently-used datasets. Some wellknown but overly small-scale datasets will not be discussed in this section, e.g., the dataset proposed by Hirata and Kato BIB001 .