Deep Learning for Free-Hand Sketch: A Survey <s> Unique-Challenge of Sketch Collection <s> We develop a system for 3D object retrieval based on sketched feature lines as input. For objective evaluation, we collect a large number of query sketches from human users that are related to an existing data base of objects. The sketches turn out to be generally quite abstract with large local and global deviations from the original shape. Based on this observation, we decide to use a bag-of-features approach over computer generated line drawings of the objects. We develop a targeted feature transform based on Gabor filters for this system. We can show objectively that this transform is better suited than other approaches from the literature developed for similar tasks. Moreover, we demonstrate how to optimize the parameters of our, as well as other approaches, based on the gathered sketches. In the resulting comparison, our approach is significantly better than any other system described so far. <s> BIB001 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Unique-Challenge of Sketch Collection <s> We present an image retrieval system for the interactive search of photo collections using free-hand sketches depicting shape. We describe Gradient Field HOG (GF-HOG); an adapted form of the HOG descriptor suitable for Sketch Based Image Retrieval (SBIR). We incorporate GF-HOG into a Bag of Visual Words (BoVW) retrieval framework, and demonstrate how this combination may be harnessed both for robust SBIR, and for localizing sketched objects within an image. We evaluate over a large Flickr sourced dataset comprising 33 shape categories, using queries from 10 non-expert sketchers. We compare GF-HOG against state-of-the-art descriptors with common distance measures and language models for image retrieval, and explore how affine deformation of the sketch impacts search performance. GF-HOG is shown to consistently outperform retrieval versus SIFT, multi-resolution HOG, Self Similarity, Shape Context and Structure Tensor. Further, we incorporate semantic keywords into our GF-HOG system to enable the use of annotated sketches for image search. A novel graph-based measure of semantic similarity is proposed and two applications explored: semantic sketch based image retrieval and a semantic photo montage. <s> BIB002 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Unique-Challenge of Sketch Collection <s> Sketch-based 3D shape retrieval has become an important research topic in content-based 3D object retrieval. To foster this research area, two Shape Retrieval Contest (SHREC) tracks on this topic have been organized by us in 2012 and 2013 based on a small-scale and large-scale benchmarks, respectively. Six and five (nine in total) distinct sketch-based 3D shape retrieval methods have competed each other in these two contests, respectively. To measure and compare the performance of the top participating and other existing promising sketch-based 3D shape retrieval methods and solicit the state-of-the-art approaches, we perform a more comprehensive comparison of fifteen best (four top participating algorithms and eleven additional state-of-the-art methods) retrieval methods by completing the evaluation of each method on both benchmarks. The benchmarks, results, and evaluation tools for the two tracks are publicly available on our websites [1,2]. <s> BIB003 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Unique-Challenge of Sketch Collection <s> We present the Sketchy database, the first large-scale collection of sketch-photo pairs. We ask crowd workers to sketch particular photographic objects sampled from 125 categories and acquire 75,471 sketches of 12,500 objects. The Sketchy database gives us fine-grained associations between particular photos and sketches, and we use this to train cross-domain convolutional networks which embed sketches and photographs in a common feature space. We use our database as a benchmark for fine-grained retrieval and show that our learned representation significantly outperforms both hand-crafted features as well as deep features trained for sketch or photo classification. Beyond image retrieval, we believe the Sketchy database opens up new opportunities for sketch and image understanding and synthesis. <s> BIB004 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Unique-Challenge of Sketch Collection <s> In this study, we present a weakly supervised approach that discovers the discriminative structures of sketch images, given pairs of sketch images and web images. In contrast to traditional approaches that use global appearance features or relay on keypoint features, our aim is to automatically learn the shared latent structures that exist between sketch images and real images, even when there are significant appearance differences across its relevant real images. To accomplish this, we propose a deep convolutional neural network, named SketchNet. We firstly develop a triplet composed of sketch, positive and negative real image as the input of our neural network. To discover the coherent visual structures between the sketch and its positive pairs, we introduce the softmax as the loss function. Then a ranking mechanism is introduced to make the positive pairs obtain a higher score comparing over negative ones to achieve robust representation. Finally, we formalize above-mentioned constrains into the unified objective function, and create an ensemble feature representation to describe the sketch images. Experiments on the TUBerlin sketch benchmark demonstrate the effectiveness of our model and show that deep feature representation brings substantial improvements over other state-of-the-art methods on sketch classification. <s> BIB005 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Unique-Challenge of Sketch Collection <s> We investigate the problem of fine-grained sketch-based image retrieval (SBIR), where free-hand human sketches are used as queries to perform instance-level retrieval of images. This is an extremely challenging task because (i) visual comparisons not only need to be fine-grained but also executed cross-domain, (ii) free-hand (finger) sketches are highly abstract, making fine-grained matching harder, and most importantly (iii) annotated cross-domain sketch-photo datasets required for training are scarce, challenging many state-of-the-art machine learning techniques. In this paper, for the first time, we address all these challenges, providing a step towards the capabilities that would underpin a commercial sketch-based image retrieval application. We introduce a new database of 1,432 sketchphoto pairs from two categories with 32,000 fine-grained triplet ranking annotations. We then develop a deep tripletranking model for instance-level SBIR with a novel data augmentation and staged pre-training strategy to alleviate the issue of insufficient fine-grained training data. Extensive experiments are carried out to contribute a variety of insights into the challenges of data sufficiency and over-fitting avoidance when training deep networks for finegrained cross-domain ranking tasks. <s> BIB006 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Unique-Challenge of Sketch Collection <s> Human sketches are unique in being able to capture both the spatial topology of a visual object, as well as its subtle appearance details. Fine-grained sketch-based image retrieval (FG-SBIR) importantly leverages on such fine-grained characteristics of sketches to conduct instance-level retrieval of photos. Nevertheless, human sketches are often highly abstract and iconic, resulting in severe misalignments with candidate photos which in turn make subtle visual detail matching difficult. Existing FG-SBIR approaches focus only on coarse holistic matching via deep cross-domain representation learning, yet ignore explicitly accounting for fine-grained details and their spatial context. In this paper, a novel deep FG-SBIR model is proposed which differs significantly from the existing models in that: (1) It is spatially aware, achieved by introducing an attention module that is sensitive to the spatial position of visual details: (2) It combines coarse and fine semantic information via a shortcut connection fusion block: and (3) It models feature correlation and is robust to misalignments between the extracted features across the two domains by introducing a novel higher-order learnable energy function (HOLEF) based loss. Extensive experiments show that the proposed deep spatial-semantic attention model significantly outperforms the state-of-the-art. <s> BIB007 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Unique-Challenge of Sketch Collection <s> Notwithstanding aerial image retrieval is an important and obligatory task, existing retrieval systems lose their efficiency when there is no available aerial image used as the exemplar query. In this paper, we take free-hand sketches into consideration and address the problem of sketch-based aerial image retrieval. This is an extremely challenging task due to the complex surface structures and huge variations of resolutions of aerial images, and few works have been devoted to it. For the first time to our knowledge, we propose a framework to bridge the gap between sketches and aerial images. Specifically, an aerial sketch-image dataset is first collected. Sketches and aerial images are augmented to varied levels of details and used to train a multi-scale deep hierarchical model. The fully-connected layers of the deep model are used as cross-domain features, and the similarity between aerial images and sketches is measured by the Euclidean distance. Experiments on several public aerial image datasets demonstrate the efficiency and superiority of the proposed method. <s> BIB008 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Unique-Challenge of Sketch Collection <s> This paper investigates the problem of retrieving aerial scene images by using semantic sketches, since the state-of-the-art retrieval systems turn out to be invalid when there is no exemplar query aerial image available. However, due to the complex surface structures and huge variations of resolutions of aerial images, it is very challenging to retrieve aerial images with sketches and few studies have been devoted to this task. In this article, for the first time to our knowledge, we propose a framework to bridge the gap between sketches and aerial images. First, an aerial sketch-image database is collected, and the images and sketches it contains are augmented to various levels of details. We then train a multi-scale deep model by the new dataset. The fully-connected layers of the network in each scale are finally connected and used as cross-domain features, and the Euclidean distance is used to measure the cross-domain similarity between aerial images and sketches. Experiments on several commonly used aerial image datasets demonstrate the superiority of the proposed method compared with the traditional approaches. <s> BIB009 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Unique-Challenge of Sketch Collection <s> Sketches and real-world user interface examples are frequently used in multiple stages of the user interface design process. Unfortunately, finding relevant user interface examples, especially in large-scale datasets, is a highly challenging task because user interfaces have aesthetic and functional properties that are only indirectly reflected by their corresponding pixel data and meta-data. This paper introduces Swire, a sketch-based neural-network-driven technique for retrieving user interfaces. We collect the first large-scale user interface sketch dataset from the development of Swire that researchers can use to develop new sketch-based data-driven design interfaces and applications. Swire achieves high performance for querying user interfaces: for a known validation task it retrieves the most relevant example as within the top-10 results for over 60% of queries. With this technique, for the first time designers can accurately retrieve relevant user interface examples with free-form sketches natural to their design workflows. We demonstrate several novel applications driven by Swire that could greatly augment the user interface design process. <s> BIB010 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Unique-Challenge of Sketch Collection <s> In this paper, we investigate the problem of zero-shot sketch-based image retrieval (ZS-SBIR), where human sketches are used as queries to conduct retrieval of photos from unseen categories. We importantly advance prior arts by proposing a novel ZS-SBIR scenario that represents a firm step forward in its practical application. The new setting uniquely recognizes two important yet often neglected challenges of practical ZS-SBIR, (i) the large domain gap between amateur sketch and photo, and (ii) the necessity for moving towards large-scale retrieval. We first contribute to the community a novel ZS-SBIR dataset, QuickDraw-Extended, that consists of 330,000 sketches and 204,000 photos spanning across 110 categories. Highly abstract amateur human sketches are purposefully sourced to maximize the domain gap, instead of ones included in existing datasets that can often be semi-photorealistic. We then formulate a ZS-SBIR framework to jointly model sketches and photos into a common embedding space. A novel strategy to mine the mutual information among domains is specifically engineered to alleviate the domain gap. External semantic knowledge is further embedded to aid semantic transfer. We show that, rather surprisingly, retrieval performance significantly outperforms that of state-of-the-art on existing datasets that can already be achieved using a reduced version of our model. We further demonstrate the superior performance of our full model by comparing with a number of alternatives on the newly proposed dataset. The new dataset, plus all training and testing code of our model, will be publicly released to facilitate future research <s> BIB011
Free-hand sketch stands as a special data domain, its collection has some unique-challenges: (i) Sketching is a dynamic and temporal processing, thus saving as the static picture (e.g., JPEG, PNG) will loss too much useful information. In order to enable more extensive research, it is better to record the stroke position and timing information. This is to TABLE 2 Summary for the representative sketch datasets. " " denotes "yes/available/provided". Both of "grouping" and "segmentation" annotations are in stroke-level. "K" and "M" means "thousand" and "million", respectively. "cat." means "category". The stroke column will be marked as available if the sketches are provided in SVG files or coordinate arrays. BIB007 568 sketches, 568 photos 1 o pairing Sketchy BIB004 75K sketches, 12K photos 125 o class 12K objects sketch&UI BIB010 1998 sketches, 1998 photos 23 o class, pairing UI QuickDrawExtended BIB011 330K sketches, 204K photos 110 o class TU-Berlin Extended BIB005 20K sketches, 191K photos 250 o class sketch Flickr15K BIB002 330 sketches, 15K photos 33 o class Aerial-SI BIB008 , BIB009 400 sketches, 3.3K photos 10 o, s class aerial scene HUST-SI  20K sketches, 31K photos 250 o class SBSR BIB001 1814 sketches, 1814 3D models 161 o class SHREC'13 BIB003 7200 sketches, 1258 3D models 90 o class SHREC'14  12680 say that generally sketch collection needs specific designed interactive interface to record the sketch domain-unique information. In some cases, for further analysis, it also needs to record some extra information of the drawers, e.g., gender, nationality. More complicated than natural photo collection (crawling and saving as picture directly), sketch collection generally needs to save sketch as Scalable Vector Graphics (SVG) BIB006 and record other additional information, e.g., stroke position/timing information. (ii) Although, with popularization of smart touch-screen devices, more and more free-hand sketches are available in the internet, it is still difficult to collection sketches via crawling. The main reasons are: (a) The ready-made sketches in the internet are saved as static pictures, without stroke temporal and timing information, so that only support to model sketch statically. (b) The quality of sketch is exceedingly dependent on the status of collection device, e.g., screen resolution, predefined stroke thickness. Moreover, some sketches in the internet are too messy to recognize even by humans. Thus, it is difficult to obtain sketches in the acceptable drawing quality. This is distinctively different to photo collection, which is mainly based on crawling in the internet. (c) When collect fine-grained cross-modal datasets, it needs to collect the fine-grained/instance-level cross-modal sample pairs. In this case, selecting and then matching the cross-modal sample pairs on the fine-grained criterion is time-consuming, while drawing sketches referring to the 5. https://en.wikipedia.org/wiki/Scalable Vector Graphics counterpart in other modality could be more efficient. In addition, it often happens that collectors can not search for a adequately well-matched sketches for the target sample from other modality. (iii) As aforementioned, sketch is full of the diversity of drawing styles. Thus, in ideal conditions, any sketch dataset should involve as many drawing participants and styles as possible. It is better to ensure that the drawing participants vary in as many ages and regions as possible. However, during crawling, we can not know any extra information about the sketchers in the internet. (iv) There are two extreme cases that always happen: (a) The crowd-sourcing participants often draw deliberately to obtain the well-drawn sketches. (b) The online game participants often draw overly arbitrarily resulting in the highly noisy sketches.