Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> A family of programs has been developed to allow graphical input through continuous digitizing. Drawing data, sampled at a high and constant rate, is compressed and mapped into lines and splines, in two and three dimensions. This is achieved by inferring a particular user's intentions from measures of speed and pressure.Recent experiments have shown that even the most basic inference making cannot rely solely upon knowledge of the user's drawing style, but needs additional knowledge of the subject being drawn, the protocols of its domain, and the stage of development of the user's design. This requirement implies a higher level of machine intelligence than currently exists. An alternate approach is to increase the user's involvement in the recognition process.Contrary to previous efforts to move from sketch to mechanical drawing without human intervention, this paper reports on an interactive system for graphical input in which the user overtly partakes in training the machine and massaging the data at all levels of interpretation. The initial routines for data compression employ parallel functions for extracting such features as bentness, straightness, and endness. These are planned for implementation in microprocessors.Results offer a system for rapid (and enjoyable) graphical input with real-time interpretation, the beginnings of an intelligent tablet. <s> BIB001 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> Humans have used sketching to depict our visual world since prehistoric times. Even today, sketching is possibly the only rendering technique readily available to all humans. This paper is the first large scale exploration of human sketches. We analyze the distribution of non-expert sketches of everyday objects such as 'teapot' or 'car'. We ask humans to sketch objects of a given category and gather 20,000 unique sketches evenly distributed over 250 object categories. With this dataset we perform a perceptual study and find that humans can correctly identify the object category of a sketch 73% of the time. We compare human performance against computational recognition methods. We develop a bag-of-features sketch representation and use multi-class support vector machines, trained on our sketch dataset, to classify sketches. The resulting recognition method is able to identify unknown sketches with 56% accuracy (chance is 0.4%). Based on the computational model, we demonstrate an interactive sketch recognition system. We release the complete crowd-sourced dataset of sketches to the community. <s> BIB002 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> We present an image retrieval system for the interactive search of photo collections using free-hand sketches depicting shape. We describe Gradient Field HOG (GF-HOG); an adapted form of the HOG descriptor suitable for Sketch Based Image Retrieval (SBIR). We incorporate GF-HOG into a Bag of Visual Words (BoVW) retrieval framework, and demonstrate how this combination may be harnessed both for robust SBIR, and for localizing sketched objects within an image. We evaluate over a large Flickr sourced dataset comprising 33 shape categories, using queries from 10 non-expert sketchers. We compare GF-HOG against state-of-the-art descriptors with common distance measures and language models for image retrieval, and explore how affine deformation of the sketch impacts search performance. GF-HOG is shown to consistently outperform retrieval versus SIFT, multi-resolution HOG, Self Similarity, Shape Context and Structure Tensor. Further, we incorporate semantic keywords into our GF-HOG system to enable the use of annotated sketches for image search. A novel graph-based measure of semantic similarity is proposed and two applications explored: semantic sketch based image retrieval and a semantic photo montage. <s> BIB003 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> This paper presents an approach to estimating the 2.1D sketch from monocular, low-level visual cues. We use a low-level segmenter to partition the image into regions, and, then, estimate their 2.1D sketch, subject to figure-ground and similarity constraints between neighboring regions. The 2.1D sketch assigns a depth ordering to image regions which are expected to correspond to objects and surfaces in the scene. This is cast as a constrained convex optimization problem, and solved within the optimization transfer framework. The optimization objective takes into account the curvature and convexity of parts of region boundaries, appearance, and spatial layout properties of regions. Our new optimization transfer algorithm admits a closed-form expression of the duality gap, and thus allows explicit computation of the achieved accuracy. The algorithm is efficient with quadratic complexity in the number of constraints between image regions. Quantitative and qualitative results on challenging, real-world images of Berkeley segmentation, Geometric Context, and Stanford Make3D datasets demonstrate our high accuracy, efficiency, and robustness. <s> BIB004 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> Sketch-based 3D shape retrieval has become an important research topic in content-based 3D object retrieval. To foster this research area, two Shape Retrieval Contest (SHREC) tracks on this topic have been organized by us in 2012 and 2013 based on a small-scale and large-scale benchmarks, respectively. Six and five (nine in total) distinct sketch-based 3D shape retrieval methods have competed each other in these two contests, respectively. To measure and compare the performance of the top participating and other existing promising sketch-based 3D shape retrieval methods and solicit the state-of-the-art approaches, we perform a more comprehensive comparison of fifteen best (four top participating algorithms and eleven additional state-of-the-art methods) retrieval methods by completing the evaluation of each method on both benchmarks. The benchmarks, results, and evaluation tools for the two tracks are publicly available on our websites [1,2]. <s> BIB005 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> Retrieving 3D models from 2D human sketches has received considerable attention in the areas of graphics, image retrieval, and computer vision. Almost always in state of the art approaches a large amount of “best views” are computed for 3D models, with the hope that the query sketch matches one of these 2D projections of 3D models using predefined features. We argue that this two stage approach (view selection - matching) is pragmatic but also problematic because the “best views” are subjective and ambiguous, which makes the matching inputs obscure. This imprecise nature of matching further makes it challenging to choose features manually. Instead of relying on the elusive concept of “best views” and the hand-crafted features, we propose to define our views using a minimalism approach and learn features for both sketches and views. Specifically, we drastically reduce the number of views to only two predefined directions for the whole dataset. Then, we learn two Siamese Convolutional Neural Networks (CNNs), one for the views and one for the sketches. The loss function is defined on the within-domain as well as the cross-domain similarities. Our experiments on three benchmark datasets demonstrate that our method is significantly better than state of the art approaches, and outperforms them in all conventional metrics. <s> BIB006 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> We propose a multi-scale multi-channel deep neural network framework that, for the first time, yields sketch recognition performance surpassing that of humans. Our superior performance is a result of explicitly embedding the unique characteristics of sketches in our model: (i) a network architecture designed for sketch rather than natural photo statistics, (ii) a multi-channel generalisation that encodes sequential ordering in the sketching process, and (iii) a multi-scale network ensemble with joint Bayesian fusion that accounts for the different levels of abstraction exhibited in free-hand sketches. We show that state-of-the-art deep networks specifically engineered for photos of natural objects fail to perform well on sketch recognition, regardless whether they are trained using photo or sketch. Our network on the other hand not only delivers the best performance on the largest human sketch dataset to date, but also is small in size making efficient training possible using just CPUs. <s> BIB007 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> We investigate the problem of fine-grained sketch-based image retrieval (SBIR), where free-hand human sketches are used as queries to perform instance-level retrieval of images. This is an extremely challenging task because (i) visual comparisons not only need to be fine-grained but also executed cross-domain, (ii) free-hand (finger) sketches are highly abstract, making fine-grained matching harder, and most importantly (iii) annotated cross-domain sketch-photo datasets required for training are scarce, challenging many state-of-the-art machine learning techniques. In this paper, for the first time, we address all these challenges, providing a step towards the capabilities that would underpin a commercial sketch-based image retrieval application. We introduce a new database of 1,432 sketchphoto pairs from two categories with 32,000 fine-grained triplet ranking annotations. We then develop a deep tripletranking model for instance-level SBIR with a novel data augmentation and staged pre-training strategy to alleviate the issue of insufficient fine-grained training data. Extensive experiments are carried out to contribute a variety of insights into the challenges of data sufficiency and over-fitting avoidance when training deep networks for finegrained cross-domain ranking tasks. <s> BIB008 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> We present the Sketchy database, the first large-scale collection of sketch-photo pairs. We ask crowd workers to sketch particular photographic objects sampled from 125 categories and acquire 75,471 sketches of 12,500 objects. The Sketchy database gives us fine-grained associations between particular photos and sketches, and we use this to train cross-domain convolutional networks which embed sketches and photographs in a common feature space. We use our database as a benchmark for fine-grained retrieval and show that our learned representation significantly outperforms both hand-crafted features as well as deep features trained for sketch or photo classification. Beyond image retrieval, we believe the Sketchy database opens up new opportunities for sketch and image understanding and synthesis. <s> BIB009 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> We investigate whether it is possible to improve the performance of automated facial forensic sketch matching by learning from examples of facial forgetting over time. Forensic facial sketch recognition is a key capability for law enforcement, but remains an unsolved problem. It is extremely challenging because there are three distinct contributors to the domain gap between forensic sketches and photos: The well-studied sketch-photo modality gap, and the less studied gaps due to (i) the forgetting process of the eye-witness and (ii) their inability to elucidate their memory. In this paper, we address the memory problem head on by introducing a database of 400 forensic sketches created at different time-delays. Based on this database we build a model to reverse the forgetting process. Surprisingly, we show that it is possible to systematically "un-forget" facial details. Moreover, it is possible to apply this model to dramatically improve forensic sketch recognition in practice: we achieve the state of the art results when matching 195 benchmark forensic sketches against corresponding photos and a 10,030 mugshot database. <s> BIB010 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> We present sketch-rnn, a recurrent neural network (RNN) able to construct stroke-based drawings of common objects. The model is trained on thousands of crude human-drawn images representing hundreds of classes. We outline a framework for conditional and unconditional sketch generation, and describe new robust training methods for generating coherent sketch drawings in a vector format. <s> BIB011 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> Human sketches are unique in being able to capture both the spatial topology of a visual object, as well as its subtle appearance details. Fine-grained sketch-based image retrieval (FG-SBIR) importantly leverages on such fine-grained characteristics of sketches to conduct instance-level retrieval of photos. Nevertheless, human sketches are often highly abstract and iconic, resulting in severe misalignments with candidate photos which in turn make subtle visual detail matching difficult. Existing FG-SBIR approaches focus only on coarse holistic matching via deep cross-domain representation learning, yet ignore explicitly accounting for fine-grained details and their spatial context. In this paper, a novel deep FG-SBIR model is proposed which differs significantly from the existing models in that: (1) It is spatially aware, achieved by introducing an attention module that is sensitive to the spatial position of visual details: (2) It combines coarse and fine semantic information via a shortcut connection fusion block: and (3) It models feature correlation and is robust to misalignments between the extracted features across the two domains by introducing a novel higher-order learnable energy function (HOLEF) based loss. Extensive experiments show that the proposed deep spatial-semantic attention model significantly outperforms the state-of-the-art. <s> BIB012 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> Face sketch to digital image matching is an important challenge of face recognition that involves matching across different domains. Current research efforts have primarily focused on extracting domain invariant representations or learning a mapping from one domain to the other. In this research, we propose a novel transform learning based approach termed as DeepTransformer, which learns a transformation and mapping function between the features of two domains. The proposed formulation is independent of the input information and can be applied with any existing learned or hand-crafted feature. Since the mapping function is directional in nature, we propose two variants of DeepTransformer: (i) semi-coupled and (ii) symmetrically-coupled deep transform learning. This research also uses a novel IIIT-D Composite Sketch with Age (CSA) variations database which contains sketch images of 150 subjects along with age-separated digital photos. The performance of the proposed models is evaluated on a novel application of sketch-to-sketch matching, along with sketch-to-digital photo matching. Experimental results demonstrate the robustness of the proposed models in comparison to existing state-of-the-art sketch matching algorithms and a commercial face recognition system. <s> BIB013 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> We propose a deep hashing framework for sketch retrieval that, for the first time, works on a multi-million scale human sketch dataset. Leveraging on this large dataset, we explore a few sketch-specific traits that were otherwise under-studied in prior literature. Instead of following the conventional sketch recognition task, we introduce the novel problem of sketch hashing retrieval which is not only more challenging, but also offers a better testbed for large-scale sketch analysis, since: (i) more fine-grained sketch feature learning is required to accommodate the large variations in style and abstraction, and (ii) a compact binary code needs to be learned at the same time to enable efficient retrieval. Key to our network design is the embedding of unique characteristics of human sketch, where (i) a two-branch CNN-RNN architecture is adapted to explore the temporal ordering of strokes, and (ii) a novel hashing loss is specifically designed to accommodate both the temporal and abstract traits of sketches. By working with a 3.8M sketch dataset, we show that state-of-the-art hashing models specifically engineered for static images fail to perform well on temporal sketch data. Our network on the other hand not only offers the best retrieval performance on various code sizes, but also yields the best generalization performance under a zero-shot setting and when re-purposed for sketch recognition. Such superior performances effectively demonstrate the benefit of our sketch-specific design. <s> BIB014 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> Human free-hand sketches have been studied in various contexts including sketch recognition, synthesis and fine-grained sketch-based image retrieval (FG-SBIR). A fundamental challenge for sketch analysis is to deal with drastically different human drawing styles, particularly in terms of abstraction level. In this work, we propose the first stroke-level sketch abstraction model based on the insight of sketch abstraction as a process of trading off between the recognizability of a sketch and the number of strokes used to draw it. Concretely, we train a model for abstract sketch generation through reinforcement learning of a stroke removal policy that learns to predict which strokes can be safely removed without affecting recognizability. We show that our abstraction model can be used for various sketch analysis tasks including: (1) modeling stroke saliency and understanding the decision of sketch recognition models, (2) synthesizing sketches of variable abstraction for a given category, or reference object instance in a photo, and (3) training a FG-SBIR model with photos only, bypassing the expensive photo-sketch pair collection step. <s> BIB015 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> Abstract The shape features of images are essential to image recognition, comparison and retrieval since most users are more interested in recognizing or comparing images by shape than by color and texture [1]. Comparing or retrieving images by shape is still envisioned as one of the most challenging works in image comparison and retrieval because of the lack of effective and efficient representations of shape features in image comparison and retrieval. In this paper, we propose a scalable and comparable shape representation, namely “Reversed Sketch”, is proposed, on which a shape feature extraction and utilization framework is built. With this representation, we represent an image object using a polygon extracted from the contour of the image object by a force-driving sliding box algorithm. A polygon evolution algorithm is then proposed for transforming the first wiggly polygon into a more sketchy form for efficiently processing, which makes our shape representation more scalable. Also, we present a comparable metric drawn from this representation combined with the comparing algorithm, which is invariant to scaling, rotation and translation and thereby is suitable for image recognition, registration and comparison. The proposed shape representation is especially suitable for image retrieval because with it a hierarchical index which is very useful for image retrieval can be built on the image dataset. Extensive experiments are carried out and the experiment results show that with our shape representation the shape features can be quickly extracted from an image, simplified as needed, and used to efficiently comparing shapes in accord with people’s perception. Experiment results derived with practical datasets indicate that our framework can achieve better comparing precision and efficiency, compared with some other methods. <s> BIB016 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> Under person re-identification (Re-ID), a query photo of the target person is often required for retrieval. However, one is not always guaranteed to have such a photo readily available under a practical forensic setting. In this paper, we define the problem of Sketch Re-ID, which instead of using a photo as input, it initiates the query process using a professional sketch of the target person. This is akin to the traditional problem of forensic facial sketch recognition, yet with the major difference that our sketches are whole-body other than just the face. This problem is challenging because sketches and photos are in two distinct domains. Specifically, a sketch is the abstract description of a person. Besides, person appearance in photos is variational due to camera viewpoint, human pose and occlusion. We address the Sketch Re-ID problem by proposing a cross-domain adversarial feature learning approach to jointly learn the identity features and domain-invariant features. We employ adversarial feature learning to filter low-level interfering features and remain high-level semantic information. We also contribute to the community the first Sketch Re-ID dataset with 200 persons, where each person has one sketch and two photos from different cameras associated. Extensive experiments have been performed on the proposed dataset and other common sketch datasets including CUFSF and QUML-shoe. Results show that the proposed method outperforms the state-of-the-arts. <s> BIB017 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> In this paper, we propose the first sketching system for interactively personalized and photorealistic face caricaturing. Input an image of a human face, the users can create caricature photos by manipulating its facial feature curves. Our system firstly performs exaggeration on the recovered 3D face model, which is conducted by assigning the laplacian of each vertex a scaling factor according to the edited sketches. The mapping between 2D sketches and the vertex-wise scaling field is constructed by a novel deep learning architecture. Our approach allows outputting different exaggerations when applying the same sketching on different input figures in term of their different geometric characteristics, which makes the generated results "personalized". With the obtained 3D caricature model, two images are generated, one obtained by applying 2D warping guided by the underlying 3D mesh deformation and the other obtained by re-rendering the deformed 3D textured model. These two images are then seamlessly integrated to produce our final output. Due to the severe stretching of meshes, the rendered texture is of blurry appearances. A deep learning approach is exploited to infer the missing details for enhancing these blurry regions. Moreover, a relighting operation is invented to further improve the photorealism of the result. These further make our results "photorealistic". The qualitative experiment results validated the efficiency of our sketching system. <s> BIB018 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> We contribute the first large-scale dataset of scene sketches, SketchyScene, with the goal of advancing research on sketch understanding at both the object and scene level. The dataset is created through a novel and carefully designed crowdsourcing pipeline, enabling users to efficiently generate large quantities of realistic and diverse scene sketches. SketchyScene contains more than 29,000 scene-level sketches, 7,000+ pairs of scene templates and photos, and 11,000+ object sketches. All objects in the scene sketches have ground-truth semantic and instance masks. The dataset is also highly scalable and extensible, easily allowing augmenting and/or changing scene composition. We demonstrate the potential impact of SketchyScene by training new computational models for semantic segmentation of scene sketches and showing how the new dataset enables several applications including image retrieval, sketch colorization, editing, and captioning, etc. The dataset and code can be found at https://github.com/SketchyScene/SketchyScene. <s> BIB019 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> Sketch-based modeling strives to bring the ease and immediacy of drawing to the 3D world. However, while drawings are easy for humans to create, they are very challenging for computers to interpret due to their sparsity and ambiguity. We propose a data-driven approach that tackles this challenge by learning to reconstruct 3D shapes from one or more drawings. At the core of our approach is a deep convolutional neural network (CNN) that predicts occupancy of a voxel grid from a line drawing. This CNN provides us with an initial 3D reconstruction as soon as the user completes a single drawing of the desired shape. We complement this single-view network with an updater CNN that refines an existing prediction given a new drawing of the shape created from a novel viewpoint. A key advantage of our approach is that we can apply the updater iteratively to fuse information from an arbitrary number of viewpoints, without requiring explicit stroke correspondences between the drawings. We train both CNNs by rendering synthetic contour drawings from hand-modeled shape collections as well as from procedurally-generated abstract shapes. Finally, we integrate our CNNs in a minimal modeling interface that allows users to seamlessly draw an object, rotate it to see its 3D reconstruction, and refine it by re-drawing from another vantage point using the 3D reconstruction as guidance. The main strengths of our approach are its robustness to freehand bitmap drawings, its ability to adapt to different object categories, and the continuum it offers between single-view and multi-view sketch-based modeling. <s> BIB020 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> In this paper, we investigate the problem of zero-shot sketch-based image retrieval (ZS-SBIR), where human sketches are used as queries to conduct retrieval of photos from unseen categories. We importantly advance prior arts by proposing a novel ZS-SBIR scenario that represents a firm step forward in its practical application. The new setting uniquely recognizes two important yet often neglected challenges of practical ZS-SBIR, (i) the large domain gap between amateur sketch and photo, and (ii) the necessity for moving towards large-scale retrieval. We first contribute to the community a novel ZS-SBIR dataset, QuickDraw-Extended, that consists of 330,000 sketches and 204,000 photos spanning across 110 categories. Highly abstract amateur human sketches are purposefully sourced to maximize the domain gap, instead of ones included in existing datasets that can often be semi-photorealistic. We then formulate a ZS-SBIR framework to jointly model sketches and photos into a common embedding space. A novel strategy to mine the mutual information among domains is specifically engineered to alleviate the domain gap. External semantic knowledge is further embedded to aid semantic transfer. We show that, rather surprisingly, retrieval performance significantly outperforms that of state-of-the-art on existing datasets that can already be achieved using a reduced version of our model. We further demonstrate the superior performance of our full model by comparing with a number of alternatives on the newly proposed dataset. The new dataset, plus all training and testing code of our model, will be publicly released to facilitate future research <s> BIB021 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> Sketches and real-world user interface examples are frequently used in multiple stages of the user interface design process. Unfortunately, finding relevant user interface examples, especially in large-scale datasets, is a highly challenging task because user interfaces have aesthetic and functional properties that are only indirectly reflected by their corresponding pixel data and meta-data. This paper introduces Swire, a sketch-based neural-network-driven technique for retrieving user interfaces. We collect the first large-scale user interface sketch dataset from the development of Swire that researchers can use to develop new sketch-based data-driven design interfaces and applications. Swire achieves high performance for querying user interfaces: for a known validation task it retrieves the most relevant example as within the top-10 results for over 60% of queries. With this technique, for the first time designers can accurately retrieve relevant user interface examples with free-form sketches natural to their design workflows. We demonstrate several novel applications driven by Swire that could greatly augment the user interface design process. <s> BIB022 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> Prototyping involves the evolution of an idea into various stages of design until it reaches a certain level of maturity. These design stages include low, medium and high fidelity prototypes. Workload analysis of prototyping using NASA-TLX showed an increase in workload specifically in frustration, temporal demand, effort, and decline in performance as the participants progressed from low to high fidelity. Upon reviewing numerous commercial and academic tools that directly or indirectly support software prototyping in one aspect or another, we identified a need for a comprehensive solution to support the entire software prototyping process. In this paper, we introduce Eve, a prototyping workbench that enables the users to sketch their concept as low fidelity prototype. It generates the consequent medium and high fidelity prototypes by means of UI element detection and code generation. We evaluated Eve using SUS with 15 UI/UX designers; the results depict good usability and high learnability (Usability score: 78.5). In future, we aim to study the impact of Eve on subjective workload experienced by users during software prototyping. <s> BIB023 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> Mid-air 3D sketching has been mainly explored in Virtual Reality (VR) and typically requires special hardware for motion capture and immersive, stereoscopic displays. The recently developed motion tracking algorithms allow real-time tracking of mobile devices, and have enabled a few mobile applications for 3D sketching in Augmented Reality (AR). However, they are more suitable for making simple drawings only, since they do not consider special challenges with mobile AR 3D sketching, including the lack of stereo display, narrow field of view, and the coupling of 2D input, 3D input and display. To address these issues, we present Mobi3DSketch, which integrates multiple sources of inputs with tools, mainly different versions of 3D snapping and planar/curves surface proxies. Our multimodal interface supports both absolute and relative drawing, allowing easy creation of 3D concept designs in situ. The effectiveness and expressiveness of Mobi3DSketch are demonstrated via a pilot study. <s> BIB024 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> Augmented Reality(AR) tools are currently primarily targeted at programmers, making designing for AR challenging and time-consuming. We developed an interactive prototype, PintAR, that enables the authoring and rapid-prototyping of situated experiences by allowing designers to bring their ideas to life using a digital pen for sketching and a head-mounted display for visualizing and interacting with virtual content. In this paper, we explore the versatility such a tool could provide through case studies of a researcher, an artist, a ballerina, and a clinician. <s> BIB025 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> We investigate the problem of stroke-level sketch segmentation, which is to automatically assign strokes of a given sketch with semantic labels. Solving the problem of sketch segmentation opens the door for fine-grained sketch interpretation, which can benefit many novel sketch-based applications, including sketch recognition and sketch-based image retrieval. In this paper, we propose an approach for multi-class sketch semantic segmentation by considering it as a sequence-to-sequence generation problem. Specifically, an end-to-end learned network SketchSegNet+, built on recurrent neural networks (RNN), is presented to translate a sequence of strokes into a sequence of semantic labels. In addition, a large-scale stroke-level sketch segmentation dataset is constructed for the first time, which is composed of 150K annotated free-hand human sketch selected from QuickDraw. The dataset will be released publicly. The experimental results of stroke-level sketch semantic segmentation on this novel dataset and the SPG dataset demonstrate the effectiveness of our approach. <s> BIB026 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> While it is trivial for humans to quickly assess the perceptual similarity between two images, the underlying mechanism are thought to be quite complex. Despite this, the most widely adopted perceptual metrics today, such as SSIM and FSIM, are simple, shallow functions, and fail to consider many factors of human perception. Recently, the facial modeling community has observed that the inclusion of both structure and texture has a significant positive benefit for face sketch synthesis (FSS). But how perceptual are these so-called “perceptual features”? Which elements are critical for their success? In this paper, we design a perceptual metric, called Structure Co-Occurrence Texture (Scoot), which simultaneously considers the block-level spatial structure and co-occurrence texture statistics. To test the quality of metrics, we propose three novel meta-measures based on various reliable properties. Extensive experiments verify that our Scoot metric exceeds the performance of prior work. Besides, we built the first largest scale (152k judgments) human-perception-based sketch database that can evaluate how well a metric consistent with human perception. Our results suggest that “spatial structure” and “co-occurrence texture” are two generally applicable perceptual features in face sketch synthesis. <s> BIB027 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> Generating sketch from colour landscape photograph is very easy while it is hard to generate colour photograph from landscape sketch. In this paper, a new automatic conversion network, named Sparse Residual Attention Generative Adversarial Networks (SRAGAN), is proposed to generate landscape colour photograph from sketch. Besides of generator adversarial loss, we not only adopt L1-regularized per-pix loss, but also combine L1-regularized perceptual loss together into our model. Due to the sparsity of L1-norm, it can preserve boundary edge information very well, which makes our model can handle well the conversion task of sketch-to-photo. In addition, we proposed a ResAttention block to our network structure, which combines the residual learning blocks with attention module. Experiments show that the landscape colour photographes generated by our SRAGAN looks more natural with bright colour and clear edge information. At the same time, we integrate two models so that we can generate winter-style and summer-style photographes from the same landscape sketch. Experiments demonstrate that our method outperforms many state-of-the-arts both in quantitative and in visual performance. <s> BIB028 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> Despite their renowned predictive power on i.i.d. data, convolutional neural networks are known to rely more on high-frequency patterns that humans deem superficial than on low-frequency patterns that agree better with intuitions about what constitutes category membership. This paper proposes a method for training robust convolutional networks by penalizing the predictive power of the local representations learned by earlier layers. Intuitively, our networks are forced to discard predictive signals such as color and texture that can be gleaned from local receptive fields and to rely instead on the global structures of the image. Across a battery of synthetic and benchmark domain adaptation tasks, our method confers improved generalization out of the domain. Also, to evaluate cross-domain transfer, we introduce ImageNet-Sketch, a new dataset consisting of sketch-like images, that matches the ImageNet classification validation set in categories and scale. <s> BIB029 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> Deep learning systems can perform well on some image recognition tasks. However, they have serious limitations, including requiring far more training data than humans do and being fooled by adversarial examples. By contrast, analogical learning over relational representations tends to be far more data-efficient, requiring only human-like amounts of training data. This paper introduces an approach that combines automatically constructed qualitative visual representations with analogical learning to tackle a hard computer vision problem, object recognition from sketches. Results from the MNIST dataset and a novel dataset, the Coloring Book Objects dataset, are provided. Comparison to existing approaches indicates that analogical generalization can be used to identify sketched objects from these datasets with several orders of magnitude fewer examples than deep learning systems require. <s> BIB030 </s> Deep Learning for Free-Hand Sketch: A Survey <s> INTRODUCTION <s> Abstract With the development of digital devices and pressure sensing equipment, research into freehand sketches from touch-screen interfaces has increased significantly in recent years. As such, we provide the first comprehensive survey of recognition tasks based on sketch generation, freehand sketch classification, sketch-based image retrieval (SBIR), fine-grained sketch-based image retrieval (FG-SBIR), and sketch-based 3D shape image retrieval. Specifically, SBIR and FG-SBIR were the main focus of the survey. Primary technologies and benchmark datasets related to all sketch-based recognition topics are also discussed, along with future trends for this promising technology. <s> BIB031
F REE-HAND sketch is a kind of vivid minimalism data formation, and easy to obtain, which has appeared before words/texts. Anybody can express his/her ideas by sketching even before he/she learned to speak and write. Anything can be depicted with the help of sketching if it is difficult to describe by words. Anywhere can allow sketching to assist communications, no matter what language is spoken there. Consequently, free-hand sketches are widely involved in human daily life from ancient times to date. Free-hand sketch is illustrative, in spite of highly concise and abstract, which can be regarded as a form of human brain's visual perception of the real world. Drawing freehand sketches is one of humans natural behavior, and sketches are useful in various scenarios, e.g., communication, design. Therefore, free-hand sketch has been studied in computer vision field BIB002 , BIB003 , BIB006 , BIB007 , BIB008 ,  , BIB014 , BIB021 and computer human interaction community BIB022 , BIB023 , BIB024 , BIB025 . In particular, early research can be traced back to the 1960s and 1970s  , BIB001 . However, free-hand sketch is essentially different to natural photo BIB002 . It stands as a special data modality/domain, having some domain-unique challenges, e.g., highly sparse, abstract, noisy, arbitrary. Moreover, free-hand sketching is also a dynamic processing, so that free-hand sketch can be understood/represented in multiple formations/modalities, e.g., static pixel space, stroke coordinate based temporal space, Euclidean space, geometrical graph space (see detailed discussion in Section 2). These intrinsic traits of free-hand sketch decide that solving sketch-related BIB002 . Image includes free-hand sketch, natural photo, etc. In this survey, natural photo denotes the image obtained by camera, e.g., the photos from ImageNet. Unless specified instructions, photo refers to natural photo thoughout this survey. problems needs to propose sketch-specific designs to overcome the sketch-specific challenges. In recent years, the prevalence of smart touchscreen devices (e.g., smart phone, tablet) has made sketch creation a much easier task than ever, and consequently made sketchoriented applications increasingly more popular. This motivates both academia and industry to study free-hand sketches more extensively. Meanwhile, deep learning techniques are developing rapidly, and have achieved the state-of-the-art in various artificial intelligence tasks. Thus, an unprecedented boom of free-hand sketch research has emerged, evidenced by the increasing publications in the top venues (as illustrated in Figure 1 ): (i) Some classic research topics (e.g., sketch recognition, sketch-based image retrieval, sketch-based 3D shape retrieval) have been restudied in deep learning manner BIB006 , BIB007 , BIB008 ,  , BIB009 and obtained significant performance improvements. (ii) Some brand new topics are proposed based on deep learning, e.g., deep learning based sketch generation/synthesis BIB011 , sketch-based deep model generation  , deep reinforcement learning based sketch abstraction BIB015 , deep sketch hashing BIB014 . (iii) Beyond the global representation based tasks (e.g., sketch recognition), more fine-grained tasks have been further studied or proposed, e.g., instance-level sketchbased image retrieval BIB008 , deep stroke-level sketch segmentation BIB026 . (iv) Compared with representing sketches as static pictures in previous, there is a significant progress on the representations for free-hand sketches in recent years. Various deep neural network architectures have been designed for sketch. Especially, thanks to the groundbreaking work SketchRNN BIB011 , the temporal traits of free-hand sketches are modeled by recurrent neural network (RNN). (v) More sketch-based applications have appeared, i.e., online sketch game QuickDraw 2 BIB011 , sketch-based commodity search engine 3 BIB008 , BIB012 . (vi) Some large-scale sketch datasets have been collected, e.g., Google QuickDraw 4 BIB011 a millionscale sketch dataset (50M+). The simultaneous prosperity of touchscreen devices and deep learning brings both opportunities and challenges to the free-hand sketch research community. In particular, there are still a lot of problems that are under-studied, although a great deal of deep learning models are proposed for free-hand sketches in recent years. This survey aims to review the state of the free-hand sketch community in deep learning era, hoping to bring insights to the related researchers. Please note that this survey focuses on freehand sketches, not including professional (forensic) facial sketch BIB010 ,  , BIB013 , BIB027 , professional pencil sketch BIB016 , human/face sketch BIB017 , professional landscape sketch BIB028 , photo-like edge-maps (artificially rendered sketch) BIB004 , BIB029 , BIB030 , cartoon/manga BIB018 , BIB019 ,  , well-drawn 3D sketch BIB020 , clip arts BIB019 . In this survey, "sketch" refers to "free-hand sketch", if without special instructions. Different to the existing sketch-related survey papers BIB005 ,  , BIB031 that focus on free-hand sketch based recognition and image/3D retrieval, this survey aims to introduce the state of free-hand sketch community in deep learning era by the detailed discussion and taxonomy for all the representative sketch-related research topics. This survey are organized as follows. Section 2 provides some background knowledge of free-hand sketches, including intrinsic traits, domain-unique challenges, milestone techniques of the existing sketch-oriented deep learning, etc. Section 3 summarizes the representative free-hand sketch datasets. In Section 4, we provide a comprehensive taxonomy for sketch-based tasks, and the representative deep learning models will be described in detail. The bottlenecks and some open problems of the free-hand sketch community in deep learning era are discussed in Section 5 and Section 6, respectively. Section 7 gives some promising and BIB006 . http://sketchx.eecs.qmul.ac.uk/demos/ 4. https://github.com/googlecreativelab/quickdraw-dataset function mapping or feature extraction F Θ (·) neural network feature extraction, parameterized by Θ D(·) distance metric, e.g., 2 distance λ weighting factor summation element-wise product σ(·) sigmoid activation function ReLu(·) ReLU activation function R set of real numbers T time step t time step α, β, γ hyper parameters by manually setting interesting topics for future work. The conclusion of this survey is presented in Section 8. Throughout this survey, bold uppercase characters and bold lowercase characters denote matrices and vectors, respectively. Unless specified instructions, the mathematical symbols follow the conventions in Table 1 .