Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> This paper describes a computational approach to edge detection. The success of the approach depends on the definition of a comprehensive set of goals for the computation of edge points. These goals must be precise enough to delimit the desired behavior of the detector while making minimal assumptions about the form of the solution. We define detection and localization criteria for a class of edges, and present mathematical forms for these criteria as functionals on the operator impulse response. A third criterion is then added to ensure that the detector has only one response to a single edge. We use the criteria in numerical optimization to derive detectors for several common image features, including step edges. On specializing the analysis to step edges, we find that there is a natural uncertainty principle between detection and localization performance, which are the two main goals. With this principle we derive a single operator shape which is optimal at any scale. The optimal detector has a simple approximate implementation in which edges are marked at maxima in gradient magnitude of a Gaussian-smoothed image. We extend this simple detector using operators of several widths to cope with different signal-to-noise ratios in the image. We present a general method, called feature synthesis, for the fine-to-coarse integration of information from operators at different scales. Finally we show that step edge detector performance improves considerably as the operator point spread function is extended along the edge. <s> BIB001 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> We present a method for training a similarity metric from data. The method can be used for recognition or verification applications where the number of categories is very large and not known during training, and where the number of training samples for a single category is very small. The idea is to learn a function that maps input patterns into a target space such that the L/sub 1/ norm in the target space approximates the "semantic" distance in the input space. The method is applied to a face verification task. The learning process minimizes a discriminative loss function that drives the similarity metric to be small for pairs of faces from the same person, and large for pairs from different persons. The mapping from raw to the target space is a convolutional network whose architecture is designed for robustness to geometric distortions. The system is tested on the Purdue/AR face database which has a very high degree of variability in the pose, lighting, expression, position, and artificial occlusions such as dark glasses and obscuring scarves. <s> BIB002 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> We introduce Deep Canonical Correlation Analysis (DCCA), a method to learn complex nonlinear transformations of two views of data such that the resulting representations are highly linearly correlated. Parameters of both transformations are jointly learned to maximize the (regularized) total correlation. It can be viewed as a nonlinear extension of the linear method canonical correlation analysis (CCA). It is an alternative to the nonparametric method kernel canonical correlation analysis (KCCA) for learning correlated nonlinear transformations. Unlike KCCA, DCCA does not require an inner product, and has the advantages of a parametric method: training time scales well with data size and the training data need not be referenced when computing the representations of unseen instances. In experiments on two real-world datasets, we find that DCCA learns representations with significantly higher correlation than those learned by CCA and KCCA. We also introduce a novel non-saturating sigmoid function based on the cube root that may be useful more generally in feedforward neural networks. <s> BIB003 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> The use of object proposals is an effective recent approach for increasing the computational efficiency of object detection. We propose a novel method for generating object bounding box proposals using edges. Edges provide a sparse yet informative representation of an image. Our main observation is that the number of contours that are wholly contained in a bounding box is indicative of the likelihood of the box containing an object. We propose a simple box objectness score that measures the number of edges that exist in the box minus those that are members of contours that overlap the box’s boundary. Using efficient data structures, millions of candidate boxes can be evaluated in a fraction of a second, returning a ranked set of a few thousand top-scoring proposals. Using standard metrics, we show results that are significantly more accurate than the current state-of-the-art while being faster to compute. In particular, given just 1000 proposals we achieve over 96% object recall at overlap threshold of 0.5 and over 75% recall at the more challenging overlap of 0.7. Our approach runs in 0.25 seconds and we additionally demonstrate a near real-time variant with only minor loss in accuracy. <s> BIB004 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> We develop a new edge detection algorithm that addresses two critical issues in this long-standing vision problem: (1) holistic image training, and (2) multi-scale feature learning. Our proposed method, holistically-nested edge detection (HED), turns pixel-wise edge classification into image-to-image prediction by means of a deep learning model that leverages fully convolutional neural networks and deeply-supervised nets. HED automatically learns rich hierarchical representations (guided by deep supervision on side responses) that are crucially important in order to approach the human ability to resolve the challenging ambiguity in edge and object boundary detection. We significantly advance the state-of-the-art on the BSD500 dataset (ODS F-score of 0.782) and the NYU Depth dataset (ODS F-score of 0.746), and do so with an improved speed (0.4 second per image) that is orders of magnitude faster than recent CNN-based edge detection algorithms. <s> BIB005 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> Despite significant recent advances in the field of face recognition [10, 14, 15, 17], implementing face verification and recognition efficiently at scale presents serious challenges to current approaches. In this paper we present a system, called FaceNet, that directly learns a mapping from face images to a compact Euclidean space where distances directly correspond to a measure of face similarity. Once this space has been produced, tasks such as face recognition, verification and clustering can be easily implemented using standard techniques with FaceNet embeddings as feature vectors. <s> BIB006 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> In recent years, supervised learning with convolutional networks (CNNs) has seen huge adoption in computer vision applications. Comparatively, unsupervised learning with CNNs has received less attention. In this work we hope to help bridge the gap between the success of CNNs for supervised learning and unsupervised learning. We introduce a class of CNNs called deep convolutional generative adversarial networks (DCGANs), that have certain architectural constraints, and demonstrate that they are a strong candidate for unsupervised learning. Training on various image datasets, we show convincing evidence that our deep convolutional adversarial pair learns a hierarchy of representations from object parts to scenes in both the generator and discriminator. Additionally, we use the learned features for novel tasks - demonstrating their applicability as general image representations. <s> BIB007 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> We investigate the problem of fine-grained sketch-based image retrieval (SBIR), where free-hand human sketches are used as queries to perform instance-level retrieval of images. This is an extremely challenging task because (i) visual comparisons not only need to be fine-grained but also executed cross-domain, (ii) free-hand (finger) sketches are highly abstract, making fine-grained matching harder, and most importantly (iii) annotated cross-domain sketch-photo datasets required for training are scarce, challenging many state-of-the-art machine learning techniques. In this paper, for the first time, we address all these challenges, providing a step towards the capabilities that would underpin a commercial sketch-based image retrieval application. We introduce a new database of 1,432 sketchphoto pairs from two categories with 32,000 fine-grained triplet ranking annotations. We then develop a deep tripletranking model for instance-level SBIR with a novel data augmentation and staged pre-training strategy to alleviate the issue of insufficient fine-grained training data. Extensive experiments are carried out to contribute a variety of insights into the challenges of data sufficiency and over-fitting avoidance when training deep networks for finegrained cross-domain ranking tasks. <s> BIB008 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> We present the Sketchy database, the first large-scale collection of sketch-photo pairs. We ask crowd workers to sketch particular photographic objects sampled from 125 categories and acquire 75,471 sketches of 12,500 objects. The Sketchy database gives us fine-grained associations between particular photos and sketches, and we use this to train cross-domain convolutional networks which embed sketches and photographs in a common feature space. We use our database as a benchmark for fine-grained retrieval and show that our learned representation significantly outperforms both hand-crafted features as well as deep features trained for sketch or photo classification. Beyond image retrieval, we believe the Sketchy database opens up new opportunities for sketch and image understanding and synthesis. <s> BIB009 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> Fine-grained sketch-based image retrieval (FG-SBIR) is a newly emerged topic in computer vision. The problem is challenging because in addition to bridging the sketch-photo domain gap, it also asks for instance-level discrimination within object categories. Most prior approaches focused on feature engineering and fine-grained ranking, yet neglected an important and central problem: how to establish a fine-grained cross-domain feature space to conduct retrieval. In this paper, for the first time we formulate a cross-domain framework specifically designed for the task of FG-SBIR that simultaneously conducts instance-level retrieval and attribute prediction. Different to conventional photo-text cross-domain frameworks that performs transfer on category-level data, our joint multi-view space uniquely learns from the instance-level pair-wise annotations of sketch and photo. More specifically, we propose a joint view selection and attribute subspace learning algorithm to learn domain projection matrices for photo and sketch, respectively. It follows that visual attributes can be extracted from such matrices through projection to build a coupled semantic space to conduct retrieval. Experimental results on two recently released fine-grained photo-sketch datasets show that the proposed method is able to perform at a level close to those of deep models, while removing the need for extensive manual annotations. <s> BIB010 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> Sketch-based image retrieval (SBIR) is a challenging task due to the ambiguity inherent in sketches when compared with photos. In this paper, we propose a novel convolutional neural network based on Siamese network for SBIR. The main idea is to pull output feature vectors closer for input sketch-image pairs that are labeled as similar, and push them away if irrelevant. This is achieved by jointly tuning two convolutional neural networks which linked by one loss function. Experimental results on Flickr15K demonstrate that the proposed method offers a better performance when compared with several state-of-the-art approaches. <s> BIB011 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> Fine-grained sketch-based image retrieval (SBIR) aims to go beyond conventional SBIR to perform instance-level cross-domain retrieval: finding the specific photo that matches an input sketch. Existing methods focus on designing/learning good features for cross-domain matching and/or learning cross-domain matching functions. However, they neglect the semantic aspect of retrieval, i.e., what meaningful object properties does a user try encode in her/his sketch? We propose a fine-grained SBIR model that exploits semantic attributes and deep feature learning in a complementary way. Specifically, we perform multi-task deep learning with three objectives, including: retrieval by fine-grained ranking on a learned representation, attribute prediction, and attribute-level ranking. Simultaneously predicting semantic attributes and using such predictions in the ranking procedure help retrieval results to be more semantically relevant. Importantly, the introduction of semantic attribute learning in the model allows for the elimination of the otherwise prohibitive cost of human annotations required for training a fine-grained deep ranking model. Experimental results demonstrate that our method outperforms the state-of-the-art on challenging fine-grained SBIR benchmarks while requiring less annotation. <s> BIB012 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> Sketch-based object search is a challenging problem mainly due to two difficulties: (1) how to match the binary sketch query with the colorful image, and (2) how to locate the small object in a big image with the sketch query. To address the above challenges, we propose to leverage object proposals for object search and localization. However, instead of purely relying on sketch features, e.g., Sketch-a-Net, to locate the candidate object proposals, we propose to fully utilize the appearance information to resolve the ambiguities among object proposals and refine the search results. Our proposed query adaptive search is formulated as a sub-graph selection problem, which can be solved by maximum flow algorithm. By performing query expansion using a smaller set of more salient matches as the query representatives, it can accurately locate the small target objects in cluttered background or densely drawn deformation intensive cartoon (Manga like) images. Our query adaptive sketch based object search on benchmark datasets exhibits superior performance when compared with existing methods, which validates the advantages of utilizing both the shape and appearance features for sketch-based search. <s> BIB013 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> We propose and evaluate several triplet CNN architectures for measuring the similarity between sketches and photographs, within the context of the sketch based image retrieval (SBIR) task. In contrast to recent fine-grained SBIR work, we study the ability of our networks to generalise across diverse object categories from limited training data, and explore in detail strategies for weight sharing, pre-processing, data augmentation and dimensionality reduction. We exceed the performance of pre-existing techniques on both the Flickr15k category level SBIR benchmark by $18\%$, and the TU-Berlin SBIR benchmark by $\sim10 \mathcal{T}_b$, when trained on the 250 category TU-Berlin classification dataset augmented with 25k corresponding photographs harvested from the Internet. <s> BIB014 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> Many vision problems require matching images of object instances across different domains. These include fine-grained sketch-based image retrieval (FG-SBIR) and Person Re-identification (person ReID). Existing approaches attempt to learn a joint embedding space where images from different domains can be directly compared. In most cases, this space is defined by the output of the final layer of a deep neural network (DNN), which primarily contains features of a high semantic level. In this paper, we argue that both high and mid-level features are relevant for cross-domain instance matching (CDIM). Importantly, mid-level features already exist in earlier layers of the DNN. They just need to be extracted, represented, and fused properly with the final layer. Based on this simple but powerful idea, we propose a unified framework for CDIM. Instantiating our framework for FG-SBIR and ReID, we show that our simple models can easily beat the state-of-the-art models, which are often equipped with much more elaborate architectures. <s> BIB015 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> We propose a novel measure of visual similarity for image retrieval that incorporates both structural and aesthetic (style) constraints. Our algorithm accepts a query as sketched shape, and a set of one or more contextual images specifying the desired visual aesthetic. A triplet network is used to learn a feature embedding capable of measuring style similarity independent of structure, delivering significant gains over previous networks for style discrimination. We incorporate this model within a hierarchical triplet network to unify and learn a joint space from two discriminatively trained streams for style and structure. We demonstrate that this space enables, for the first time, styleconstrained sketch search over a diverse domain of digital artwork comprising graphics, paintings and drawings. We also briefly explore alternative query modalities. <s> BIB016 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> Fine-grained Sketch-based Image Retrieval (Fine-grained SBIR), which uses hand-drawn sketches to search the target object images, has been an emerging topic over the last few years. The difficulties of this task not only come from the ambiguous and abstract characteristics of sketches with less useful information, but also the cross-modal gap at both visual and semantic level. However, images on the web are always exhibited with multimodal contents. In this paper, we consider Fine-grained SBIR as a cross-modal retrieval problem and propose a deep multimodal embedding model that exploits all the beneficial multimodal information sources in sketches and images. In our experiment with large quantity of public data, we show that the proposed method outperforms the state-of-the-art methods for Fine-grained SBIR. <s> BIB017 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> A novel scheme with deep cross-modal correlation learning is developed in this paper to facilitate more effective Sketch-based Image Retrieval (SBIR) for large-scale annotated images. It integrates the deep multimodal feature generation, deep cross-modal correlation learning and similarity search optimization through mining all the beneficial multimodal information sources in sketches and images, which can be treated as an inter-related correlation distribution over deep representations of sketches and images. Very positive results were obtained in our experiments using a large quantity of public data. <s> BIB018 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> Freehand sketches are a simple and powerful tool for communication. They are easily recognized across cultures and suitable for various applications. In this paper, we use deep convolutional neural networks (ConvNets) to address sketch-based image retrieval (SBIR). We first train our ConvNets on sketch and image object recognition in a large scale benchmark for SBIR (the sketchy database). We then conduct a comprehensive study of ConvNets features for SBIR, using a kNN similarity search paradigm in the ConvNet feature space. In contrast to recent SBIR works, we propose a new architecture the quadruplet networks which enhance ConvNet features for SBIR. This new architecture enables ConvNets to extract more robust global and local features. We evaluate our approach on three large scale datasets. Our quadruplet networks outperform previous state-of-the-art on two of them by a significant margin and gives competitive results on the third. Our system achieves a recall of 42.16% (at k=1) for the sketchy database (more than 5% improvement), a Kendal score of 43.28Τb on the TU-Berlin SBIR benchmark (close to 6Τb improvement) and a mean average precision (MAP) of 32.16% on Flickr15k (a category level SBIR benchmark). <s> BIB019 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> Existing methods on sketch based image retrieval (SBIR) are usually based on the hand-crafted features whose ability of representation is limited. In this paper, we propose a sketch based image retrieval method via image-aided cross domain learning. First, the deep learning model is introduced to learn the discriminative features. However, it needs a large number of images to train the deep model, which is not suitable for the sketch images. Thus, we propose to extend the sketch training images via introducing the real images. Specifically, we initialize the deep models with extra image data, and then extract the generalized boundary from real images as the sketch approximation. The using of generalized boundary is under the assumption that their domain is similar with sketch domain. Finally, the neural network is fine-tuned with the sketch approximation data. Experimental results on Flicker15 show that the proposed method has a strong ability to link the associated image-sketch pairs and the results outperform state-of-the-arts methods. <s> BIB020 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> Abstract We present an efficient representation for sketch based image retrieval (SBIR) derived from a triplet loss convolutional neural network (CNN). We treat SBIR as a cross-domain modelling problem, in which a depiction invariant embedding of sketch and photo data is learned by regression over a siamese CNN architecture with half-shared weights and modified triplet loss function. Uniquely, we demonstrate the ability of our learned image descriptor to generalise beyond the categories of object present in our training data, forming a basis for general cross-category SBIR. We explore appropriate strategies for training, and for deriving a compact image descriptor from the learned representation suitable for indexing data on resource constrained e. g. mobile devices. We show the learned descriptors to outperform state of the art SBIR on the defacto standard Flickr15k dataset using a significantly more compact (56 bits per image, i. e. ≈ 105KB total) search index than previous methods. Datasets and models are available from the CVSSP datasets server at www.cvssp.org . <s> BIB021 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> The development of image diversity has led multiple fields' application of cross-domain image retrieval. In this paper, we propose a heterogeneous dual network (two different networks) based on sketches and images. End-to-end cross-domain image retrieval is realized by limiting the similarity of features extracted from the two networks through the method of combining the contrastive loss with the triplet ranking loss. We also study how the order of drawings affects the sketch retrieval. Compared to the Siamese network, we avoid edge extraction for preprocessing, and the effect of fine-grained retrieval is further enhanced. <s> BIB022 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> For sketch-based image retrieval (SBIR), we propose a generative adversarial network trained on a large number of sketches and their corresponding real images. To imitate human search process, we attempt to match candidate images with theimaginary image in user single s mind instead of the sketch query, i.e., not only the shape information of sketches but their possible content information are considered in SBIR. Specifically, a conditional generative adversarial network (cGAN) is employed to enrich the content information of sketches and recover the imaginary images, and two VGG-based encoders, which work on real and imaginary images respectively, are used to constrain their perceptual consistency from the view of feature representations. During SBIR, we first generate an imaginary image from a given sketch via cGAN, and then take the output of the learned encoder for imaginary images as the feature of the query sketch. Finally, we build an interactive SBIR system that shows encouraging performance. <s> BIB023 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> In this work we introduce a cross modal image retrieval system that allows both text and sketch as input modalities for the query. A cross-modal deep network architecture is formulated to jointly model the sketch and text input modalities as well as the the image output modality, learning a common embedding between text and images and between sketches and images. In addition, an attention model is used to selectively focus the attention on the different objects of the image, allowing for retrieval with multiple objects in the query. Experiments show that the proposed method performs the best in both single and multiple object image retrieval in standard datasets. <s> BIB024 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> Sketch-based object search is a challenging problem mainly due to three difficulties: 1) how to match the primary sketch query with the colorful image; 2) how to locate the small object in a big image that is similar to the sketch query; and 3) given the large image database, how to ensure an efficient search scheme that is reasonably scalable. To address the above challenges, we propose leveraging object proposals for object search and localization. However, instead of purely relying on sketch features, we propose fully utilizing the appearance features of object proposals to resolve the ambiguities between the matching sketch query and object proposals. Our proposed query adaptive search is formulated as a subgraph selection problem, which can be solved by the maximum flow algorithm. By performing query expansion, it can accurately locate the small target objects in a cluttered background or densely drawn deformation-intensive cartoon (Manga like) images. To improve the computing efficiency of matching proposal candidates, the proposed Multi View Spatially Constrained Proposal Selection encodes each identified object proposal in terms of a small local basis of anchor objects. The results on benchmark datasets validate the advantages of utilizing both the sketch and appearance features for sketch-based search, while ensuring sufficient scalability at the same time. <s> BIB025 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> Abstract Sketch-based Image Retrieval (SBIR) has received a lot of attentions recently. In this paper we aim to enhance SBIR with deep visual semantic descriptor and related optimization mechanisms. Our scheme significantly differs from other earlier work in: 1) A feature representation via deep visual semantic descriptor is established to bridge the gap between sketches and images, which can encode both low-level local features and high-level semantic features; 2) A clustering-based re-ranking optimization is introduced to further improve SBIR by dynamically adjusting the correlations of images in the ranking list. The main contribution of our work is that we effectively apply the deep visual semantic descriptor to enable deep sketch-image matching, which has provided a more reasonable base for us to fuse local low-level visual features with high-level semantic features by determining an optimal correlated mapping. Our experiments on a large number of public data have obtained very positive results. <s> BIB026 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> Fine-grained sketch-based image retrieval (FG-SBIR) addresses matching specific photo instance using free-hand sketch as a query modality. Existing models aim to learn an embedding space in which sketch and photo can be directly compared. While successful, they require instance-level pairing within each coarse-grained category as annotated training data. Since the learned embedding space is domain-specific, these models do not generalise well across categories. This limits the practical applicability of FG-SBIR. In this paper, we identify cross-category generalisation for FG-SBIR as a domain generalisation problem, and propose the first solution. Our key contribution is a novel unsupervised learning approach to model a universal manifold of prototypical visual sketch traits. This manifold can then be used to paramaterise the learning of a sketch/photo representation. Model adaptation to novel categories then becomes automatic via embedding the novel sketch in the manifold and updating the representation and retrieval function accordingly. Experiments on the two largest FG-SBIR datasets, Sketchy and QMUL-Shoe-V2, demonstrate the efficacy of our approach in enabling cross-category generalisation of FG-SBIR. <s> BIB027 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> We propose a novel semi-heterogeneous network for sketch-based image retrieval (SBIR). By exploring different feature-extracting network structures and data augmentation algorithms, we design a high-performance deep-network based method for SBIR. Our work consists of three stages: 1) we propose a novel deep SBIR optimization model, termed a semi-heterogeneous network, to capture the cross-view similarities between different categories; 2) we develop a novel feature extraction method to find a cross-domain representation which contains the line information of sketches while retaining the original information of images; 3) we explore a more comprehensive structure though different parameter and network settings for further performance improvement. Based on our experiments on a widely used dataset, our approach significantly outperforms state-of-the-art methods. <s> BIB028 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> LiveSketch is a novel algorithm for searching large image collections using hand-sketched queries. LiveSketch tackles the inherent ambiguity of sketch search by creating visual suggestions that augment the query as it is drawn, making query specification an iterative rather than one-shot process that helps disambiguate users' search intent. Our technical contributions are: a triplet convnet architecture that incorporates an RNN based variational autoencoder to search for images using vector (stroke-based) queries; real-time clustering to identify likely search intents (and so, targets within the search embedding); and the use of backpropagation from those targets to perturb the input stroke sequence, so suggesting alterations to the query in order to guide the search. We show improvements in accuracy and time-to-task over contemporary baselines using a 67M image corpus. <s> BIB029 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> Sketch has been employed as an effective communication tool to express the abstract and intuitive meaning of object. While content-based sketch recognition has been studied for several decades, the instance-level Sketch Based Image Retrieval (iSBIR) task has attracted significant research attention recently. In many previous iSBIR works -- TripletSN, and DSSA, edge maps were employed as intermediate representations in bridging the cross-domain discrepancy between photos and sketches. However, it is nontrivial to efficiently train and effectively use the edge maps in an iSBIR system. Particularly, we find that such an edge map based iSBIR system has several major limitations. First, the system has to be pre-trained on a significant amount of edge maps, either from large-scale sketch datasets, e.g., TU-Berlin~\citeeitz2012hdhso, or converted from other large-scale image datasets, e.g., ImageNet-1K\citedeng2009imagenet dataset. Second, the performance of such an iSBIR system is very sensitive to the quality of edge maps. Third and empirically, the multi-cropping strategy is essentially very important in improving the performance of previous iSBIR systems. To address these limitations, this paper advocates an end-to-end iSBIR system without using the edge maps. Specifically, we present a Triplet Classification Network (TC-Net) for iSBIR which is composed of two major components: triplet Siamese network, and auxiliary classification loss. Our TC-Net can break the limitations existed in previous works. Extensive experiments on several datasets validate the efficacy of the proposed network and system. <s> BIB030 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Category and Instance Level SBIR: <s> Abstract Fine-grained Sketch-based Image Retrieval (FG-SBIR), which utilizes hand-drawn sketches to search the target object images, has recently drawn much attention. It is a challenging task because sketches and images belong to different modalities and sketches are highly abstract and ambiguous. Existing solutions to this problem either focus on visual comparisons between sketches and images and ignore the multimodal characteristics of annotated images, or treat the retrieval as a one-time process. In this paper, we formulate FG-SBIR as a coarse-to-fine process, and propose a Deep Cascaded Cross-modal Ranking Model (DCCRM) that can exploit all the beneficial multimodal information in sketches and annotated images and improve both the retrieval efficiency and the top-K ranked effectiveness. Our goal concentrates on constructing deep representations for sketches, images, and descriptions, and learning the optimized deep correlations across such different domains. Thus for a given query sketch, its relevant images with fine-grained instance-level similarities in a specific category can be returned, and the strict requirement of the instance-level retrieval for FG-SBIR is satisfied. Very positive results have been obtained in our experiments by using a large quantity of public data. <s> BIB031
Sketch-based image retrieval (SBIR) has been widely studied BIB008 , BIB009 , BIB002 , BIB010 , BIB011 , BIB012 , BIB015 , BIB016 ,  , BIB017 , BIB024 , BIB031 , BIB027 in recent years. The existing SBIR solutions mainly include (i) discriminative methods and (ii) discriminative-generative hybrid methods. The discriminative methods mainly contains: (a) contrastive comparison based methods (implemented by pair-wise loss),(b) ranking based methods BIB008 , BIB009 , (c) deep canonical correlation analysis (DCCA) BIB003 based methods BIB018 , (d) cross-domain dictionary learning  , etc. The most widely-studied method is ranking based methods, including triplet ranking BIB015 , BIB028 , BIB029 , quadruplet ranking BIB019 . The common motivation of the existing discriminative SBIR methods is using the discriminative loss terms to learn a joint embedding space where sketches and photos can be compared. Some previous SBIR works BIB013 , BIB020 , BIB025 use edge maps (image contours) as sketch approximation to conduct the cross-modal matching. Canny edge detector BIB001 , Edge Boxes toolbox BIB004 , and holistically-nested edge detection (HED) BIB005 are usually used to extract the edges from natural photos. However, this kind of process does not make full use of the end-to-end feature extraction ability of deep neural network. Some previous SBIR methods work in two-stage pipeline, where the initial retrieval combines a postprocessing re-ranking BIB013 , BIB025 , BIB026 . Actually, re-ranking is able to refine the initial learned embedding spaces. The following will introduce the triplet-ranking and quadruplet-ranking SBIR methods in detail. As shown in Figure 11 , given a sketch anchor X n and its positive and negative candidates (X n,+ , X n,− ), the goal of triplet ranking is D(F(X n ), F(X n,+ )) < D(F(X n ), F(X n,− )). where D(·, ·) is a distance metric (e.g., 2 distance). In common practice BIB008 , BIB015 , negative sample is usually selected from the identical class of the anchor. Specifically, the loss function of triplet ranking goes as where ∆ is the margin to guarantee the minimum distance between the embedding pairs of {F(X n ), F(X n,+ )} and {F(X n ), F(X n,− )}. For quadruplet ranking BIB019 , the input atom is a quadruplet of anchor X n , positive candidate X n,+ , negative candidate X n,− from the class of anchor, negative candidate X n,−− from a different class to anchor. As illustrated in Figure 11 , the goal of quadruplet ranking is to ensure Based on this, quadruplet ranking is essentially multitask or multiple triplet ranking by constructing two extra triplet relationships, so that encode more semantic information into the embedding space . In particular, Seddati et al. BIB019 construct three triplets from each quadruplet, including triplet a = {X n , X n,+ , X n,− }, triplet b = {X n , X n,+ , X n,−− } , and triplet c = {X n , X n,− , X n,−− }. Therefore, the quadruplet ranking loss is defined as where λ b , λ c are the weights. Generally, for SBIR tasks, in each triplet or quadruplet, the anchor is from sketch domain, and other components are from photo domain. Both of triplet and quadruplet ranking can be widely used for both category-level and instancelevel SBIR tasks. As shown in Figure 12 , for both triplet and quadruplet ranking based SBIR models, the backbone network architecture can adopt (i) Siamese network, (ii) semi-heterogeneous, and (iii) heterogeneous networks: (i) Siamese networks BIB008 have the fully-shared weight/parameter sharing manners. (ii) Semi-heterogeneous network BIB021 ,  is in the halfshared manner, where the cross-modal shared layers are located in the high levels of the networks. Thus, semiheterogeneous networks also can be treated as a kind of late-fusion architecture. (iii) Generally, for heterogeneous networks BIB022 , sketch anchor will have a independent subnetwork, and the positive and negative photos will share another sub-network. Various weigh sharing manners are discussed in BIB014 . Furthermore, triplet or quadruplet ranking based SBIR models can be improved by multi-task training, via combining with classification BIB009 , BIB014 , BIB030 . Comments: The essential principle of triplet loss is using the local partial orderings to establish the global ordered relationship in the embedding space, so that triplet ranking BIB006 can be understood as Topological Sorting  . The triplet annotations work as a partially ordered set. Compared with other loss functions, the main advantages of triplet loss on sketch-related retrieval are: (i) It helps to involve more locally partial orderings/annotations to learn more fine-grained embedding space. (ii) Given limited training samples of N , their triplet orderings have C 3 N combinations, producing significant data augmentations. This is benefit to train deep neural networks on small scale sketch datasets. It should be noted that the performance of triplet loss is heavily dependent on (i) choice of margin parameter and (ii) the triplet construction strategies. Pang et al.  propose a novel discriminativegenerative hybrid model to solve FG-SBIR, which is also intrinsically a multi-task learning network, involving discriminative task and generative task. In particular, as shown in Figure 13 , this hybrid model uses a Siamese encoderdecoder architecture with a classification subnet, taking triplets as input. Thus, there are totally three subnets in this model, i.e., encoder subnet, decoder subnet, classification subnet. Given a triplet input, the outputs of encoder subnet will be used for two discriminative tasks, i.e., triplet ranking, object category-level classification. The output of decoder subnet will be used for generative task. Therefore, its loss function has three terms: where λ cla. , λ rec. are the weights, and "cla." and "rec." are short for "classification" and "reconstruction", respectively. Specifically, in (6), L triplet , L cla. , and L rec. are triplet, cross-entropy softmax, and auto-encoder reconstruction loss terms, respectively. Sharing some similar motivations with  , generative adversarial network based SBIR models also have been studied. For instance, Guo et al. BIB023 propose a SBIR model based on conditional generative adversarial network (cGAN) BIB007 .