Deep Learning for Free-Hand Sketch: A Survey <s> Sketch-Photo Retrieval <s> We present an image retrieval system for the interactive search of photo collections using free-hand sketches depicting shape. We describe Gradient Field HOG (GF-HOG); an adapted form of the HOG descriptor suitable for Sketch Based Image Retrieval (SBIR). We incorporate GF-HOG into a Bag of Visual Words (BoVW) retrieval framework, and demonstrate how this combination may be harnessed both for robust SBIR, and for localizing sketched objects within an image. We evaluate over a large Flickr sourced dataset comprising 33 shape categories, using queries from 10 non-expert sketchers. We compare GF-HOG against state-of-the-art descriptors with common distance measures and language models for image retrieval, and explore how affine deformation of the sketch impacts search performance. GF-HOG is shown to consistently outperform retrieval versus SIFT, multi-resolution HOG, Self Similarity, Shape Context and Structure Tensor. Further, we incorporate semantic keywords into our GF-HOG system to enable the use of annotated sketches for image search. A novel graph-based measure of semantic similarity is proposed and two applications explored: semantic sketch based image retrieval and a semantic photo montage. <s> BIB001 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch-Photo Retrieval <s> Although many online shops allow users to search for clothes by categories or keywords, it is usually impossible to specify the details of the design. This paper presents a new technology for retrieving skirt images based on sketches. We first conducted a user study to investigate the typical features illustrated in a sketch. Then algorithms have been developed for automatically extracting those features from both the skirt images and the sketches. A prototype system has been implemented to retrieve and present the best matched skirts in real time when a user interactively sketches her imagined skirt on the canvas. <s> BIB002 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch-Photo Retrieval <s> We investigate the problem of fine-grained sketch-based image retrieval (SBIR), where free-hand human sketches are used as queries to perform instance-level retrieval of images. This is an extremely challenging task because (i) visual comparisons not only need to be fine-grained but also executed cross-domain, (ii) free-hand (finger) sketches are highly abstract, making fine-grained matching harder, and most importantly (iii) annotated cross-domain sketch-photo datasets required for training are scarce, challenging many state-of-the-art machine learning techniques. In this paper, for the first time, we address all these challenges, providing a step towards the capabilities that would underpin a commercial sketch-based image retrieval application. We introduce a new database of 1,432 sketchphoto pairs from two categories with 32,000 fine-grained triplet ranking annotations. We then develop a deep tripletranking model for instance-level SBIR with a novel data augmentation and staged pre-training strategy to alleviate the issue of insufficient fine-grained training data. Extensive experiments are carried out to contribute a variety of insights into the challenges of data sufficiency and over-fitting avoidance when training deep networks for finegrained cross-domain ranking tasks. <s> BIB003 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch-Photo Retrieval <s> Sketch-based image retrieval (SBIR) has become a prominent research topic in recent years due to the proliferation of touch screens. The problem is however very challenging for that photos and sketches are inherently modeled in different modalities. Photos are accurate (colored and textured) depictions of the real-world, whereas sketches are highly abstract (black and white) renderings often drawn from human memory. This naturally motivates us to study the effectiveness of various cross-modal retrieval methods in SBIR. However, to the best of our knowledge, all established cross-modal algorithms are designed to traverse the more conventional cross-modal gap of image and text, making their general applicableness to SBIR unclear. In this paper, we design a series of experiments to clearly illustrate circumstances under which cross-modal methods can be best utilized to solve the SBIR problem. More specifically, we choose six state-of-the-art cross-modal subspace learning approaches that were shown to work well on image-text and conduct extensive experiments on a recently released SBIR dataset. Finally, we present detailed comparative analysis of the experimental results and offer insights to benefit future research. <s> BIB004 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch-Photo Retrieval <s> Sketch-based image retrieval (SBIR) is challenging due to the inherent domain-gap between sketch and photo. Compared with pixel-perfect depictions of photos, sketches are iconic renderings of the real world with highly abstract. Therefore, matching sketch and photo directly using low-level visual clues are unsufficient, since a common low-level subspace that traverses semantically across the two modalities is non-trivial to establish. Most existing SBIR studies do not directly tackle this cross-modal problem. This naturally motivates us to explore the effectiveness of cross-modal retrieval methods in SBIR, which have been applied in the image-text matching successfully. In this paper, we introduce and compare a series of state-of-the-art cross-modal subspace learning methods and benchmark them on two recently released fine-grained SBIR datasets. Through thorough examination of the experimental results, we have demonstrated that the subspace learning can effectively model the sketch-photo domain-gap. In addition we draw a few key insights to drive future research. <s> BIB005 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch-Photo Retrieval <s> Human sketches are unique in being able to capture both the spatial topology of a visual object, as well as its subtle appearance details. Fine-grained sketch-based image retrieval (FG-SBIR) importantly leverages on such fine-grained characteristics of sketches to conduct instance-level retrieval of photos. Nevertheless, human sketches are often highly abstract and iconic, resulting in severe misalignments with candidate photos which in turn make subtle visual detail matching difficult. Existing FG-SBIR approaches focus only on coarse holistic matching via deep cross-domain representation learning, yet ignore explicitly accounting for fine-grained details and their spatial context. In this paper, a novel deep FG-SBIR model is proposed which differs significantly from the existing models in that: (1) It is spatially aware, achieved by introducing an attention module that is sensitive to the spatial position of visual details: (2) It combines coarse and fine semantic information via a shortcut connection fusion block: and (3) It models feature correlation and is robust to misalignments between the extracted features across the two domains by introducing a novel higher-order learnable energy function (HOLEF) based loss. Extensive experiments show that the proposed deep spatial-semantic attention model significantly outperforms the state-of-the-art. <s> BIB006 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch-Photo Retrieval <s> We present an algorithm for visually searching image collections using free-hand sketched queries. Prior sketch based image retrieval (SBIR) algorithms adopt either a category-level or fine-grain (instance-level) definition of cross-domain similarity—returning images that match the sketched object class (category-level SBIR), or a specific instance of that object (fine-grain SBIR). In this paper we take the middle-ground; proposing an SBIR algorithm that returns images sharing both the object category and key visual characteristics of the sketched query without assuming photo-approximate sketches from the user. We describe a deeply learned cross-domain embedding in which ‘mid-grain’ sketch-image similarity may be measured, reporting on the efficacy of unsupervised and semi-supervised manifold alignment techniques to encourage better intra-category (mid-grain) discrimination within that embedding. We propose a new mid-grain sketch-image dataset (MidGrain65c) and demonstrate not only mid-grain discrimination, but also improved category-level discrimination using our approach. <s> BIB007 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch-Photo Retrieval <s> In the early days, content-based image retrieval (CBIR) was studied with global features. Since 2003, image retrieval based on local descriptors ( de facto SIFT) has been extensively studied for over a decade due to the advantage of SIFT in dealing with image transformations. Recently, image representations based on the convolutional neural network (CNN) have attracted increasing interest in the community and demonstrated impressive performance. Given this time of rapid evolution, this article provides a comprehensive survey of instance retrieval over the last decade. Two broad categories, SIFT-based and CNN-based methods, are presented. For the former, according to the codebook size, we organize the literature into using large/medium-sized/small codebooks. For the latter, we discuss three lines of methods, i.e., using pre-trained or fine-tuned CNN models, and hybrid methods. The first two perform a single-pass of an image to the network, while the last category employs a patch-based feature extraction scheme. This survey presents milestones in modern instance retrieval, reviews a broad selection of previous works in different categories, and provides insights on the connection between SIFT and CNN-based methods. After analyzing and comparing retrieval performance of different categories on several datasets, we discuss promising directions towards generic and specialized instance retrieval. <s> BIB008
Sketch-photo retrieval is also termed as sketch-based image retrieval (SBIR) BIB001 , BIB003 , BIB004 , BIB005 . Note that in sketchbased image retrieval scenarios, the retrieval is mutual retrieval that both of sketch and photo can be used as query to search target in the gallery of the other modality. Sketch-photo retrieval is a challenging task. Free-hand sketch is highly abstract and arbitrary, so that it is absolutely not the edge-map of the natural photo. The domain gap between sketch and photo means: (i) the intrinsic difference between sparse line drawing and dense pixel spaces. (ii) the significant distinction between human abstraction expression and the pixel-perfect copy of the real world. Figure 6 provides a taxonomy for SBIR. From the perspective of evaluation criterion, SBIR can be divided into conventional/coarse-grained SBIR (i.e., category-level SBIR), mid-grained BIB007 , and fine-grained SBIR (i.e., instance-level SBIR). FG-SBIR is essentially a kind of instance-level retrieval BIB008 . From the perspective of retrieval embedding space, SBIR can be divided into Euclidean/common retrieval and hashing/fast retrieval. From the perspective of supervision involved in training, SBIR can be divided into fully-supervised and zero-shot retrievals. In coarse-grained SBIR, given a target sketch as query, a ranking list will be returned based on the similarity (e.g., Euclidean distance, Hamming distance). The retrieval will be judged as correct, if the photo ranked at the top has the identical class label as the query. However, in fine-grained SBIR, the evaluation metric is based on the instance-level sketch-photo pairings. The retrieval will be judged as correct only when the returned photo is from the instance pair that contains the query sketch. With the development of deep learning and popularity of the smart touch-screen devices, it is more convenient to obtain free-hand sketches, so that more and more sketchbased human-computer interaction (SBHCI) applications are able to serve for our daily life. Several sketch-based commodity search engines have already been implemented, e.g., sketch-based skirt image retrieval system BIB002 , finegrained sketch-based shoe BIB003 , BIB006 /chair BIB003 , BIB006 /handbag BIB006 retrieval systems. The following discussion will be divided into three main parts, i.e., category and instance level SBIR, zero-shot SBIR, and SBIR hashing.