Deep Learning for Free-Hand Sketch: A Survey <s> Retrieval and Hashing <s> The Scale-Invariant Feature Transform (or SIFT) algorithm is a highly robust method to extract and consequently match distinctive invariant features from images. These features can then be used to reliably match objects in diering images. The algorithm was rst proposed by Lowe [12] and further developed to increase performance resulting in the classic paper [13] that served as foundation for SIFT which has played an important role in robotic and machine vision in the past decade. <s> BIB001 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Retrieval and Hashing <s> We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and D equal to ½ everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with backpropagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples. <s> BIB002 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Retrieval and Hashing <s> Matching hand drawn sketches is an attractive topic in image understanding and potentially has many applications. Previous sketch matching algorithms often rely on extracted feature points and their correspondence. However, the nature of hand drawn sketches, such as lack of constraints and having significantly large variations, makes the matching task extremely challenging. In this paper, we propose a metric learning method to match hand drawn sketches without explicitly localizing the feature points. We train a Siamese Convolutional Neural Network (CNN) with pure convolutional layers to represent the sketch features. This allows us to benefit from the rich representative power of CNN, as well as to preserve the spatial information of features. We evaluated the sketch retrieval performance of our model on a large dataset. Experiment results showed the effectiveness of our model. <s> BIB003 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Retrieval and Hashing <s> Generative Adversarial Networks (GAN) are able to learn excellent representations for unlabelled data which can be applied to image generation and scene classification. Representations learned by GANs have not yet been applied to retrieval. In this paper, we show that the representations learned by GANs can indeed be used for retrieval. We consider heritage documents that contain unlabelled Merchant Marks, sketch-like symbols that are similar to hieroglyphs. We introduce a novel GAN architecture with design features that make it suitable for sketch retrieval. The performance of this sketch-GAN is compared to a modified version of the original GAN architecture with respect to simple invariance properties. Experiments suggest that sketch-GANs learn representations that are suitable for retrieval and which also have increased stability to rotation, scale and translation compared to the standard GAN architecture. <s> BIB004 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Retrieval and Hashing <s> We propose a deep hashing framework for sketch retrieval that, for the first time, works on a multi-million scale human sketch dataset. Leveraging on this large dataset, we explore a few sketch-specific traits that were otherwise under-studied in prior literature. Instead of following the conventional sketch recognition task, we introduce the novel problem of sketch hashing retrieval which is not only more challenging, but also offers a better testbed for large-scale sketch analysis, since: (i) more fine-grained sketch feature learning is required to accommodate the large variations in style and abstraction, and (ii) a compact binary code needs to be learned at the same time to enable efficient retrieval. Key to our network design is the embedding of unique characteristics of human sketch, where (i) a two-branch CNN-RNN architecture is adapted to explore the temporal ordering of strokes, and (ii) a novel hashing loss is specifically designed to accommodate both the temporal and abstract traits of sketches. By working with a 3.8M sketch dataset, we show that state-of-the-art hashing models specifically engineered for static images fail to perform well on temporal sketch data. Our network on the other hand not only offers the best retrieval performance on various code sizes, but also yields the best generalization performance under a zero-shot setting and when re-purposed for sketch recognition. Such superior performances effectively demonstrate the benefit of our sketch-specific design. <s> BIB005 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Retrieval and Hashing <s> Text-based retrieval systems have been popular, but content-based retrieval systems have gained widespread acceptance in recent years to directly retrieve diverse media based on their visual content, such as color, texture, and shape. Among many content-based retrieval systems, sketch-based media retrieval systems have attracted attention recently with the proliferation of tablet PCs and smart mobile devices. Sketch-based retrieval requires the user to draw a freehand sketch query, but freehand drawing can be challenging for those with limited drawing skills. This degrades retrieval performance, since successful retrieval depends on the quality of the sketch query image drawn by the user. To address this issue, we propose a real-time stroke guidance for freehand sketch retrieval that continuously displays next-stroke shadow sketches on the canvas based on the user's step-by-step partial strokes. We train a stroke guidance network that learns the mapping between the step-wise stroke relations to predict the user's next stroke. The proposed stroke guidance for freehand sketch retrieval system runs on a five step next-stroke prediction model that identifies candidate next-stroke sketches from a database of millions of sketches. The system retrieves variable number of sketch object classes at different drawing stages. During the initial sketching stage, diverse drawing possibilities are covered by retrieving multiple sketch classes; as the sketching progresses, the intended sketch class is narrowed down to one. Deep binary hashing is employed for efficient similarity matching of relevant next-stroke sketches. We extend the Google QuickDraw dataset to create a five step sketch stroke database. Qualitative and quantitative experiments are conducted to verify the effectiveness of the proposed system, which can be utilized for drawing guidance, tracing, and sketch retrieval. Tracing refers to the act of copying the shadowed line of a guiding image by drawing over its lines. <s> BIB006
Sketch retrieval BIB006 , BIB003 , BIB004 means that use sketch as query/target to retrieve the similar samples from sketch gallery. This is also a fundamental task for sketch understanding, with several practical applications. With the popularization of touch-screen devices, more and more sketches are spreading in the internet, activating the study of sketch fast retrieval. Sketch hashing BIB005 is a kind of sketch fast retrieval that all the sketches are encoded and searched as binary hashing code vectors. Sketch retrieval is a challenging task due to the highly abstract and intra-class variations, so that it's difficult to explicitly localizing the feature points across sketches (e.g., the manner of SIFT BIB001 ). After entering the deep learning era, the end-to-end feature learning manners of CNNs outperform the shallow feature extractions on various computer vision tasks. CNNs have also been used for sketch retrieval. In particular, the common practice is using CNNs to learn embedding representations and then conduct the retrieval/matching as k Nearest Neighbors (kNN) problem. Therefore, most existing deep sketch retrieval models work as metric learning manners, which focus on the loss function designs to learn metrics for sketch matching in CNN-based embedding spaces. Wang et al. BIB003 proposed a representative sketch retrieval framework, which has two key components, i.e., pure convolutional layer based Siamese CNN backbone, 1 norm distance based pair-wise loss. The idea is two-fold: (i) Use convolutional feature map to preserve the spatial information for sketches without point correspondence. (ii) Pair-wise loss aims to drive the similar pairs closer in the feature space. In addition to low-level feture based spatial matching, higher level semantic features also work well on sketch retrieval. Xu et al. propose a dual-branch CNN-RNN architecture to represent sketches via dual-branch late-fusion. The aforementioned sketch retrieval/hashing methods work in supervised learning manner. If the class labels are unavailable, adversarial training can be used to force the networks learn how to capture the features for sketches. Based on Generative Adversarial Network (GAN) BIB002 framework, Creswell et al. BIB004 propose the Sketch-GAN to solve sketch retrieval in unsupervised manner, where both the query and gallery sketches are represented by the output features of the discriminator network.