Deep Learning for Free-Hand Sketch: A Survey <s> Sketch-3D Retrieval <s> In this paper, we describe a non-photorealistic rendering system that conveys shape using lines. We go beyond contours and creases by developing a new type of line to draw: the suggestive contour. Suggestive contours are lines drawn on clearly visible parts of the surface, where a true contour would first appear with a minimal change in viewpoint. We provide two methods for calculating suggestive contours, including an algorithm that finds the zero crossings of the radial curvature. We show that suggestive contours can be drawn consistently with true contours, because they anticipate and extend them. We present a variety of results, arguing that these images convey shape more effectively than contour alone. <s> BIB001 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch-3D Retrieval <s> The Scale-Invariant Feature Transform (or SIFT) algorithm is a highly robust method to extract and consequently match distinctive invariant features from images. These features can then be used to reliably match objects in diering images. The algorithm was rst proposed by Lowe [12] and further developed to increase performance resulting in the classic paper [13] that served as foundation for SIFT which has played an important role in robotic and machine vision in the past decade. <s> BIB002 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch-3D Retrieval <s> This article gives a survey of recent research related to the Monge-Kantorovich problem. Principle results are presented on the existence of solutions and their properties both in the Monge optimal transportation problem and the Kantorovich optimal plan problem, along with results on the connections between both problems and the cases when they are equivalent. Diverse applications of these problems in non-linear analysis, probability theory, and differential geometry are discussed. Bibliography: 196 titles. <s> BIB003 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch-3D Retrieval <s> Sketch-based 3D shape retrieval has become an important research topic in content-based 3D object retrieval. To foster this research area, two Shape Retrieval Contest (SHREC) tracks on this topic have been organized by us in 2012 and 2013 based on a small-scale and large-scale benchmarks, respectively. Six and five (nine in total) distinct sketch-based 3D shape retrieval methods have competed each other in these two contests, respectively. To measure and compare the performance of the top participating and other existing promising sketch-based 3D shape retrieval methods and solicit the state-of-the-art approaches, we perform a more comprehensive comparison of fifteen best (four top participating algorithms and eleven additional state-of-the-art methods) retrieval methods by completing the evaluation of each method on both benchmarks. The benchmarks, results, and evaluation tools for the two tracks are publicly available on our websites [1,2]. <s> BIB004 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch-3D Retrieval <s> Retrieving 3D models from 2D human sketches has received considerable attention in the areas of graphics, image retrieval, and computer vision. Almost always in state of the art approaches a large amount of “best views” are computed for 3D models, with the hope that the query sketch matches one of these 2D projections of 3D models using predefined features. We argue that this two stage approach (view selection - matching) is pragmatic but also problematic because the “best views” are subjective and ambiguous, which makes the matching inputs obscure. This imprecise nature of matching further makes it challenging to choose features manually. Instead of relying on the elusive concept of “best views” and the hand-crafted features, we propose to define our views using a minimalism approach and learn features for both sketches and views. Specifically, we drastically reduce the number of views to only two predefined directions for the whole dataset. Then, we learn two Siamese Convolutional Neural Networks (CNNs), one for the views and one for the sketches. The loss function is defined on the within-domain as well as the cross-domain similarities. Our experiments on three benchmark datasets demonstrate that our method is significantly better than state of the art approaches, and outperforms them in all conventional metrics. <s> BIB005 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch-3D Retrieval <s> Sketch-based 3D model retrieval is to retrieve 3D models given a user's hand-drawn sketch. Due to the big semantic gap between rough sketch representation and accurate 3D model coordinates, sketch-based 3D model retrieval (SBR) is one of the most challenging research topics in the field of 3D model retrieval. To bridge the semantic gap, a novel semantic tree-based SBR algorithm is proposed in this paper. Given a 2D sketch query and a collection of 3D models, a 3D semantic tree is built up first based on the ontology structure of WordNet. Every leaf node in the tree contains a set of 3D models assigned to this class according to their semantic classification/label information. Then, sketch components of the 2D query sketch are identified by sketch segmentation and annotation. Finally, by measuring the semantic relatedness between the sketch components' annotations and tree nodes in the 3D semantic tree, the similarities between the 2D sketch and 3D models are computed to find out the most relevant 3D models. Experimental results demonstrate the effectiveness and promising potentials of our approach on sketch-based 3D model retrieval. <s> BIB006 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch-3D Retrieval <s> Sketch-based 3D shape retrieval, which returns a set of relevant 3D shapes based on users' input sketch queries, has been receiving increasing attentions in both graphics community and vision community. In this work, we address the sketch-based 3D shape retrieval problem with a novel Cross-Domain Neural Networks (CDNN) approach, which is further extended to Pyramid Cross-Domain Neural Networks (PCDNN) by cooperating with a hierarchical structure. In order to alleviate the discrepancies between sketch features and 3D shape features, a neural network pair that forces identical representations at the target layer for instances of the same class is trained for sketches and 3D shapes respectively. By constructing cross-domain neural networks at multiple pyramid levels, a many-to-one relationship is established between a 3D shape feature and sketch features extracted from different scales. We evaluate the effectiveness of both CDNN and PCDNN approach on the extended large-scale SHREC 2014 benchmark and compare with some other well established methods. Experimental results suggest that both CDNN and PCDNN can outperform state-of-the-art performance, where PCDNN can further improve CDNN when employing a hierarchical structure. <s> BIB007 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch-3D Retrieval <s> 3D sketch-based 3D model retrieval is to retrieve similar 3D models using users' hand-drawn 3D sketches as input. Compared with traditional 2D sketch-based retrieval, 3D sketch-based 3D model retrieval is a brand new and challenging research topic. In this paper, we employ advanced deep learning method and propose a novel 3D sketch based 3D model retrieval system. Our system has been comprehensively tested on two benchmark datasets and compared with other existing 3D model retrieval algorithms. The experimental results reveal our approach outperforms other competing state-of-the-arts and demonstrate promising potential of our approach on 3D sketch based applications. <s> BIB008 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch-3D Retrieval <s> Recent progress in sketch-based 3D shape retrieval creates a novel and user-friendly way to explore massive 3D shapes on the Internet. However, current methods on this topic rely on designing invariant features for both sketches and 3D shapes, or complex matching strategies. Therefore, they suffer from problems like arbitrary drawings and inconsistent viewpoints. To tackle this problem, we propose a probabilistic framework based on Multi-View Pairwise Relationship (MVPR) learning. Our framework includes multiple views of 3D shapes as the intermediate layer between sketches and 3D shapes, and transforms the original retrieval problem into the form of inferring pairwise relationship between sketches and views. We accomplish pairwise relationship inference by a novel MVPR net, which can automatically predict and merge the pairwise relationships between a sketch and multiple views, thus freeing us from exhaustively selecting the best view of 3D shapes. We also propose to learn robust features for sketches and views via fine-tuning pre-trained networks. Extensive experiments on a large dataset demonstrate that the proposed method can outperform state-of-the-art methods significantly. <s> BIB009 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch-3D Retrieval <s> Retrieving 3D shapes with sketches is a challenging problem since 2D sketches and 3D shapes are from two heterogeneous domains, which results in large discrepancy between them. In this paper, we propose to learn barycenters of 2D projections of 3D shapes for sketch-based 3D shape retrieval. Specifically, we first use two deep convolutional neural networks (CNNs) to extract deep features of sketches and 2D projections of 3D shapes. For 3D shapes, we then compute the Wasserstein barycenters of deep features of multiple projections to form a barycentric representation. Finally, by constructing a metric network, a discriminative loss is formulated on the Wasserstein barycenters of 3D shapes and sketches in the deep feature space to learn discriminative and compact 3D shape and sketch features for retrieval. The proposed method is evaluated on the SHREC13 and SHREC14 sketch track benchmark datasets. Compared to the state-of-the-art methods, our proposed method can significantly improve the retrieval performance. <s> BIB010 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch-3D Retrieval <s> Due to the large cross-modality discrepancy between 2D sketches and 3D shapes, retrieving 3D shapes by sketches is a significantly challenging task. To address this problem, we propose a novel framework to learn a discriminative deep cross-modality adaptation model in this paper. Specifically, we first separately adopt two metric networks, following two deep convolutional neural networks (CNNs), to learn modality-specific discriminative features based on an importance-aware metric learning method. Subsequently, we explicitly introduce a cross-modality transformation network to compensate for the divergence between two modalities, which can transfer features of 2D sketches to the feature space of 3D shapes. We develop an adversarial learning based method to train the transformation model, by simultaneously enhancing the holistic correlations between data distributions of two modalities, and mitigating the local semantic divergences through minimizing a cross-modality mean discrepancy term. Experimental results on the SHREC 2013 and SHREC 2014 datasets clearly show the superior retrieval performance of our proposed model, compared to the state-of-the-art approaches. <s> BIB011 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch-3D Retrieval <s> How to effectively retrieve desired 3D models with simple queries is a long-standing problem in computer vision community. The model-based approach is quite straightforward but nontrivial, since people could not always have the desired 3D query model available by side. Recently, large amounts of wide-screen electronic devices are prevail in our daily lives, which makes the sketch-based 3D shape retrieval a promising candidate due to its simpleness and efficiency. The main challenge of sketch-based approach is the huge modality gap between sketch and 3D shape. In this paper, we proposed a novel deep correlated holistic metric learning (DCHML) method to mitigate the discrepancy between sketch and 3D shape domains. The proposed DCHML trains two distinct deep neural networks (one for each domain) jointly, which learns two deep nonlinear transformations to map features from both domains into a new feature space. The proposed loss, including discriminative loss and correlation loss, aims to increase the discrimination of features within each domain as well as the correlation between different domains. In the new feature space, the discriminative loss minimizes the intra-class distance of the deep transformed features and maximizes the inter-class distance of the deep transformed features to a large margin within each domain, while the correlation loss focused on mitigating the distribution discrepancy across different domains. Different from existing deep metric learning methods only with loss at the output layer, our proposed DCHML is trained with loss at both hidden layer and output layer to further improve the performance by encouraging features in the hidden layer also with desired properties. Our proposed method is evaluated on three benchmarks, including 3D Shape Retrieval Contest 2013, 2014, and 2016 benchmarks, and the experimental results demonstrate the superiority of our proposed method over the state-of-the-art methods. <s> BIB012 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Sketch-3D Retrieval <s> Sketch-based 3D shape retrieval has been extensively studied in recent works, most of which focus on improving the retrieval accuracy, whilst neglecting the efficiency. In this paper, we propose a novel framework for efficient sketch-based 3D shape retrieval, i.e., Deep Sketch-Shape Hashing (DSSH), which tackles the challenging problem from two perspectives. Firstly, we propose an intuitive 3D shape representation method to deal with unaligned shapes with arbitrary poses. Specifically, the proposed Segmented Stochastic-viewing Shape Network models discriminative 3D representations by a set of 2D images rendered from multiple views, which are stochastically selected from non-overlapping spatial segments of a 3D sphere. Secondly, Batch-Hard Binary Coding (BHBC) is developed to learn semantics-preserving compact binary codes by mining the hardest samples. The overall framework is jointly learned by developing an alternating iteration algorithm. Extensive experimental results on three benchmarks show that DSSH improves both the retrieval efficiency and accuracy remarkably, compared to the state-of-the-art methods. <s> BIB013
Using sketch as query to retrieve the target 3D model is termed as sketch based 3D retrieval or sketch-3D retrieval. Compared with the sketch based image retrieval, sketch based 3D retrieval is more challenging due to the larger domain gap across the 2D space to the 3D space. For more background knowledge, please read the survey paper BIB004 on sketch-based 3D retrieval. Before deep learning era, sketch-3D retrieval is often solved in a two-stage manner, which involves two separated steps BIB006 , i.e., (i) "best view" selection for 3D model: use the automatic procedure to select the representative viewpoints of a given 3D model, hoping that one of the selected viewpoints is similar to that of the query sketches, and (ii) matching between sketches and the 2D projection of the selected best views, based on manually predefined features, e.g., SIFT BIB002 . More specifically, based on the selected views, 3D models can be projected into 2D space by the line rendering algorithm BIB001 . Note that each 3D model can be projected as multiple 2D views, and allowed to have multiple best views. Thus, a sketch matches a given 3D model, if it matches one of the views of the 3D model (see Figure 15 ). However, as argued in some previous works BIB005 , the view selection is the bottleneck to the two-stage approach, since selecting best views is subjective and ambiguous. Moreover, selecting hand-crafted features manually will also bring biases. Gradually, sketch based 3D model retrieval has been studied in the end-to-end pipeline based on deep learning BIB005 , BIB009 , BIB007 , BIB008 , BIB011 , BIB013 . Representatively, Wang et al. BIB005 propose to use two Siamese networks to learn the sketch and projected views directly in the endto-end manner, which takes a quadruplet of sketches and projected viewpoints as input, and uses multiple pair-wise losses. In each quadruplet atoms, two sketches (X 1 , X 2 ) and two viewpoints (V 1 , V 2 ) are randomly selected from sketch and 3D domains, respectively. For simplicity, Wang et al. BIB005 assume that X 1 and X 2 are from the same category sharing a Siamese network, while V 1 and V 2 are also from the same category sharing another Siamese network. The loss of a quadruplet is defined as Where L pair (·, ·) is the pair-wise loss function that will be detailed in the following. In (7), L pair (X 1 , X 2 ) and L pair (V 1 , V 2 ) will enable the network to learn the categorylevel similarity within each domain, and L pair (X 1 , V 1 ) will force the network to learn the cross-modal similarity in the end-to-end procedure. Given input samples a and b, one of their pair-wise loss functions can be defined as: where y a and y b are the corresponding class labels, and F a (·) and F b (·) denotes the feature extractions that have been applied to a and b, respectively. Addition to the pair-wise deep metric learning, other deep metric learning methods also can be applied to the cross-domain matching across sketch and 3d model, e.g., triplet ranking  ,  , deep correlation metric learning  , BIB012 . Moreover, some previous works also studied how to represent 3D models more comprehensively in sketch based 3D retrieval tasks. For instance, Xie et al. BIB010 propose to represent 3D models by computing the Wasserstein distance BIB003 based barycenters of multiple projections of 3D models. Comments: Due to the significant discrepancy across sketches and 3D models, it is unpractical to force sketch domain and 3D domain share the backbone network in sketch-based 3D retrieval models. This is different to that of the sketch-based image retrieval models.