Deep Learning for Free-Hand Sketch: A Survey <s> Representative Dataset Taxonomy <s> We introduce a benchmark for evaluating the performance of large-scale sketch-based image retrieval systems. The necessary data are acquired in a controlled user study where subjects rate how well given sketch/image pairs match. We suggest how to use the data for evaluating the performance of sketch-based image retrieval systems. The benchmark data as well as the large image database are made publicly available for further studies of this type. Furthermore, we develop new descriptors based on the bag-of-features approach and use the benchmark to demonstrate that they significantly outperform other descriptors in the literature. <s> BIB001 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Representative Dataset Taxonomy <s> The paper presents a sketch-based image retrieval algorithm. One of the main challenges in sketch-based image retrieval (SBIR) is to measure the similarity between a sketch and an image. To tackle this problem, we propose an SBIR-based approach by salient contour reinforcement. In our approach, we divide the image contour into two types. The first is the global contour map. The second, called the salient contour map, is helpful to find out the object in images similar to the query. In addition, based on the two contour maps, we propose a new descriptor, namely an angular radial orientation partitioning (AROP) feature. It fully utilizes the edge pixels’ orientation information in contour maps to identify the spatial relationships. Our AROP feature based on the two candidate contour maps is both efficient and effective to discover false matches of local features between sketches and images, and can greatly improve the retrieval performance. The application of the retrieval system based on this algorithm is established. The experiments on the image dataset with 0.3 million images show the effectiveness of the proposed method and comparisons with other algorithms are also given. Compared to baseline performance, the proposed method achieves 10% higher precision in top 5. <s> BIB002 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Representative Dataset Taxonomy <s> Studies from neuroscience show that part-mapping computations are employed by human visual system in the process of object recognition. In this paper, we present an approach for analyzing semantic-part characteristics of object category representations. For our experiments, we use category-epitome, a recently proposed sketch-based spatial representation for objects. To enable part-importance analysis, we first obtain semantic-part annotations of hand-drawn sketches originally used to construct the epitomes. We then examine the extent to which the semantic-parts are present in the epitomes of a category and visualize the relative importance of parts as a word cloud. Finally, we show how such word cloud visualizations provide an intuitive understanding of category-level structural trends that exist in the category-epitome object representations. Our method is general in applicability and can also be used to analyze part-based visual object representations for other depiction methods such as photographic images. <s> BIB003 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Representative Dataset Taxonomy <s> The ability to semantically interpret hand-drawn line sketches, although very challenging, can pave way for novel applications in multimedia. We propose SketchParse, the first deep-network architecture for fully automatic parsing of freehand object sketches. SketchParse is configured as a two-level fully convolutional network. The first level contains shared layers common to all object categories. The second level contains a number of expert sub-networks. Each expert specializes in parsing sketches from object categories which contain structurally similar parts. Effectively, the two-level configuration enables our architecture to scale up efficiently as additional categories are added. We introduce a router layer which (i) relays sketch features from shared layers to the correct expert (ii) eliminates the need to manually specify object category during inference. To bypass laborious part-level annotation, we sketchify photos from semantic object-part image datasets and use them for training. Our architecture also incorporates object pose prediction as a novel auxiliary task which boosts overall performance while providing supplementary information regarding the sketch. We demonstrate SketchParse's abilities (i) on two challenging large-scale sketch datasets (ii) in parsing unseen, semantically related object categories (iii) in improving fine-grained sketch-based image retrieval. As a novel application, we also outline how SketchParse's output can be used to generate caption-style descriptions for hand-drawn sketches. <s> BIB004 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Representative Dataset Taxonomy <s> Deep generative models have shown great promise when it comes to synthesising novel images. While they can generate images that look convincing on a higher-level, generating fine-grained details is still a challenge. In order to foster research on more powerful generative approaches, this paper proposes a novel task: generative modelling of 2D tree skeletons. Trees are an interesting shape class because they exhibit complexity and variations that are well-suited to measure the ability of a generative model to generated detailed structures. We propose a new dataset for this task and demonstrate that state-of-the-art generative models fail to synthesise realistic images on our benchmark, even though they perform well on current datasets like MNIST digits. Motivated by these results, we propose a novel network architecture based on combining a variational autoencoder using Recurrent Neural Networks and a convolutional discriminator. The network, error metrics and training procedure are adapted to the task of fine-grained sketching. Through quantitative and perceptual experiments, we show that our model outperforms previous work and that our dataset is a valuable benchmark for generative models. We will make our dataset publicly available. <s> BIB005
In the last decade, the prosperity of both deep learning and touch-screen based human-computer interaction, motivates the fast development of free-hand sketch research. Some brand new topics related to sketch have been proposed. New topics often need new datasets. Moreover, nowadays, almost all the sketch topics are studied in deep learning techniques, which need large-scale datasets. Therefore, in recent ten years, a number of sketch datasets BIB001 , BIB002 , BIB003 , BIB004 , BIB005 are proposed. This part will introduce and compare the recently representative sketch datasets via a detailed taxonomy, and describe what tasks/topics each dataset can be used for. Please see details in Table 2 . Table 2 summarizes the representative sketch datasets of both single-modal and multi-modal, where the comparison is based on the aspects of modalities, sample amount, sample category amount, stroke information, annotations, etc. Note that SVG file is able to generate picture files, e.g., JPEG, PNG (not vice versa).