Deep Learning for Free-Hand Sketch: A Survey <s> Simplification, and Abstraction <s> We present a new approach to the simplification of line drawings, in which a smaller set of lines is created to represent the geometry of the original lines. An important feature of our method is that it maintains the morphological structure of the original drawing while allowing user-de ned decisions about the appearance of lines. The technique works by analyzing the structure of the drawing at a certain scale and identifying clusters of lines that can be merged given an extendable error threshold. These clusters are then processed to create new lines, in a separate stage where different scenarios can be favored based on the application. Successful results are presented for a variety of drawings including scanned and vectorized artwork, original vector drawings, drawings created from 3d models, and hatching marks. The clustering technique is shown to be effective in all these situations. <s> BIB001 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Simplification, and Abstraction <s> In this paper, we propose a new algorithm for simplifying line drawing sketches. First, we segment the strokes at the ::: points of large curvature if desired. Then, we perform a low-pass filter and use the result to assign a weight to every ::: stroke. The strokes are moved to the position of the higher weight. After that, we find the stroke pairs and combine them ::: to reduce the total number of the strokes, resulting in a cleaner line drawing art. This system also cuts down the ::: disordered and confusing small strokes and combines them to form long strokes. <s> BIB002 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Simplification, and Abstraction <s> In this paper, we propose a novel approach to simplify sketch drawings. The core problem is how to group sketchy strokes meaningfully, and this depends on how humans understand the sketches. The existing methods mainly rely on thresholding low-level geometric properties among the strokes, such as proximity, continuity and parallelism. However, it is not uncommon to have strokes with equal geometric properties but different semantics. The lack of semantic analysis will lead to the inability in differentiating the above semantically different scenarios. In this paper, we point out that, due to the gestalt phenomenon of closure, the grouping of strokes is actually highly influenced by the interpretation of regions. On the other hand, the interpretation of regions is also influenced by the interpretation of strokes since regions are formed and depicted by strokes. This is actually a chicken-or-the-egg dilemma and we solve it by an iterative cyclic refinement approach. Once the formed stroke groups are stabilized, we can simplify the sketchy strokes by replacing each stroke group with a smooth curve. We evaluate our method on a wide range of different sketch styles and semantically meaningful simplification results can be obtained in all test cases. <s> BIB003 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Simplification, and Abstraction <s> As a form of visual representation, freehand line sketches are typically studied as an end product of the sketching process. However, from a recognition point of view, one can also study various orderings and properties of the primitive strokes that compose the sketch. Studying sketches in this manner has enabled us to create novel sparse yet discriminative sketch-based representations for object categories which we term category-epitomes. Concurrently, the epitome construction provides a natural measure for quantifying the sparseness underlying the original sketch, which we term epitome-score. We analyze category-epitomes and epitome-scores for hand-drawn sketches from a sketch dataset of 160 object categories commonly encountered in daily life. Our analysis provides a novel viewpoint for examining the complexity of representation for visual object categories. <s> BIB004 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Simplification, and Abstraction <s> In this paper, we propose a novel approach to creating clean line drawing from a scribbled sketch automatically. The main problem is determining which strokes of a scribbled sketch should be merged. We use a machine learning approach to solve this problem. Our method can automatically generate training data by comparing scribbled sketches with manually drawn line drawings without using annotations. In order to verify the generated training data, we merged strokes and created clean line drawings in accordance with the generated training data. In addition, we trained a support vector machine to estimate the pairs of strokes to be merged. Further, we verified that our method can create line drawings using this estimator. <s> BIB005 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Simplification, and Abstraction <s> Human free-hand sketches have been studied in various contexts including sketch recognition, synthesis and fine-grained sketch-based image retrieval (FG-SBIR). A fundamental challenge for sketch analysis is to deal with drastically different human drawing styles, particularly in terms of abstraction level. In this work, we propose the first stroke-level sketch abstraction model based on the insight of sketch abstraction as a process of trading off between the recognizability of a sketch and the number of strokes used to draw it. Concretely, we train a model for abstract sketch generation through reinforcement learning of a stroke removal policy that learns to predict which strokes can be safely removed without affecting recognizability. We show that our abstraction model can be used for various sketch analysis tasks including: (1) modeling stroke saliency and understanding the decision of sketch recognition models, (2) synthesizing sketches of variable abstraction for a given category, or reference object instance in a photo, and (3) training a FG-SBIR model with photos only, bypassing the expensive photo-sketch pair collection step. <s> BIB006 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Simplification, and Abstraction <s> In this paper, we propose a novel deep framework for part-level semantic parsing of freehand sketches, which makes three main contributions that are experimentally shown to have substantial practical merit. First, we introduce a new idea named homogeneous transformation to address the problem of domain adaptation. For the task of sketch parsing, there is no available data of labeled freehand sketches that can be directly used for model training. An alternative solution is to learn from the existing parsing data of real images, while the domain adaptation is an inevitable problem. Unlike existing methods that utilize the edge maps of real images to approximate freehand sketches, the proposed homogeneous transformation method transforms the data from two different domains into a homogeneous space to minimize the semantic gap. Second, we design a soft-weighted loss function as guidance for the training process, which gives attention to both the ambiguous label boundary and class imbalance. Third, we present a staged learning strategy to improve the parsing performance of the trained model, which takes advantage of the shared information and specific characteristic from different sketch categories. Extensive experimental results demonstrate the effectiveness of these methods. Specifically, to evaluate the generalization ability of our homogeneous transformation method, additional experiments at the task of sketch-based image retrieval are conducted on the QMUL FG-SBIR dataset. By integrating the proposed three methods into a unified framework, our final deep semantic sketch parsing (DeepSSP) model achieves the state-of-the-art on the public SketchParse dataset. <s> BIB007 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Simplification, and Abstraction <s> Automatic data abstraction is an important capability for both benchmarking machine intelligence and supporting summarization applications. In the former one asks whether a machine can `understand' enough about the meaning of input data to produce a meaningful but more compact abstraction. In the latter this capability is exploited for saving space or human time by summarizing the essence of input data. In this paper we study a general reinforcement learning based framework for learning to abstract sequential data in a goal-driven way. The ability to define different abstraction goals uniquely allows different aspects of the input data to be preserved according to the ultimate purpose of the abstraction. Our reinforcement learning objective does not require human-defined examples of ideal abstraction. Importantly our model processes the input sequence holistically without being constrained by the original input order. Our framework is also domain agnostic -- we demonstrate applications to sketch, video and text data and achieve promising results in all domains. <s> BIB008
Before deep learning era, sketch simplification has been widely studied BIB003 , BIB002 , BIB005 by computer graphic researchers to obtain the sparse yet discriminative representation, which aims to determine which strokes of a given scribbled sketch should be merged BIB005 by some artificial principles, e.g., Gestalt principle. A typical pipeline is the Fig. 10 . Sketches (bus, car, cat) and ground truth annotations selected from sketch parse paper BIB007 . The semantic parts and background are annotated by colors. Best viewed in color. two-stage sketch simplification BIB001 , which geometrically clusters the stroke lines into groups, and then generates a new line for each group. This solution to sketch simplification is based on the computer graphic techniques. This is to say that in some cases the sketch simplification is a geometric calculation problem, but not a machine learning problem. With the prevalence of deep learning, CNN-based sketch representations have been used for the sketch simplification BIB004 . Furthermore, a brand new concept "stroke-level sketch abstraction" appears in sketch community recently. Note that "stroke-level sketch abstraction" is completely different to "sketch simplification". Stroke-level sketch abstraction aims to remove the redundant information on stroke basis yet keeping the recognizability of the sketch, which can be used for various sketch understanding tasks, including: (i) modeling stroke saliency for sketch and (ii) synthesizing sketches of variable abstraction for a given category. Moreover, stroke-level sketch abstraction is a strokelevel fine-grained sketch understanding task, so that it can not be modeled by CNN efficiently. This is its task-unique challenge. Removing the unnecessary strokes from the given sketch is intrinsically a dynamic process of trading off between the recognizability of a sketch and the number of strokes used to draw it BIB006 . However, it is difficult to write a concrete math formulations as optimization objective to train the models to abstract free-hand sketch. Thus, reinforcement learning framework is an useful tool for this task. Muhammad et al. BIB006 proposed a deep reinforcement learning (RL) based RNN model that learns to predict which strokes can be removed without affecting recognizability. Following this, Muhammad et al. BIB008 proposed a novel goal-driven sequential abstractor trained by RL that can abstract sketches depending on different abstraction goals, e.g., category, attribute. Comments: Generally, the CNN-based deep learning and the non-deep models for sketch simplification work from the perspective of graphics. The deep reinforcement learning based stroke-level sketch abstraction methods focus on how to understand the logic of sketching.