Sketching is more fundamental to human cognition than speech. Deep Neural Networks (DNNs) have achieved the state-of-the-art in speech-related tasks but have not made significant development in generating stroke-based sketches a.k.a sketches in vector format. Though there are Variational Auto Encoders (VAEs) for generating sketches in vector format, there is no Generative Adversarial Network (GAN) architecture for the same. In this paper, we propose a standalone GAN architecture SkeGAN and a VAE-GAN architecture VASkeGAN, for sketch generation in vector format. SkeGAN is a stochastic policy in Reinforcement Learning (RL), capable of generating both multidimensional continuous and discrete outputs. VASkeGAN hybridizes a VAE and a GAN, in order to couple the efficient representation of data by VAE with the powerful generating capabilities of a GAN, to produce visually appealing sketches. We also propose a new metric called the Ske-score which quantifies the quality of vector sketches. We have validated that SkeGAN and VASkeGAN generate visually appealing sketches by using Human Turing Test and Ske-score.
There are a very few approaches of sketch generation which use stochastic techniques such as Hidden Markov Models (HMMs) @cite_15 and others that use pure image processing techniques such as @cite_3 . There are quite a lot of work done relating to human-drawn sketches in general using DL such as recognition @cite_23 @cite_7 @cite_13 , eye-fixation or saliency @cite_19 , guessing a sketch being drawn @cite_12 and parsing @cite_27 . Specifically @cite_24 uses GANs to generate sketches of human faces given digital portraits of their faces. There are also works such as @cite_0 @cite_4 which discuss the approaches to convert rasterized sketches into realistic images. One commonality amongst all of them is that all of them work with sketches in the raster format.
@cite_4 @cite_7 @cite_3 @cite_24 @cite_19 @cite_27 @cite_0 @cite_23 @cite_15 @cite_13 @cite_12
2884466206 2249139947 2018548648 2964356534 2594648402 2963914894 2963561004 2949877834 2160333382 2063559668 2963647718
In this paper we investigate image generation guided by hand sketch. When the input sketch is badly drawn, the output of common image-to-image translation follows the input edges due to the hard condition imposed by the translation process. Instead, we propose to use sketch as weak constraint, where the output edges do not necessarily follow the input edges. We address this problem using a novel joint image completion approach, where the sketch provides the image context for completing, or generating the output image. We train a generated adversarial network, i.e, contextual GAN to learn the joint distribution of sketch and the corresponding image by using joint images. Our contextual GAN has several advantages. First, the simple joint image representation allows for simple and effective learning of joint distribution in the same image-sketch space, which avoids complicated issues in cross-domain learning. Second, while the output is related to its input overall, the generated features exhibit more freedom in appearance and do not strictly align with the input features as previous conditional GANs do. Third, from the joint image’s point of view, image and sketch are of no difference, thus exactly the same deep joint image completion network can be used for image-to-sketch generation. Experiments evaluated on three different datasets show that our contextual GAN can generate more realistic images than state-of-the-art conditional GANs on challenging inputs and generalize well on common categories. Studies from neuroscience show that part-mapping computations are employed by human visual system in the process of object recognition. In this paper, we present an approach for analyzing semantic-part characteristics of object category representations. For our experiments, we use category-epitome, a recently proposed sketch-based spatial representation for objects. To enable part-importance analysis, we first obtain semantic-part annotations of hand-drawn sketches originally used to construct the epitomes. We then examine the extent to which the semantic-parts are present in the epitomes of a category and visualize the relative importance of parts as a word cloud. Finally, we show how such word cloud visualizations provide an intuitive understanding of category-level structural trends that exist in the category-epitome object representations. Our method is general in applicability and can also be used to analyze part-based visual object representations for other depiction methods such as photographic images. We describe Paul, a robotic installation that produces observational face drawings of people. Paul is a naive drawer: it does not have highlevel knowledge of the structures constitutive of the human face (such as the mouth, nose, eyes) nor the capability of learning expertise based on experience as a human would. However, Paul is able to draw using the equivalent of an artist's stylistic signature based on a number of processes mimicking drawing skills and technique, which together form a drawing cycle. Furthermore, we present here our first efforts in implementing two different versions of visual feedback to permit the robot to iteratively augment and improve a drawing which initially is built from a process of salient lines recovery. The first form of visual feedback we study we refer to as computational as it involves a purely internal (memory-based) representation of regions to render via shading by the robot. The second version we call physical as it involves the use of a camera as an 'eye' taking new snapshots of the artefact in progress. This is then analysed to take decisions on where and how to render shading next. A main point we emphasise in this work is the issue of embodiment of graphical systems, in our case in a robotic platform. We present our arguments in favour of such a position for the graphics community to reflect upon. Finally, we emphasise that the drawings produced by Paul have been considered of interest by fine art professionals in recent international art fairs and exhibitions, as well as by the public at large. One drawing is now in the Victoria and Albert museum collection. We identify a number of factors that may account for such perceived qualities of the produced drawings. Graphical abstractPaul drawing at the Merge Festival Gallery, London, October 2012. Paul is a robotic installation that produces observational face drawings of people using the equivalent of an artist s stylistic signature forming a drawing cycle with visual feedback.Display Omitted HighlightsÂ? Paul is a robotic system which integrates graphics with vision for face portraying. Â? The NPR community can explore the embodiment of artistic skills jointly with software. Â? We design, implement and test computational (software only) visual feedback. Â? We also design, implement and test physical (hardware and software) visual feedback. Â? The production of Paul is being exhibited internationally in well known art venues. The study of eye gaze fixations on photographic images is an active research area. In contrast, the image sub-category of freehand sketches has not received as much attention for such studies. In this paper, we analyze the results of a free-viewing gaze fixation study conducted on 3904 freehand sketches distributed across 160 object categories. Our analysis shows that fixation sequences exhibit marked consistency within a sketch, across sketches of a category and even across suitably grouped sets of categories. This multi-level consistency is remarkable given the variability in depiction and extreme image content sparsity that characterizes hand-drawn object sketches. In this paper, we show that the multi-level consistency in the fixation data can be exploited to 1) predict a test sketch’s category given only its fixation sequence and 2) build a computational model which predicts part-labels underlying fixations on objects. We hope that our findings motivate the community to deem sketch-like representations worthy of gaze-based studies vis-a-vis photographic images. The ability to semantically interpret hand-drawn line sketches, although very challenging, can pave way for novel applications in multimedia. We propose SKETCHPARSE, the first deep-network architecture for fully automatic parsing of freehand object sketches. SKETCHPARSE is configured as a two-level fully convolutional network. The first level contains shared layers common to all object categories. The second level contains a number of expert sub-networks. Each expert specializes in parsing sketches from object categories which contain structurally similar parts. Effectively, the two-level configuration enables our architecture to scale up efficiently as additional categories are added. We introduce a router layer which (i) relays sketch features from shared layers to the correct expert (ii) eliminates the need to manually specify object category during inference. To bypass laborious part-level annotation, we sketchify photos from semantic object-part image datasets and use them for training. Our architecture also incorporates object pose prediction as a novel auxiliary task which boosts overall performance while providing supplementary information regarding the sketch. We demonstrate SKETCHPARSE's abilities (i) on two challenging large-scale sketch datasets (ii) in parsing unseen, semantically related object categories (iii) in improving fine-grained sketch-based image retrieval. As a novel application, we also outline how SKETCHPARSE's output can be used to generate caption-style descriptions for hand-drawn sketches. Synthesizing realistic images from human drawn sketches is a challenging problem in computer graphics and vision. Existing approaches either need exact edge maps, or rely on retrieval of existing photographs. In this work, we propose a novel Generative Adversarial Network (GAN) approach that synthesizes plausible images from 50 categories including motorcycles, horses and couches. We demonstrate a data augmentation technique for sketches which is fully automatic, and we show that the augmented data is helpful to our task. We introduce a new network building block suitable for both the generator and discriminator which improves the information flow by injecting the input image at multiple scales. Compared to state-of-the-art image translation methods, our approach generates more realistic images and achieves significantly higher Inception Scores. Freehand sketching is an inherently sequential process. Yet, most approaches for hand-drawn sketch recognition either ignore this sequential aspect or exploit it in an ad-hoc manner. In our work, we propose a recurrent neural network architecture for sketch object recognition which exploits the long-term sequential and structural regularities in stroke data in a scalable manner. Specifically, we introduce a Gated Recurrent Unit based framework which leverages deep sketch features and weighted per-timestep loss to achieve state-of-the-art results on a large database of freehand object sketches across a large number of object categories. The inherently online nature of our framework is especially suited for on-the-fly recognition of objects as they are being drawn. Thus, our framework can enable interesting applications such as camera-equipped robots playing the popular party game Pictionary with human players and generating sparsified yet recognizable sketches of objects. We present a system for generating 2D illustrations from hand drawn outlines consisting of only curve strokes. A user can draw a coarse sketch and the system would automatically augment the shape, thickness, color and surrounding texture of the curves making up the sketch. The styles for these refinements are learned from examples whose semantics have been pre-classified. There can be several styles applicable on a curve and the system automatically identifies which one to use and how to use it based on a curve's shape and its context in the illustration. Our approach is based on a Hierarchical Hidden Markov Model. We present a two level hierarchy in which the refinement process is applied at: the curve level and the scene level. As a form of visual representation, freehand line sketches are typically studied as an end product of the sketching process. However, from a recognition point of view, one can also study various orderings and properties of the primitive strokes that compose the sketch. Studying sketches in this manner has enabled us to create novel sparse yet discriminative sketch-based representations for object categories which we term category-epitomes. Concurrently, the epitome construction provides a natural measure for quantifying the sparseness underlying the original sketch, which we term epitome-score. We analyze category-epitomes and epitome-scores for hand-drawn sketches from a sketch dataset of 160 object categories commonly encountered in daily life. Our analysis provides a novel viewpoint for examining the complexity of representation for visual object categories.