  We introduce a model that learns to convert simple hand drawings
  into graphics programs written in a subset of \LaTeX.~ The model
  combines techniques from deep learning and program synthesis.   We
  learn a convolutional neural network that proposes plausible drawing
  primitives that explain an image. These drawing primitives are like
  a trace of the set of primitive commands issued by a graphics
  program. We learn a model that uses program synthesis techniques to
  recover a graphics program from that trace. These programs have
  constructs like variable bindings, iterative loops, or simple kinds
  of conditionals. With a graphics program in hand, we can correct
  errors made by the deep network and extrapolate drawings.   Taken
  together these results are a step towards agents that induce useful,
  human-readable programs from perceptual input. How can an agent convert noisy, high-dimensional perceptual input to a symbolic, abstract object, such as a computer program? Here we consider this problem within a graphics program synthesis domain. We develop an approach for converting hand drawings into executable source code for drawing the original image. The graphics programs in our domain draw simple figures like those found in machine learning papers (see FIG0 ). The key observation behind our work is that generating a programmatic representation from an image of a diagram involves two distinct steps that require different technical approaches. The first step involves identifying the components such as rectangles, lines and arrows that make up the image. The second step involves identifying the high-level structure in how the components were drawn. In FIG0 , it means identifying a pattern in how the circles and rectangles are being drawn that is best described with two nested loops, and which can easily be extrapolated to a bigger diagram.We present a hybrid architecture for inferring graphics programs that is structured around these two steps. For the first step, a deep network to infers a set of primitive shape-drawing commands. We refer FIG8 : Both the paper and the system pipeline are structured around the trace hypothesisThe new contributions of this work are: (1) The trace hypothesis: a framework for going from perception to programs, which connects this work to other trace-based models, like the Neural Program Interpreter BID17 ; BID26 A model based on the trace hypothesis that converts sketches to high-level programs: in contrast to converting images to vectors or low-level parses BID11 BID14 BID24 BID1 BID2 . FORMULA8 A generic algorithm for learning a policy for efficiently searching for programs, building on Levin search BID13 and recent work like DeepCoder BID0 . Even with the high-level idea of a trace set, going from hand drawings to programs remains difficult. We address these challenges: (1) Inferring trace sets from images requires domain-specific design choices from the deep learning and computer vision toolkits (Sec. 2) . FORMULA4 Generalizing to noisy hand drawings, we will show, requires learning a domain-specific noise model that is invariant to the variations across hand drawings (Sec. 2.1). (3) Discovering good programs requires solving a difficult combinatorial search problem, because the programs are often long and complicated (e.g., 9 lines of code, with nested loops and conditionals). We give a domain-general framework for learning a search policy that quickly guides program synthesizers toward the target programs (Sec. 3.1).
Learn to convert a hand drawn sketch into a high-level program
{"src": "scitldr-AIC_train", "paper_id": "H1DJFybC-"}