We've seen tremendous success of image generating models these years. Generating images through a neural network is usually pixel-based, which is fundamentally different from how humans create artwork using brushes. To imitate human drawing, interactions between the environment and the agent is required to allow trials. However, the environment is usually non-differentiable, leading to slow convergence and massive computation. In this paper we try to address the discrete nature of software environment with an intermediate, differentiable simulation. We present  StrokeNet, a novel model where the agent is trained upon a well-crafted neural approximation of the painting environment. With this approach, our agent was able to learn to write characters such as MNIST digits faster than reinforcement learning approaches in an unsupervised manner. Our primary contribution is the neural simulation of a real-world environment. Furthermore, the agent trained with the emulated environment is able to directly transfer its skills to real-world software. To learn drawing or writing, a person first observes (encodes) the target image visually and uses a pen or a brush to scribble (decode), to reconstruct the original image. For an experienced painter, he or she foresees the consequences before taking any move, and could choose the optimal action.Stroke-based image generation is fairly different from traditional image generation problems due to the intermediate rendering program. Raster-based deep learning approaches for image generation allow effective optimization using back-propagation. While for stroke-based approaches, rather than learning to generate the image, it is more of learning to manipulate the painting program.An intuitive yet potentially effective way to tackle the problem is to first learn this mapping from "stroke data" to the resulting image with a neural network, which is analogous to learning painting experience. An advantage of such a mapping over software is that it provides a continuous transformation. For any painting program, the pixel values of an image are calcuated based on the coordinate points along the trajectory of an action. Specific pixels are indexed by the discrete pixel coordinates, which cuts the gradient flow with respect to the action. In our implementation, the indexing is done by an MLP described in Section 3.We further define "drawing" by giving a formal definition of "stroke". In our context, a "stroke" consists of color, brush radius, and a sequence of tuples containing the coordinate and pressure of each point along the trajectory. We will later describe this in detail in Section 3.Based on these ideas, we train a differentiable approximator of our painting software, which we call a "generator". We then tested the generator by training a vanilla CNN as an agent that encodes the image into "stroke" data as an input for the environment. Our proposed architecture, StrokeNet, basically comprises the two components, a generator and an agent.Finally, an agent is trained to write and draw pictures of several popular datasets upon the generator. For the MNIST (LeCun & Cortes, 2010 ) digits, we evaluated the quality of the agent with a classifier trained solely on the original MNIST dataset, and tested the classifier on generated images. We also compared our method with others to show the efficiency. We explored the latent space of the agent as well. For future work, there are several major improvements we want to make both to the network structure and to the algorithm.The recurrent structure adopted here is of the simplest form. We use this setup because we consider drawing as a Markov process, where the current action only depends on what the agent sees, the target image and the previous frame. More advanced structures like LSTM BID10 or GRU BID3 may boost the performance. A stop sign can be also introduced to determine when to stop drawing, which can be useful in character reconstruction. For the agent, various attention mechanism could be incorporated to help the agent focus on undrawn regions, so that smear and blurry scribbles might be prevented.Secondly, The generator and the agent were trained as two separate parts throughout the experiment. We can somehow train them as a whole: during the training of the agent, store all the intermediate stroke data. After a period of training, sample images from the real environment with the stroke data just collected, and train the generator with the data. By doing so in an iterative manner, the generator could fit better to the current agent and provide more reliable reconstructions, while a changing generator may potentially provide more valuable overall gradients.It is also found useful to add a bit of randomness to the learning rate. Since different decoders of the agent learn at different rates, stochasticity results in more appealing results. For example, the agent usually fails to generalize to color images because it always sticks with one global average color (as shown in FIG0 ). However, it sometimes generates appealing results with some randomness added during the training. As a result of this immobility, the way agent writes is dull compared to humans and reinforcement learning agents like SPIRAL. For instance, when writing the digit "8", the agent is simply writing "3" with endpoints closed. Also, the agent avoids to make intersecting strokes over all datasets, although such actions are harmless and should be totally encouraged and explored! Thus, random sampling techniques could be added to the decision making process to encourage bolder moves. Finally, for the evaluation metrics, the naive l 2 loss can be combined with adversarial learning. If paired sequential data is available, we believe adding it to training will also improve the results. In this paper we bring a proof-of-concept that an agent is able to learn from its neural simulation of an environment. Especially when the environment is deterministic given the action, or contains a huge action space, the proposed approach could be useful. Our primary contribution is that we devised a model-based method to approximate non-differentiable environment with neural network, and the agent trained with our method converges quickly on several datasets. It is able to adapt its skills to real world. Hopefully such approaches can be useful when dealing with more difficult reinforcement learning problems.
StrokeNet is a novel architecture where the agent is trained to draw by strokes on a differentiable simulation of the environment, which could effectively exploit the power of back-propagation.
{"src": "scitldr-AIC_train", "paper_id": "HJxwDiActX"}