Simulation is a useful tool in situations where training data for machine learning models is costly to annotate or even hard to acquire. In this work, we propose a reinforcement learning-based method for automatically adjusting the parameters of any (non-differentiable) simulator, thereby controlling the distribution of synthesized data in order to maximize the accuracy of a model trained on that data. In contrast to prior art that hand-crafts these simulation parameters or adjusts only parts of the available parameters, our approach fully controls the simulator with the actual underlying goal of maximizing accuracy, rather than mimicking the real data distribution or randomly generating a large volume of data. We find that our approach (i) quickly converges to the optimal simulation parameters in controlled experiments and (ii) can indeed discover good sets of parameters for an image rendering simulator in actual computer vision applications. In order to train deep neural networks, significant effort has been directed towards collecting largescale datasets for tasks such as machine translation BID16 , image recognition BID7 or semantic segmentation BID14 BID6 . It is, thus, natural for recent works to explore simulation as a cheaper alternative to human annotation BID11 BID19 BID18 . Besides, simulation is sometimes the most viable way to acquire data for rare events such as traffic accidents. However, while simulation makes data collection and annotation easier, it is still an open question what distribution should be used to synthesize data. Consequently, prior approaches have used human knowledge to shape the generating distribution of the simulator BID20 , or synthesized large-scale data with random parameters BID18 . In contrast, this paper proposes to automatically determine simulation parameters such that the performance of a model trained on synthesized data is maximized.Traditional approaches seek simulation parameters that try to model a distribution that resembles real data as closely as possible, or generate enough volume to be sufficiently representative. By learning the best set of simulation parameters to train a model, we depart from the above in three crucial ways. First, the need for laborious human expertise to create a diverse training dataset is eliminated. Second, learning to simulate may allow generating a smaller training dataset that achieves similar or better performances than random or human-synthesized datasets BID18 , thereby saving training resources. Third, it allows questioning whether mimicking real data is indeed the best use of simulation, since a different distribution might be optimal for maximizing a test-time metric (for example, in the case of events with a heavy-tailed distribution).More formally, a typical machine learning setup aims to learn a function h θ that is parameterized by θ and maps from domain X to range Y, given training samples (x, y) ∼ p(x, y). Data x usually arises from a real world process (for instance, someone takes a picture with a camera) and labels y are often annotated by humans (someone describing the content of that picture). The distribution p(x, y) is assumed unknown and only an empirical sample D = {(x i , y i )} N i=1 is available. The simulator attempts to model a distribution q(x, y; ψ). In prior works, the aim is to adjust the form of q and parameters ψ to mimic p as closely as possible. In this work, we attempt to automatically learn the parameters of the simulator ψ such that the loss L of a machine learning model h θ is minimized over some validation data set D val . This objective can be formulated as the bi-level optimization problem DISPLAYFORM0 where h θ is parameterized by model parameters θ, D q(x,y| ψ) describes a data set generated by the simulator and θ(ψ) denotes the implicit dependence of the model parameters θ on the model's training data and consequently, for synthetic data, the simulation parameters ψ. In contrast to BID10 , who propose a similar setup, we focus on the actual data generation process q(x, y; ψ) and are not limited to selecting subsets of existing data. In our formulation , the upper-level problem (equation 1a) can be seen as a meta-learner that learns how to generate data (by adjusting ψ) while the lower-level problem (equation 1b) is the main task model (MTM) that learns to solve the actual task at hand. In Section 2, we describe an approximate algorithm based on policy gradients BID24 to optimize the objective 1. For our algorithm to interact with a black-box simulator, we also present an interface between our model's output ψ and the simulator input.In various experiments on both toy data and real computer vision problems, Section 4 analyzes different variants of our approach and investigates interesting questions, such as: "Can we train a model h θ with less but targeted high-quality data?", or "Are simulation parameters that approximate real data the optimal choice for training models?". The experiments indicate that our approach is able to quickly identify good scene parameters ψ that compete and in some cases even outperform the actual validation set parameters for synthetic as well as real data, on computer vision problems such as object counting or semantic segmentation. The proposed approach can be seen as a meta-learner that alters the data a machine learning model is trained on to achieve high accuracy on a validation set. This concept is similar to recent papers that learn policies for neural network architectures BID25 and optimizers BID0 . In contrast to these works, we are focusing on the data generation parameters and actually create new, randomly sampled data in each iteration. While BID10 proposes a subset selection approach for altering the training data, we are actually creating new data. This difference is important because we are not limited to a fixed probability distribution at data acquisition time. We can thus generate or oversample unusual situations that would otherwise not be part of the training data. Similar to the above-mentioned papers, we also choose a variant of stochastic gradients (policy gradients BID24 ) to overcome the non-differentiable sampling and rendering and estimate the parameters of the policy π ω . While alternatives for black-box optimization exist, like evolutionary algorithms BID21 or sampling-based methods BID1 , we favor policy gradients in this work for their sample efficiency and success in prior art. BID13 train a policy to generate a program that creates a copy of an input image. Similar to us, they use policy gradients to train the policy, although they use an adversarial loss to construct their reward. Again, BID15 seek to tune parameters of a simulator such that the marginal distribution of the synthetic data matches the distribution of the observed data. In contrast to both works, we learn parameters of a simulator that maximize performance of a main task model on a specific task. The learned distribution need not match the distribution of the observed data.When relying on simulated data for training machine learning models, the issue of "domain gap" between real and synthetic data arises. Many recent works BID12 BID23 focus on bridging this domain gap, particularly for computer vision tasks. Even if we are able to tune parameters perfectly, there exists a simulation-to-real domain gap which needs to be addressed. Thus, we believe the contributions of our work are orthogonal. Learning to simulate can be seen as a meta-learning algorithm that adjusts parameters of a simulator to generate synthetic data such that a machine learning model trained on this data achieves high accuracies on validation and test sets, respectively. Given the need for large-scale data sets to feed deep learning models and the often high cost of annotation and acquisition, we believe our approach is a sensible avenue for practical applications to leverage synthetic data. Our experiments illustrate the concept and demonstrate the capability of learning to simulate on both synthetic and real data. For future work, we plan to expand the label space in our segmentation experiments, apply the algorithm to other tasks like object detection and to explore a dynamic memory of previously generated data for improving our learning to simulate procedure. A TRAFFIC SCENE MODEL Our model comprises the following elements:• A straight road of variable length.• Either an L, T or X intersection at the end of the road.• Cars of 5 different types which are spawned randomly on the straight road.• Houses of a unique type which are spawned randomly on the sides of the road.• Four different types of weather.All of these elements are tied to parameters: ρ k can be decomposed into parameters which regulate each of these objects. The scene is generated "block" by "block". A block consists of a unitary length of road with sidewalks. Buildings can be generated on both sides of the road and cars can be generated on the road. ρ k,car designates the probability of car presence in any road block. Cars are sampled block by block from a Bernouilli distribution X ∼ Bern (ρ k,car ). To determine which type of car is spawned (from our selection of 5 cars) we sample from a Categorical distribution which is determined by 5 parameters ρ k,cari where i is an integer representing the identity of the car and i ∈ [1, 5]. ρ k,house designates the probability of house presence in any road block. Houses are sampled block by block from a Bernouilli distribution X ∼ Bern (ρ k,house ).Length to intersection is sampled from a Categorical distribution determined by 10 parameters ρ k,lengthi with i ∈ [8, 18] where i denotes the length from the camera to the intersection in "block" units. Weather is sampled randomly from a Categorical distribution determined by 4 parameters φ k,weatheri where i is an integer representing the identity of the weather and i ∈ [1, 4]. L, T and X intersections are sampled randomly with equal probability.
We propose an algorithm that automatically adjusts parameters of a simulation engine to generate training data for a neural network such that validation accuracy is maximized.
{"src": "scitldr-AIC_train", "paper_id": "HJgkx2Aqt7"}