In this work, we aim to solve data-driven optimization problems, where the goal is to find an input that maximizes an unknown score function given access to a dataset of input, score pairs. Inputs may lie on extremely thin manifolds in high-dimensional spaces, making the optimization prone to falling-off the manifold. Further, evaluating the unknown function may be expensive, so the algorithm should be able to exploit static, offline data. We propose model inversion networks (MINs) as an approach to solve such problems. Unlike prior work, MINs scale to extremely high-dimensional input spaces and can efficiently leverage offline logged datasets for optimization in both contextual and non-contextual settings. We show that MINs can also be extended to the active setting, commonly studied in prior work, via a simple, novel and effective scheme for active data collection. Our experiments show that MINs act as powerful optimizers on a range of contextual/non-contextual, static/active problems including optimization over images and protein designs and learning from logged bandit feedback. Data-driven optimization problems arise in a range of domains: from protein design (Brookes et al., 2019) to automated aircraft design (Hoburg & Abbeel, 2012) , from the design of robots (Liao et al., 2019) to the design of neural net architectures (Zoph & Le, 2017) and learning from logged feedback, such as optimizing user preferences in recommender systems. Such problems require optimizing unknown reward or score functions using previously collected data consisting of pairs of inputs and corresponding score values, without direct access to the score function being optimized. This can be especially challenging when valid inputs lie on a low-dimensional manifold in the space of all inputs, e.g., the space of valid aircraft designs or valid images. Existing methods to solve such problems often use derivative-free optimization (Snoek et al.) . Most of these techniques require active data collection where the unknown function is queried at new inputs. However, when function evaluation involves a complex real-world process, such as testing a new aircraft design or evaluating a new protein, such active methods can be very expensive. On the other hand, in many cases there is considerable prior data -existing aircraft and protein designs, and advertisements and user click rates, etc. -that could be leveraged to solve the optimization problem. In this work, our goal is to develop an optimization approach to solve such optimization problems that can (1) readily operate on high-dimensional inputs comprising a narrow, low-dimensional manifold, such as natural images, (2) readily utilize offline static data, and (3) learn with minimal active data collection if needed. We can define this problem setting formally as the optimization problem where the function f (x) is unknown, and we have access to a dataset D = {(x 1 , y 1 ), . . . , (x N , y N )}, where y i denotes the value f (x i ). If no further data collection is possible, we call this the data-driven model-based optimization setting. This can also be extended to the contextual setting, where the aim is to optimize the expected score function value across a context distribution. That is, where π maps contexts c to inputs x, such that the expected score under the context distribution p 0 (c) is optimized. As before, f (c, x) is unknown and we have access to a dataset D = {(c i , , where y i is the value of f (c i , x i ). Such contextual problems with logged datasets have been studied in the context of contextual bandits Joachims et al., 2018) . A simple way to approach these model-based optimization problems is to train a proxy function f θ (x) or f θ (c, x), with parameters θ, to approximate the true score, using the dataset D. However, directly using f θ (x) in place of the true function f (x) in Equation (1) generally works poorly, because the optimizer will quickly find an input x for which f θ (x) outputs an erroneously large value. This issue is especially severe when the inputs x lie on a narrow manifold in a high-dimensional space, such as the set of natural images (Zhu et al., 2016) . The function f θ (x) is only valid near the training distribution, and can output erroneously large values when queried at points chosen by the optimizer. Prior work has sought to addresses this issue by using uncertainty estimation and Bayesian models (Snoek et al., 2015) for f θ (x), as well as active data collection (Snoek et al.) . However, explicit uncertainty estimation is difficult when the function f θ (x) is very complex or when x is high-dimensional. Instead of learning f θ (x), we propose to learn the inverse function, mapping from values y to corresponding inputs x. This inverse mapping is one-to-many, and therefore requires a stochastic mapping, which we can express as f −1 θ (y, z) → x, where z is a random variable. We term such models model inversion networks (MINs). MINs provide us with a number of desirable properties: they can utilize static datasets, handle high-dimensional input spaces such as images, can handle contextual problems, and can accommodate both static datasets and active data collection. We discuss how to design simple active data collection methods for MINs, leverage advances in deep generative modeling (Goodfellow et al.; Brock et al., 2019) , and scale to very high-dimensional input spaces. We experimentally demonstrate MINs in a range of settings, showing that they outperform prior methods on high-dimensional input spaces, perform competitively to Bayesian optimization methods on tasks with active data collection and lower-dimensional inputs, and substantially outperform prior methods on contextual optimization from logged data (Swaminathan & Joachims, a) . Prior work has usually considered MBO in the active or "onpolicy" setting, where the algorithm actively queries data as it learns. In this work, we introduced the data-driven MBO problem statement and devised a method to perform optimization in such scenarios. This is important in settings where data collection is expensive and where abundant datasets exist, for example, protein design, aircraft design and drug design. Further, MINs define a family of algorithms that show promising results on MBO problems on extremely large input spaces. While MINs scale to high-dimensional tasks such as model-based optimization over images, and are performant in both contextual and non-contextual settings, we believe there are a number of interesting open questions for future work. The interaction between active data collection and reweighting should be investigated in more detail, and poses interesting consequences for MBO, bandits and reinforcement learning. Better and more principled inference procedures are also a direction for future work. Another avenue is to study various choices of training objectives in MIN optimization. In this section, we show that the inference scheme described in Equation 4, Section 3.2 emerges as a deterministic relaxation of the probabilistic inference scheme described below. We re-iterate that in Section 3.2, a singleton x * is the output of optimization, however the procedure can be motivated from the perspective of the following probabilistic inference scheme. Let p(x|y) denote a stochastic inverse map, and let p f (y|x) be a probabilistic forward map. Consider the following optimization problem: arg max where p θ (x|y) is the probability distribution induced by the learned inverse map (in our case, this corresponds to the distribution of f −1 θ (y, z) induced due to randomness in z ∼ p 0 (·)), p f (x|y) is the learned forward map, H is Shannon entropy, and D is KL-divergence measure between two distributions. In Equation 4, maximization is carried out over the input y to the inverse-map, and the input z which is captured inp in the above optimization problem, i.e. maximization over z in Equation 4 is equivalent to choosingp subject to the choice of singleton/ Dirac-deltap. The Lagrangian is given by: In order to derive Equation 4, we restrictp to the Dirac-delta distribution generated by querying the learned inverse map f −1 θ at a specific value of z. Now note that the first term in the Lagrangian corresponds to maximizing the "reconstructed"ŷ similarly to the first term in Equation 4. If p f is assumed to be a Gaussian random variable with a fixed variance, then log p f (ŷ|x) = −||ŷ − µ(x)|| Finally, in order to obtain the log p 0 (z) term, note that, D(p(x|y), p θ (x|y)) ≤ D(δ z (·), p 0 (·)) = − log p 0 (z) (by the data processing inequality for KL-divergence). Hence, constraining log p 0 (z) instead of the true divergence gives us a lower bound on L. Maximizing this lower bound (which is the same as Equation 4) hence also maximizes the true Lagrangian L.
We propose a novel approach to solve data-driven model-based optimization problems in both passive and active settings that can scale to high-dimensional input spaces.
{"src": "scitldr-AIC_train", "paper_id": "SklsBJHKDS"}