Learning to Optimize is a recently proposed framework for learning optimization algorithms using reinforcement learning. In this paper, we explore learning an optimization algorithm for training shallow neural nets. Such high-dimensional stochastic optimization problems present interesting challenges for existing reinforcement learning algorithms. We develop an extension that is suited to learning optimization algorithms in this setting and demonstrate that the learned optimization algorithm consistently outperforms other known optimization algorithms even on unseen tasks and is robust to changes in stochasticity of gradients and the neural net architecture. More specifically, we show that an optimization algorithm trained with the proposed method on the problem of training a neural net on MNIST generalizes to the problems of training neural nets on the Toronto Faces Dataset, CIFAR-10 and CIFAR-100. Machine learning is centred on the philosophy that learning patterns automatically from data is generally better than meticulously crafting rules by hand. This data-driven approach has delivered: today, machine learning techniques can be found in a wide range of application areas, both in AI and beyond. Yet, there is one domain that has conspicuously been left untouched by machine learning: the design of tools that power machine learning itself.One of the most widely used tools in machine learning is optimization algorithms. We have grown accustomed to seeing an optimization algorithm as a black box that takes in a model that we design and the data that we collect and outputs the optimal model parameters. The optimization algorithm itself largely stays static: its design is reserved for human experts, who must toil through many rounds of theoretical analysis and empirical validation to devise a better optimization algorithm. Given this state of affairs, perhaps it is time for us to start practicing what we preach and learn how to learn.Recently, BID20 and BID0 introduced two different frameworks for learning optimization algorithms. Whereas BID0 focuses on learning an optimization algorithm for training models on a particular task, BID20 sets a more ambitious objective of learning an optimization algorithm for training models that is task-independent. We study the latter paradigm in this paper and develop a method for learning an optimization algorithm for high-dimensional stochastic optimization problems, like the problem of training shallow neural nets.Under the "Learning to Optimize" framework proposed by BID20 , the problem of learning an optimization algorithm is formulated as a reinforcement learning problem. We consider the general structure of an unconstrained continuous optimization algorithm, as shown in Algorithm 1. In each iteration, the algorithm takes a step ∆x and uses it to update the current iterate x (i) . In hand-engineered optimization algorithms, ∆x is computed using some fixed formula φ that depends on the objective function, the current iterate and past iterates. Often, it is simply a function of the current and past gradients. Different choices of φ yield different optimization algorithms and so each optimization algorithm is essentially characterized by its update formula φ. Hence, by learning φ, we can learn an optimization algorithm. BID20 observed that an optimization algorithm can be viewed as a Markov decision process (MDP), where the state includes the current iterate, the action is the step vector ∆x end if x (i) ← x (i−1) + ∆x end for and the policy is the update formula φ. Hence, the problem of learning φ simply reduces to a policy search problem.In this paper, we build on the method proposed in BID20 and develop an extension that is suited to learning optimization algorithms for high-dimensional stochastic problems. We use it to learn an optimization algorithm for training shallow neural nets and show that it outperforms popular hand-engineered optimization algorithms like ADAM BID18 , AdaGrad BID10 and RMSprop BID28 and an optimization algorithm learned using the supervised learning method proposed in BID0 . Furthermore, we demonstrate that our optimization algorithm learned from the experience of training on MNIST generalizes to training on other datasets that have very dissimilar statistics, like the Toronto Faces Dataset, CIFAR-10 and CIFAR-100. In this paper, we presented a new method for learning optimization algorithms for high-dimensional stochastic problems. We applied the method to learning an optimization algorithm for training shallow neural nets. We showed that the algorithm learned using our method on the problem of training a neural net on MNIST generalizes to the problems of training neural nets on unrelated tasks/datasets like the Toronto Faces Dataset, CIFAR-10 and CIFAR-100. We also demonstrated that the learned optimization algorithm is robust to changes in the stochasticity of gradients and the neural net architecture.
We learn an optimization algorithm that generalizes to unseen tasks
{"src": "scitldr-AIC_train", "paper_id": "BkM27IxR-"}