Meta-learning algorithms learn to acquire new tasks more quickly from past experience. In the context of reinforcement learning, meta-learning algorithms can acquire reinforcement learning procedures to solve new problems more efficiently by utilizing experience from prior tasks. The performance of meta-learning algorithms depends on the tasks available for meta-training: in the same way that supervised learning generalizes best to test points drawn from the same distribution as the training points, meta-learning methods generalize best to tasks from the same distribution as the meta-training tasks. In effect, meta-reinforcement learning offloads the design burden from algorithm design to task design. If we can automate the process of task design as well, we can devise a meta-learning algorithm that is truly automated. In this work, we take a step in this direction, proposing a family of unsupervised meta-learning algorithms for reinforcement learning. We motivate and describe a general recipe for unsupervised meta-reinforcement learning, and present an instantiation of this approach. Our conceptual and theoretical contributions consist of formulating the unsupervised meta-reinforcement learning problem and describing how task proposals based on mutual information can in principle be used to train optimal meta-learners. Our experimental results indicate that unsupervised meta-reinforcement learning effectively acquires accelerated reinforcement learning procedures without the need for manual task design and significantly exceeds the performance of learning from scratch. Reusing past experience for faster learning of new tasks is a key challenge for machine learning. Meta-learning methods achieve this by using past experience to explicitly optimize for rapid adaptation (Mishra et al., 2017; Snell et al., 2017; Schmidhuber, 1987; Finn et al., 2017a; Wang et al., 2016; Al-Shedivat et al., 2017) . In the context of reinforcement learning (RL), meta-reinforcement learning (meta-RL) algorithms can learn to solve new RL tasks more quickly through experience on past tasks (Duan et al., 2016b; Finn et al., 2017a) . Typical meta-RL algorithms assume the ability to sample from a pre-specified task distribution, and these algorithms learn to solve new tasks drawn from this distribution very quickly. However, specifying a task distribution is tedious and requires a significant amount of supervision (Finn et al., 2017b; Duan et al., 2016b ) that may be difficult to provide for large, real-world problem settings. The performance of meta-learning algorithms critically depends on the meta-training task distribution, and meta-learning algorithms generalize best to new tasks which are drawn from the same distribution as the meta-training tasks . In effect, meta-RL offloads much of the design burden from algorithm design to designing a sufficiently broad and relevant distribution of meta-training tasks. While this offloading helps in acquiring representations for fast adaptation to the specified task distribution, specifying this is often tedious and challenging. A natural question is whether we can do away with manual task design and develop meta-RL algorithms that learn only from unsupervised environment interaction. In this paper, we take an initial step toward the formalization and design of such methods. Our goal is to automate the meta-training process by removing the need for hand-designed metatraining tasks. To that end, we introduce unsupervised meta-RL: meta-learning from a task distribution that is acquired automatically, rather than requiring manual design of the meta-training tasks. Unsupervised meta-RL methods must solve two difficult problems together: meta-RL with broad task distributions, and unsupervised exploration for proposing a wide variety of tasks for meta-learning. Since the assumptions of our method differ fundamentally from prior meta-RL methods (we do not assume access to hand-specified meta-training tasks that use human-specified reward functions), the best points of comparison for our approach are learning meta-test tasks entirely from scratch with conventional RL algorithms. Our method can also be thought of as automatically acquiring an environment-specific learning procedure for deep neural network policies, somewhat related to data-driven initialization procedures explored in supervised learning (Krähenbühl et al., 2015; Hsu et al., 2018) . The primary contributions of our work are to propose a framework for unsupervised meta-RL; to sketch out a family of unsupervised meta-RL algorithms; to provide a theoretical derivation that allows us to reason about the optimality of unsupervised meta-RL methods in terms of mutual information objectives; and to describe an instantiation of an algorithm from this family that builds on a recently proposed procedure for unsupervised exploration and modelagnostic meta-learning (MAML) (Finn et al., 2017a) . In addition to our theoretical derivations, we provide an empirical evaluation that studies the performance of two variants of our approach on simulated control tasks. Our experimental evaluation shows that, for a variety of tasks, unsupervised meta-RL can effectively acquire RL procedures that perform significantly better than standard RL methods that learn from scratch, without requiring additional task knowledge. We presented an unsupervised approach to meta-RL, where meta-learning is used to acquire an efficient RL procedure without requiring hand-specified task distributions for meta-training. This approach accelerates RL without relying on the manual supervision required for conventional metalearning algorithms. We provide a theoretical derivation that argues that task proposals based on mutual information maximization can provide for a minimum worst-case regret meta-learner, under certain assumptions. We then instantiate an approximation to the theoretically-motivated method by building on recently developed unsupervised task proposal and meta-learning algorithms. Our experiments indicate that unsupervised meta-RL can accelerate learning on a range of tasks, outperforming learning from scratch and often matching the performance of meta-learning from hand-specified task distributions. As our work is the first foray into unsupervised meta-RL, our approach opens a number of questions about unsupervised meta-learning algorithms. One limitation of our analysis is that it only considers deterministic dynamics, and only considers task distributions where posterior sampling is optimal. Extending our analysis to stochastic dynamics and more realistic task distributions may allow unsupervised meta-RL to acquire learning algorithms that can explore and adapt more intelligently, and more effectively solve real-world tasks.
Meta-learning on self-proposed task distributions to speed up reinforcement learning without human specified task distributions 
{"src": "scitldr-AIC_train", "paper_id": "S1et1lrtwr"}