Reinforcement learning in robotics: A survey <s> Reinforcement learning in the context of optimal control <s> Neural network reinforcement learning methods are described and considered as a direct approach to adaptive optimal control of nonlinear systems. These methods have their roots in studies of animal learning and in early learning control work. An emerging deeper understanding of these methods is summarized that is obtained by viewing them as a synthesis of dynamic programming and stochastic approximation methods. The focus is on Q-learning systems, which maintain estimates of utilities for all state-action pairs and make use of these estimates to select actions. The use of hybrid direct/indirect methods is briefly discussed. > <s> BIB001 </s> Reinforcement learning in robotics: A survey <s> Reinforcement learning in the context of optimal control <s> This paper describes some of the interactions of model learning algorithms and planning algorithms we have found in exploring model-based reinforcement learning. The paper focuses on how local trajectory optimizers can be used effectively with learned non-parametric models. We find that trajectory planners that are fully consistent with the learned model often have difficulty finding reasonable plans in the early stages of learning. Trajectory planners that balance obeying the learned model with minimizing cost (or maximizing reward) often do better, even if the plan is not fully consistent with the learned model. <s> BIB002 </s> Reinforcement learning in robotics: A survey <s> Reinforcement learning in the context of optimal control <s> Many control problems in the robotics field can be cast as partially observed Markovian decision problems (POMDPs), an optimal control formalism. Finding optimal solutions to such problems in general, however is known to be intractable. It has often been observed that in practice, simple structured controllers suffice for good sub-optimal control, and recent research in the artificial intelligence community has focused on policy search methods as techniques for finding sub-optimal controllers when such structured controllers do exist. Traditional model-based reinforcement learning algorithms make a certainty equivalence assumption on their learned models and calculate optimal policies for a maximum-likelihood Markovian model. We consider algorithms that evaluate and synthesize controllers under distributions of Markovian models. Previous work has demonstrated that algorithms that maximize mean reward with respect to model uncertainty leads to safer and more robust controllers. We consider briefly other performance criterion that emphasize robustness and exploration in the search for controllers, and note the relation with experiment design and active learning. To validate the power of the approach on a robotic application we demonstrate the presented learning control algorithm by flying an autonomous helicopter. We show that the controller learned is robust and delivers good performance in this real-world domain. <s> BIB003 </s> Reinforcement learning in robotics: A survey <s> Reinforcement learning in the context of optimal control <s> Autonomous helicopter flight is widely regarded to be a highly challenging control problem. As helicopters are highly unstable and exhibit complicated dynamical behavior, it is particularly difficult to design controllers that achieve high performance over a broad flight regime. While these aircraft are notoriously difficult to control, there are expert human pilots who are nonetheless capable of demonstrating a wide variety of maneuvers, including aerobatic maneuvers at the edge of the helicopter's performance envelope. In this paper, we present algorithms for modeling and control that leverage these demonstrations to build high-performance control systems for autonomous helicopters. More specifically, we detail our experiences with the Stanford Autonomous Helicopter, which is now capable of extreme aerobatic flight meeting or exceeding the performance of our own expert pilot. <s> BIB004 </s> Reinforcement learning in robotics: A survey <s> Reinforcement learning in the context of optimal control <s> Stochastic optimization arises in a wide range of problems, and as a result the ideas have been developed in different communities, creating fragmented styles in terms of notation, modeling and algorithms. Some of these variations can be explained by differences in application, as well as differences in research styles. We attempt to bridge these communities by describing how to translate notational systems, while contrasting modeling and algorithmic strategies. We also touch on differences in research styles which reflect expectations for a publishable contribution as well as the style of publication outlets used by different communities. This version of the paper has been written largely without references (more references may be added later). At this point, I think it is fair to say that all of the concepts contained in this paper have been expressed somewhere in the literature, or are at least known among groups of researchers. This article synthesizes these ideas in a coherent way, and while multiple notational systems are reviewed, there is also an attempt to develop some common notational principles that will help to foster communication across communities. This article is intended as a discussion piece and is not intended for journal publication. The goal is to help foster communication between the different communities. Constructive thoughts and comments are warmly appreciated. Since this document will evolve over time, please email your comments to powell@princeton.edu. You may put notes directly on the pdf (this is best), but if you put them in email, please include the date on the front page so I know which version you are using. <s> BIB005
Reinforcement learning is very closely related to the theory of classical optimal control, as well as dynamic programming, stochastic programming, simulation-optimization, stochastic search, and optimal stopping BIB005 . Both reinforcement learning and optimal control address the problem of finding an optimal policy (often also called the controller or control policy) that optimizes an objective function (i.e. the accumulated cost or reward), and both rely on the notion of a system being described by an underlying set of states, controls, and a plant or model that describes transitions between states. However, optimal control assumes perfect knowledge of the system's description in the form of a model (i.e. a function T that describes what the next state of the robot will be given the current state and action). For such models, optimal control ensures strong guarantees which, nevertheless, often break down due to model and computational approximations. In contrast, reinforcement learning operates directly on measured data and rewards from interaction with the environment. Reinforcement learning research has placed great focus on addressing cases which are analytically intractable using approximations and data-driven techniques. One of the most important approaches to reinforcement learning within robotics centers on the use of classical optimal control techniques (e.g. linear-quadratic regulation and differential dynamic programming (DDP)) to system models learned via repeated interaction with the environment BIB002 BIB003 BIB004 . A concise discussion of viewing reinforcement learning as "adaptive optimal control" is presented by BIB001 .