Approximate reinforcement learning: An overview <s> I. INTRODUCTION <s> From the Publisher: ::: This is the first textbook that fully explains the neuro-dynamic programming/reinforcement learning methodology, which is a recent breakthrough in the practical application of neural networks and dynamic programming to complex problems of planning, optimal decision making, and intelligent control. <s> BIB001 </s> Approximate reinforcement learning: An overview <s> I. INTRODUCTION <s> Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning. <s> BIB002 </s> Approximate reinforcement learning: An overview <s> I. INTRODUCTION <s> This paper addresses the problem of computing optimal structured treatment interruption strategies for HIV infected patients. We show that reinforcement learning may be useful to extract such strategies directly from clinical data, without the need of an accurate mathematical model of HIV infection dynamics. To support our claims, we report simulation results obtained by running a recently proposed batch-mode reinforcement learning algorithm, known as fitted Q iteration, on numerically generated data. <s> BIB003 </s> Approximate reinforcement learning: An overview <s> I. INTRODUCTION <s> Autonomous helicopter flight is widely regarded to be a highly challenging control problem. This paper presents the first successful autonomous completion on a real RC helicopter of the following four aerobatic maneuvers: forward flip and sideways roll at low speed, tail-in funnel, and nose-in funnel. Our experimental results significantly extend the state of the art in autonomous helicopter flight. We used the following approach: First we had a pilot fly the helicopter to help us find a helicopter dynamics model and a reward (cost) function. Then we used a reinforcement learning (optimal control) algorithm to find a controller that is optimized for the resulting model and reward function. More specifically, we used differential dynamic programming (DDP), an extension of the linear quadratic regulator (LQR). <s> BIB004 </s> Approximate reinforcement learning: An overview <s> I. INTRODUCTION <s> Autonomous learning is one of the hallmarks of human and animal behavior, and understanding the principles of learning will be crucial in order to achieve true autonomy in advanced machines like humanoid robots. In this paper, we examine learning of complex motor skills with human-like limbs. While supervised learning can offer useful tools for bootstrapping behavior, e.g., by learning from demonstration, it is only reinforcement learning that offers a general approach to the final trial-and-error improvement that is needed by each individual acquiring a skill. Neither neurobiological nor machine learning studies have, so far, offered compelling results on how reinforcement learning can be scaled to the high-dimensional continuous state and action spaces of humans or humanoids. Here, we combine two recent research developments on learning motor control in order to achieve this scaling. First, we interpret the idea of modular motor control by means of motor primitives as a suitable way to generate parameterized control policies for reinforcement learning. Second, we combine motor primitives with the theory of stochastic policy gradient learning, which currently seems to be the only feasible framework for reinforcement learning for humanoids. We evaluate different policy gradient methods with a focus on their applicability to parameterized motor primitives. We compare these algorithms in the context of motor primitive learning, and show that our most modern algorithm, the Episodic Natural Actor-Critic outperforms previous algorithms by at least an order of magnitude. We demonstrate the efficiency of this reinforcement learning method in the application of learning to hit a baseball with an anthropomorphic robot arm. <s> BIB005 </s> Approximate reinforcement learning: An overview <s> I. INTRODUCTION <s> This paper introduces and demonstrates a novel brain-machine interface (BMI) architecture based on the concepts of reinforcement learning (RL), coadaptation, and shaping. RL allows the BMI control algorithm to learn to complete tasks from interactions with the environment, rather than an explicit training signal. Coadaption enables continuous, synergistic adaptation between the BMI control algorithm and BMI user working in changing environments. Shaping is designed to reduce the learning curve for BMI users attempting to control a prosthetic. Here, we present the theory and in vivo experimental paradigm to illustrate how this BMI learns to complete a reaching task using a prosthetic arm in a 3-D workspace based on the user's neuronal activity. This semisupervised learning framework does not require user movements. We quantify BMI performance in closed-loop brain control over six to ten days for three rats as a function of increasing task difficulty. All three subjects coadapted with their BMI control algorithms to control the prosthetic significantly above chance at each level of difficulty. <s> BIB006 </s> Approximate reinforcement learning: An overview <s> I. INTRODUCTION <s> Markov Decision Processes (MDPs) are a mathematical framework for modeling sequential decision problems under uncertainty as well as Reinforcement Learning problems. Written by experts in the field, this book provides a global view of current research using MDPs in Artificial Intelligence. It starts with an introductory presentation of the fundamental aspects of MDPs (planning in MDPs, Reinforcement Learning, Partially Observable MDPs, Markov games and the use of non-classical criteria). Then it presents more advanced research trends in the domain and gives some concrete examples using illustrative applications. <s> BIB007
Using reinforcement learning (RL), agents (controllers) can learn how to optimally interact with complex environments (systems). RL can address problems from a variety of domains, including artificial intelligence (AI), robotics, automatic control, operations research, economics, medicine, etc. For instance, in AI, RL provides a way to build learning agents that optimize their behavior in initially unknown environments BIB002 - BIB007 . In automatic control, RL can in principle solve nonlinear and stochastic optimal control problems, without requiring a model BIB001 -  . An RL agent interacts with its environment by measuring states and applying actions according to its policy. A reward signal indicates the immediate performance, and the goal is to find an optimal policy that maximizes the value function, i.e., the cumulative long-term reward as a function of the states and possibly of the actions. Classical RL requires exact representations of value functions and policies, and is therefore limited to small, discrete problems. However, most realistic problems have variables with a large or infinite number of possible values (e.g., continuous variables); in such problems, value functions and policies must be approximated. Fueled by recent advances in approximation-based algorithms, RL has obtained impressive successes in applications such as robot control BIB005 , autonomous helicopter flight BIB004 , interfacing an animal brain with a robot arm BIB006 , simulated treatment of HIV infections BIB003 , etc. Unfortunately, a newcomer to the field of approximate RL will find it difficult to grasp, due to the intricate and extensive landscape of algorithms and results, coupled with the scarcity of survey papers BIB002 . In this overview, we take a step toward alleviating this situation, by painting a high-level picture of the field, and going in-depth into representative approaches. Starting from their exact dynamic programming (DP) and RL roots, we describe three major classes of approximate RL techniques: approximate value iteration, policy iteration, and policy search. We introduce offline algorithms, as well as online algorithms, which collect their own data by interaction with the system. We pay attention to policy gradient and actorcritic methods, as well as to simulation-based policy iteration. In closing, the various types of approaches are compared, several important ways to enhance approximate RL methods are outlined, and pointers to further reading are provided. Next, Sec. II introduces classical, exact DP and RL. Then, Sec. III discusses the need for approximation in RL and a taxonomy of approximate RL methods. Sections IV -VI present, in turn, approximate value iteration, policy iteration, and policy search. Sec. VII closes the paper with a discussion and an outlook.