Approximate reinforcement learning: An overview <s> Comparison of approaches: <s> From the Publisher: ::: This is the first textbook that fully explains the neuro-dynamic programming/reinforcement learning methodology, which is a recent breakthrough in the practical application of neural networks and dynamic programming to complex problems of planning, optimal decision making, and intelligent control. <s> BIB001 </s> Approximate reinforcement learning: An overview <s> Comparison of approaches: <s> Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning. <s> BIB002 </s> Approximate reinforcement learning: An overview <s> Comparison of approaches: <s> We develop a methodological framework and present a few different ways in which dynamic programming and compact representations can be combined to solve large scale stochastic control problems. In particular, we develop algorithms that employ two types of feature-based compact representations; that is, representations that involve feature extraction and a relatively simple approximation architecture. We prove the convergence of these algorithms and provide bounds on the approximation error. As an example, one of these algorithms is used to generate a strategy for the game of Tetris. Furthermore, we provide a counter-example illustrating the difficulties of integrating compact representations with dynamic programming, which exemplifies the shortcomings of certain simple approaches. <s> BIB003 </s> Approximate reinforcement learning: An overview <s> Comparison of approaches: <s> Autonomous learning is one of the hallmarks of human and animal behavior, and understanding the principles of learning will be crucial in order to achieve true autonomy in advanced machines like humanoid robots. In this paper, we examine learning of complex motor skills with human-like limbs. While supervised learning can offer useful tools for bootstrapping behavior, e.g., by learning from demonstration, it is only reinforcement learning that offers a general approach to the final trial-and-error improvement that is needed by each individual acquiring a skill. Neither neurobiological nor machine learning studies have, so far, offered compelling results on how reinforcement learning can be scaled to the high-dimensional continuous state and action spaces of humans or humanoids. Here, we combine two recent research developments on learning motor control in order to achieve this scaling. First, we interpret the idea of modular motor control by means of motor primitives as a suitable way to generate parameterized control policies for reinforcement learning. Second, we combine motor primitives with the theory of stochastic policy gradient learning, which currently seems to be the only feasible framework for reinforcement learning for humanoids. We evaluate different policy gradient methods with a focus on their applicability to parameterized motor primitives. We compare these algorithms in the context of motor primitive learning, and show that our most modern algorithm, the Episodic Natural Actor-Critic outperforms previous algorithms by at least an order of magnitude. We demonstrate the efficiency of this reinforcement learning method in the application of learning to hit a baseball with an anthropomorphic robot arm. <s> BIB004
A general, qualitative comparison of the classes of algorithms introduced is provided next. In the offline case, policy iteration often converges in a small number of iterations BIB002 Sec. 4.3] , possibly smaller than value iteration. However, this does not mean that policy iteration is computationally less demanding, since policy evaluation is a difficult problem in itself, and must be solved at every policy iteration. In certain conditions, approximate value iteration provably converges to a unique solution BIB003 , whereas this is more difficult to guarantee for approximate policy iteration, which generally only converges to a sequence of policies that all provide a guaranteed performance BIB001 . Offline algorithms like fitted Q-iteration and least-squares policy iteration typically exploit data more efficiently than gradient-based, online algorithms such as approximate Qlearning and approximate SARSA; this comes at the cost of increased computational complexity. Interspersing offline updates with online sample collection leads to 'pseudo-online', yet still data-efficient, algorithms (see the 'Outlook' below for other ways to improve data efficiency). Within the approximate policy search class, policy gradient and actor-critic algorithms have useful convergence guarantees and moderate computational demands. They can be applied when an appropriate, differentiable policy parametrization can be found, which makes them very suitable to certain classes of problems such as robot control BIB004 . However, their convergence point is only locally optimal, which is a disadvantage with respect to value and policy iteration, which do provide global performance guarantees. Even when a good policy parametrization cannot be obtained, approximate policy search can still be useful in its gradient-free, global optimization forms, which can however incur larger computational costs. An important benefit of policy search is that, in contrast to value and policy iteration, it can handle continuous action spaces without difficulty.