Reinforcement learning in robotics: A survey <s> Policy search <s> Direct policy search is a practical way to solve reinforcement learning problems involving continuous state and action spaces. The goal becomes finding policy parameters that maximize a noisy objective function. The Pegasus method converts this stochastic optimization problem into a deterministic one, by using fixed start states and fixed random number sequences for comparing policies (Ng & Jordan, 1999). We evaluate Pegasus, and other paired comparison methods, using the mountain car problem, and a difficult pursuer-evader problem. We conclude that: (i) Paired tests can improve performance of deterministic and stochastic optimization procedures. (ii) Our proposed alternatives to Pegasus can generalize better, by using a different test statistic, or changing the scenarios during learning. (iii) Adapting the number of trials used for each policy comparison yields fast and robust learning. <s> BIB001 </s> Reinforcement learning in robotics: A survey <s> Policy search <s> Many control problems in the robotics field can be cast as partially observed Markovian decision problems (POMDPs), an optimal control formalism. Finding optimal solutions to such problems in general, however is known to be intractable. It has often been observed that in practice, simple structured controllers suffice for good sub-optimal control, and recent research in the artificial intelligence community has focused on policy search methods as techniques for finding sub-optimal controllers when such structured controllers do exist. Traditional model-based reinforcement learning algorithms make a certainty equivalence assumption on their learned models and calculate optimal policies for a maximum-likelihood Markovian model. We consider algorithms that evaluate and synthesize controllers under distributions of Markovian models. Previous work has demonstrated that algorithms that maximize mean reward with respect to model uncertainty leads to safer and more robust controllers. We consider briefly other performance criterion that emphasize robustness and exploration in the search for controllers, and note the relation with experiment design and active learning. To validate the power of the approach on a robotic application we demonstrate the presented learning control algorithm by flying an autonomous helicopter. We show that the controller learned is robust and delivers good performance in this real-world domain. <s> BIB002 </s> Reinforcement learning in robotics: A survey <s> Policy search <s> This paper studies the reinforcement learning (RL) method for central pattern generators (CPG) that generates stable rhythmic movements such as biped locomotion. RL for biped locomotion is very difficult, since the biped robot is highly unstable and the system has continuous state and action spaces with a high degree of freedom. In order to deal with RL for CPG, we propose a new RL method called the CPG-actor-critic method. We applied this method to the RL for the biped robot. The computer simulation showed that our RL method was able to train the CPG such that the biped robot walk stably. <s> BIB003 </s> Reinforcement learning in robotics: A survey <s> Policy search <s> Helicopters have highly stochastic, nonlinear, dynamics, and autonomous helicopter flight is widely regarded to be a challenging control problem. As helicopters are highly unstable at low speeds, it is particularly difficult to design controllers for low speed aerobatic maneuvers. In this paper, we describe a successful application of reinforcement learning to designing a controller for sustained inverted flight on an autonomous helicopter. Using data collected from the helicopter in flight, we began by learning a stochastic, nonlinear model of the helicopter’s dynamics. Then, a reinforcement learning algorithm was applied to automatically learn a controller for autonomous inverted hovering. Finally, the resulting controller was successfully tested on our autonomous helicopter platform. <s> BIB004 </s> Reinforcement learning in robotics: A survey <s> Policy search <s> This paper presents a machine learning approach to optimizing a quadrupedal trot gait for forward speed. Given a parameterized walk designed for a specific robot, we propose using a form of policy gradient reinforcement learning to automatically search the set of possible parameters with the goal of finding the fastest possible walk. We implement and test our approach on a commercially available quadrupedal robot platform, namely the Sony Aibo robot. After about three hours of learning, all on the physical robots and with no human intervention other than to change the batteries, the robots achieved a gait faster than any previously known gait known for the Aibo, significantly outperforming a variety of existing hand-coded and learned solutions. <s> BIB005 </s> Reinforcement learning in robotics: A survey <s> Policy search <s> In this paper, we present our design and experiments of a planar biped robot ("RunBot") under pure reflexive neuronal control. The goal of this study is to combine neuronal mechanisms with biomechanics to obtain very fast speed and the on-line learning of circuit parameters. Our controller is built with biologically inspired sensor- and motor-neuron models, including local reflexes and not employing any kind of position or trajectory-tracking control algorithm. Instead, this reflexive controller allows RunBot to exploit its own natural dynamics during critical stages of its walking gait cycle. To our knowledge, this is the first time that dynamic biped walking is achieved using only a pure reflexive controller. In addition, this structure allows using a policy gradient reinforcement learning algorithm to tune the parameters of the reflexive controller in real-time during walking. This way RunBot can reach a relative speed of 3.5 leg-lengths per second after a few minutes of online learning, which is faster than that of any other biped robot, and is also comparable to the fastest relative speed of human walking. In addition, the stability domain of stable walking is quite large supporting this design strategy. <s> BIB006 </s> Reinforcement learning in robotics: A survey <s> Policy search <s> (Please note: The main body of this paper is written in Japanese) When humans interact in a social context, there are many factors apart from the actual communication that need to be considered. Pr ... <s> BIB007 </s> Reinforcement learning in robotics: A survey <s> Policy search <s> This work describes the development of a model-free reinforcement learning-based control methodology for the heaving plate, a laboratory experimental fluid system that serves as a model of flapping flight. Through an optimized policy gradient algorithm, we were able to demonstrate rapid convergence (requiring less than 10 minutes of experiments) to a stroke form which maximized the propulsive efficiency of this very complicated fluid-dynamical system. This success was due in part to an improved sampling distribution and carefully selected policy parameterization, both motivated by a formal analysis of the signal-to-noise ratio of policy gradient algorithms. The resulting optimal policy provides insight into the behavior of the fluid system, and the effectiveness of the learning strategy suggests a number of exciting opportunities for machine learning control of fluid dynamics. <s> BIB008
The primal formulation of the problem in terms of policy rather then value offers many features relevant to robotics. It allows for a natural integration of expert knowledge, e.g. through both structure and initializations of the policy. It allows domain-appropriate prestructuring of the policy in an approximate form without changing the original problem. Optimal policies often have many fewer parameters than optimal value functions. For example, in linear quadratic control, the value function has quadratically many parameters in the dimensionality of the state variables while the policy requires only linearly many parameters. Local search in policy space can directly lead to good results as exhibited by early hill-climbing approaches  , as well as more recent successes (see Table 2 ). Additional constraints can be incorporated naturally, e.g. regularizing the change in the path distribution. As a result, policy search often appears more natural to robotics. Nevertheless, policy search has been considered the harder problem for a long time as the optimal solution cannot directly be determined from Equations (1)-  while the solution of the dual problem leveraging Bellman principle of optimality  enables dynamicprogramming-based solutions. Notwithstanding this, in robotics, policy search has recently become an important alternative to value-functionbased methods due to better scalability as well as the convergence problems of approximate value-function methods (see Sections 2.3 and 4.2). Most policy-search methods optimize locally around existing policies π , parametrized by a set of policy parameters θ i , by computing changes in the policy parameters θ i that will increase the expected return and results in iterative updates of the form The computation of the policy update is the key step here and a variety of updates have been proposed ranging from pairwise comparisons BIB001 BIB004 over gradient estimation using finite policy differences BIB003 BIB005 BIB007  BIB006 BIB008 , and general stochastic optimization methods (such as Nelder-Mead BIB002 , cross-entropy  ) and population-based methods  ) to approaches coming from optimal control such as DDP (Atkeson, 1998) Table 1 . Value-function-based reinforcement learning methods employed for robotic tasks (both average and discounted reward cases) and associated publications.