Reinforcement learning in robotics: A survey <s> Approach <s> Model-Based Control of a Robot Manipulator presents the first integrated treatment of many of the most important recent developments in using detailed dynamic models of robots to improve their control. The authors' work on automatic identification of kinematic and dynamic parameters, feedforward position control, stability in force control, and trajectory learning has significant implications for improving performance in future robot systems. All of the main ideas discussed in this book have been validated by experiments on a direct-drive robot arm.The book addresses the issues of building accurate robot models and of applying them for high performance control. It first describes how three sets of models - the kinematic model of the links and the inertial models of the links and of rigid-body loads - can be obtained automatically using experimental data. These models are then incorporated into position control, single trajectory learning, and force control. The MIT Serial Link Direct Drive Arm, on which these models were developed and applied to control, is one of the few manipulators currently suitable for testing such concepts.Contents: Introduction. Direct Drive Arms. Kinematic Calibration. Estimation of Load Inertial Parameters. Estimation of Link Inertial Parameters. Feedforward and Computed Torque Control. Model-Based Robot Learning. Dynamic Stability Issues in Force Control. Kinematic Stability Issues in Force Control. Conclusion.Chae An is Research Staff Member, IBM T.J. Watson Research Center, Christopher Atkeson is an Assistant Professor and John Hollerbach is an Associate Professor in the MIT Department of Brain and Cognitive Sciences and the MIT Artificial Intelligence Laboratory. Model-Based Control of a Robot Manipulator is included in the Artificial Intelligence Series edited by Patrick Winston and Michael Brady. <s> BIB001 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Abstract This paper describes a general approach for automatically programming a behavior-based robot. New behaviors are learned by trial and error using a performance feedback function as reinforcement. Two algorithms for behavior learning are described that combine Q learning, a well-known scheme for propagating reinforcement values temporally across actions, with statistical clustering and Hamming distance, two ways of propagating reinforcement values spatially across states. A real behavior-based robot called OBELIX is described that learns several component behaviors in an example task involving pushing boxes. A simulator for the box pushing task is also used to gather data on the learning techniques. A detailed experimental study using the real robot and the simulator suggests two conclusions. 1. (1) The learning techniques are able to learn the individual behaviors, sometimes outperforming a handcoded program. 2. (2) Using a behavior-based architecture speeds up reinforcement learning by converting the problem of learning a complex task into that of learning a simpler set of special-purpose reactive subtasks. <s> BIB002 </s> Reinforcement learning in robotics: A survey <s> Approach <s> In the Real-Time Learning Laboratory at GTE Laboratories, machine learning algorithms are being implemented on hardware testbeds. A modified connectionist actor-critic system has been applied to a ball balancing task. The system learns to balance a ball on a beam in less than 5 min and maintains the balance. A ball can roll along a few inches of a track on a flat metal beam, which an electric motor can rotate. A computer learning system running on a PC senses the position of the ball and the angular position of the beam. The system learns to prevent the ball from reaching either end of the beam. The system has shown to be robust through sensor noise and mechanical changes; it has also generated many interesting questions for future research. > <s> BIB003 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Issues involved in implementing robot learning for a challenging dynamic task are explored in this article, using a case study from robot juggling. We use a memory-based local modeling approach (locally weighted regression) to represent a learned model of the task to be performed. Statistical tests are given to examine the uncertainty of a model, to optimize its prediction quality, and to deal with noisy and corrupted data. We develop an exploration algorithm that explicitly deals with prediction accuracy requirements during exploration. Using all these ingredients in combination with methods from optimal control, our robot achieves fast real-time learning of the task within 40 to 100 trials. > <s> BIB004 </s> Reinforcement learning in robotics: A survey <s> Approach <s> The pitfalls of naive robot simulations have been recognised for areas such as evolutionary robotics. It has been suggested that carefully validated simulations with a proper treatment of noise may overcome these problems. This paper reports the results of experiments intended to test some of these claims. A simulation was constructed of a two-wheeled Khepera robot with IR and ambient light sensors. This included detailed mathematical models of the robot-environment interaction dynamics with empirically determined parameters. Artificial evolution was used to develop recurrent dynamical network controllers for the simulated robot, for obstacle-avoidance and light-seeking tasks, using different levels of noise in the simulation. The evolved controllers were down-loaded onto the real robot and the correspondence between behaviour in simulation and in reality was tested. The level of correspondence varied according to how much noise was used in the simulation, with very good results achieved when realistic quantities were applied. It has been demonstrated that it is possible to develop successful robot controllers in simulation that generate almost identical behaviours in reality, at least for a particular class of robot-environment interaction dynamics. <s> BIB005 </s> Reinforcement learning in robotics: A survey <s> Approach <s> By now it is widely accepted that learning a task from scratch, i.e., without any prior knowledge, is a daunting undertaking. Humans, however, rarely attempt to learn from scratch. They extract initial biases as well as strategies how to approach a learning problem from instructions and/or demonstrations of other humans. For teaming control, this paper investigates how learning from demonstration can be applied in the context of reinforcement learning. We consider priming the Q-function, the value function, the policy, and the model of the task dynamics as possible areas where demonstrations can speed up learning. In general nonlinear learning problems, only model-based reinforcement learning shows significant speed-up after a demonstration, while in the special case of linear quadratic regulator (LQR) problems, all methods profit from the demonstration. In an implementation of pole balancing on a complex anthropomorphic robot arm, we demonstrate that, when facing the complexities of real signal processing, model-based reinforcement learning offers the most robustness for LQR problems. Using the suggested methods, the robot learns pole balancing in just a single trial after a 30 second long demonstration of the human instructor. <s> BIB006 </s> Reinforcement learning in robotics: A survey <s> Approach <s> The behaviour of reinforcement learning (RL) algorithms is best understood in completely observable, finite state- and action-space, discrete-time controlled Markov-chains. Robot-learning domains, on the other hand, are inherently infinite both in time and space, and moreover they are only partially observable. In this article we suggest a systematic design method whose motivation comes from the desire to transform the task-to-be-solved into a finite-state, discrete-time, “approximately” Markovian task, which is completely observable too. The key idea is to break up the problem into subtasks and design controllers for each of the subtasks. Then operating conditions are attached to the controllers (together the controllers and their operating conditions which are called modules) and possible additional features are designed to facilitate observability. A new discrete time-counter is introduced at the “module-level” that clicks only when a change in the value of one of the features is observed. The approach was tried out on a real-life robot. Several RL algorithms were compared and it was found that a model-based approach worked best. The learnt switching strategy performed equally well as a handcrafted version. Moreover, the learnt strategy seemed to exploit certain properties of the environment which could not have been seen in advance, which predicted the promising possibility that a learnt controller might overperform a handcrafted switching strategy in the future. <s> BIB007 </s> Reinforcement learning in robotics: A survey <s> Approach <s> This paper presents some results from a study of biped dynamic walking using reinforcement learning. During this study a hardware biped robot was built, a new reinforcement learning algorithm as well as a new learning architecture were developed. The biped learned dynamic walking without any previous knowledge about its dynamic model. The self scaling reinforcement (SSR) learning algorithm was developed in order to deal with the problem of reinforcement learning in continuous action domains. The learning architecture was developed in order to solve complex control problems. It uses different modules that consist of simple controllers and small neural networks. The architecture allows for easy incorporation of new modules that represent new knowledge, or new requirements for the desired task. <s> BIB008 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Abstract This paper addresses adaptive control architectures for systems that respond autonomously to changing tasks. Such systems often have many sensory and motor alternatives and behavior drawn from these produces varying quality solutions. The objective is then to ground behavior in control laws which, combined with resources, enumerate closed-loop behavioral alternatives. Use of such controllers leads to analyzable and predictable composite systems, permitting the construction of abstract behavioral models. Here, discrete event system and reinforcement learning techniques are employed to constrain the behavioral alternatives and to synthesize behavior on-line. To illustrate this, a quadruped robot learning a turning gait subject to safety and kinematic constraints is presented. <s> BIB009 </s> Reinforcement learning in robotics: A survey <s> Approach <s> We present work on a six-legged walking machine that uses a hierarchical version of Q-learning (HQL) to learn both the elementary swing and stance movements of individual legs as well as the overall coordination scheme to perform forward movements. The architecture consists of a hierarchy of local controllers implemented in layers. The lowest layer consists of control modules performing elementary actions, like moving a leg up, down, left or right to achieve the elementary swing and stance motions for individual legs. The next level consists of controllers that learn to perform more complex tasks like forward movement by using the previously learned, lower level modules. On the third the highest layer in the architecture presented here the previously learned complex movements are themselves reused to achieve goals in the environment using external sensory input. The work is related to similar, although simulation-based, work by Lin (1993) on hierarchical reinforcement learning and Singh (1994) on compositional Q-learning. We report on the HQL architecture as well as on its implementation on the walking machine SIR ARTHUR. Results from experiments carried out on the real robot are reported to show the applicability of the HQL approach to real world robot problems. <s> BIB010 </s> Reinforcement learning in robotics: A survey <s> Approach <s> This paper describes a formulation of reinforcement learning that enables learning in noisy, dynamic environments such as in the complex concurrent multi-robot learning domain. The methodology involves minimizing the learning space through the use of behaviors and conditions, and dealing with the credit assignment problem through shaped reinforcement in the form of heterogeneous reinforcement functions and progress estimators. We experimentally validate the approach on a group of four mobile robots learning a foraging task. <s> BIB011 </s> Reinforcement learning in robotics: A survey <s> Approach <s> We discuss Hinton's (1989) relative payoff procedure (RPP), a static reinforcement learning algorithm whose foundation is not stochastic gradient ascent. We show circumstances under which applying the RPP is guaranteed to increase the mean return, even though it can make large changes in the values of the parameters. The proof is based on a mapping between the RPP and a form of the expectation-maximization procedure of Dempster, Laird, and Rubin (1977). <s> BIB012 </s> Reinforcement learning in robotics: A survey <s> Approach <s> This paper describes some of the interactions of model learning algorithms and planning algorithms we have found in exploring model-based reinforcement learning. The paper focuses on how local trajectory optimizers can be used effectively with learned non-parametric models. We find that trajectory planners that are fully consistent with the learned model often have difficulty finding reasonable plans in the early stages of learning. Trajectory planners that balance obeying the learned model with minimizing cost (or maximizing reward) often do better, even if the plan is not fully consistent with the learned model. <s> BIB013 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Robots are now being used in complex, unstructured environments, performing ever more sophisticated tasks. As the task and environmental complexity increases, the need for effective learning on such systems is becoming more and more apparent. Robot programmers often find it difficult to articulate their knowledge of how to perform a given task in a form suitable for robots to use. Even when they can, the limitations of robot sensors and actuators might render their intuitions less effective. Also, it is often not possible to anticipate (and code for) all environments in which the robot might find itself having to perform a certain task. Therefore, it seems useful to have the robot be able to learn to act, in the hope of overcoming these two difficulties. One promising approach to learning on real robots that is attracting considerable interest at the moment is reinforcement learning. The programmer must supply a reward function which maps states of the world onto a scalar reward, essentially saying how good or bad it is to be in a given state. Once given this function, the robot can, in theory, learn an appropriate action policy to maximize some measure of reward over time. Designing such a function seems more intuitive than writing code to perform the task, since we can do so in more general terms. However, there are some significant problems when we attempt to use this strategy on a real, physical robot, including (but not limited to) the following. <s> BIB014 </s> Reinforcement learning in robotics: A survey <s> Approach <s> This paper proposes a method that acquires robots' behaviors based on the estimation of the state vectors. In order to acquire the cooperative behaviors in multi-robot environments, each learning robot estimates the local predictive model between the learner and the other objects separately. Based on the local predictive models, the robots learn the desired behaviors using reinforcement learning. The proposed method is applied to a soccer playing situation, where a rolling ball and other moving robots are well modeled and the learner's behaviors are successfully acquired by the method. Computer simulations and real experiments are shown and a discussion is given. <s> BIB015 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Presents an adaptive control architecture for the four legged walking machine BISAM. This architecture uses coupled neuro-oscillators as representation of periodic behaviours on different control levels such as joint movement, leg control and leg coordination. Coupled neuro-oscillators together with adaptive sensor based reflexes provide a robust and efficient representation for quadrupedal locomotion and support the use of online learning approaches to realize adaptation and optimization of locomotion behaviours. <s> BIB016 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Function approximation is essential to reinforcement learning, but the standard approach of approximating a value function and determining a policy from it has so far proven theoretically intractable. In this paper we explore an alternative approach in which the policy is explicitly represented by its own function approximator, independent of the value function, and is updated according to the gradient of expected reward with respect to the policy parameters. Williams's REINFORCE method and actor-critic methods are examples of this approach. Our main new result is to show that the gradient can be written in a form suitable for estimation from experience aided by an approximate action-value or advantage function. Using this result, we prove for the first time that a version of policy iteration with arbitrary differentiable function approximation is convergent to a locally optimal policy. <s> BIB017 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Reinforcement learning systems improve behaviour based on scalar rewards from a critic. In this work vision based behaviours, servoing and wandering, are learned through a Q-learning method which handles continuous states and actions. There is no requirement for camera calibration, an actuator model, or a knowledgeable teacher. Learning through observing the actions of other behaviours improves learning speed. Experiments were performed on a mobile robot using a real-time vision system. <s> BIB018 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Abstract In this paper, we propose a hierarchical reinforcement learning architecture that realizes practical learning speed in real hardware control tasks. In order to enable learning in a practical number of trials, we introduce a low-dimensional representation of the state of the robot for higher-level planning. The upper level learns a discrete sequence of sub-goals in a low-dimensional state space for achieving the main goal of the task. The lower-level modules learn local trajectories in the original high-dimensional state space to achieve the sub-goal specified by the upper level. We applied the hierarchical architecture to a three-link, two-joint robot for the task of learning to stand up by trial and error. The upper-level learning was implemented by Q-learning, while the lower-level learning was implemented by a continuous actor–critic method. The robot successfully learned to stand up within 750 trials in simulation and then in an additional 170 trials using real hardware. The effects of the setting of the search steps in the upper level and the use of a supplementary reward for achieving sub-goals are also tested in simulation. <s> BIB019 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Many control problems in the robotics field can be cast as partially observed Markovian decision problems (POMDPs), an optimal control formalism. Finding optimal solutions to such problems in general, however is known to be intractable. It has often been observed that in practice, simple structured controllers suffice for good sub-optimal control, and recent research in the artificial intelligence community has focused on policy search methods as techniques for finding sub-optimal controllers when such structured controllers do exist. Traditional model-based reinforcement learning algorithms make a certainty equivalence assumption on their learned models and calculate optimal policies for a maximum-likelihood Markovian model. We consider algorithms that evaluate and synthesize controllers under distributions of Markovian models. Previous work has demonstrated that algorithms that maximize mean reward with respect to model uncertainty leads to safer and more robust controllers. We consider briefly other performance criterion that emphasize robustness and exploration in the search for controllers, and note the relation with experiment design and active learning. To validate the power of the approach on a robotic application we demonstrate the presented learning control algorithm by flying an autonomous helicopter. We show that the controller learned is robust and delivers good performance in this real-world domain. <s> BIB020 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Abstract Emergence of stable gaits in locomotion robots is studied in this paper. A classifier system, implementing an instance-based reinforcement-learning scheme, is used for the sensory-motor control of an eight-legged mobile robot and for the synthesis of the robot gaits. The robot does not have a priori knowledge of the environment and its own internal model. It is only assumed that the robot can acquire stable gaits by learning how to reach a goal area. During the learning process the control system is self-organized by reinforcement signals. Reaching the goal area defines a global reward. Forward motion gets a local reward, while stepping back and falling down get a local punishment. As learning progresses, the number of the action rules in the classifier systems is stabilized to a certain level, corresponding to the acquired gait patterns. Feasibility of the proposed self-organized system is tested under simulation and experiment. A minimal simulation model that does not require sophisticated computational schemes is constructed and used in simulations. The simulation data, evolved on the minimal model of the robot, is downloaded to the control system of the real robot. Overall, of 10 simulation data seven are successful in running the real robot. <s> BIB021 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Preface 1. Introduction to nonlinear programming 2. Large, sparse nonlinear programming 3. Optimal control preliminaries 4. The optimal control problem 5. Optimal control examples Appendix A. Software Bibliography, Index. <s> BIB022 </s> Reinforcement learning in robotics: A survey <s> Approach <s> It is difficult to apply traditional reinforcement learning algorithms to robots, due to problems with large and continuous domains, partial observability, and limited numbers of learning experiences. This paper deals with these problems by combining: (1) reinforcement learning with memory, implemented using an LSTM recurrent neural network whose inputs are discrete events extracted from raw inputs; (2) online exploration and offline policy learning. An experiment with a real robot demonstrates the methodology's feasibility. <s> BIB023 </s> Reinforcement learning in robotics: A survey <s> Approach <s> From the Publisher: ::: * Unique in its survey of the range of topics. ::: * Contains a strong, interdisciplinary format that will appeal to both students and researchers. ::: * Features exercises and web links to software and data sets. <s> BIB024 </s> Reinforcement learning in robotics: A survey <s> Approach <s> This paper presents a method of vision-based reinforcement learning by which a robot learns to shoot a ball into a goal. We discuss several issues in applying the reinforcement learning method to a real robot with vision sensor by which the robot can obtain information about the changes in an environment. First, we construct a state space in terms of size, position, and orientation of a ball and a goal in an image, and an action space is designed in terms of the action commands to be sent to the left and right motors of a mobile robot. This causes a “state-action deviation” problem in constructing the state and action spaces that reflect the outputs from physical sensors and actuators, respectively. To deal with this issue, an action set is constructed in a way that one action consists of a series of the same action primitive which is successively executed until the current state changes. Next, to speed up the learning time, a mechanism of Learning from Easy Missions (or LEM) is implemented. LEM reduces the learning time from exponential to almost linear order in the size of the state space. The results of computer simulations and real robot experiments are given. <s> BIB025 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Since sensors have limited range and coverage, mobile robots often have to make decisions on where to point their sensors. A good sensing strategy allows a robot to collect information that is useful for its tasks. Most existing solutions to this active sensing problem choose the direction that maximally reduces the uncertainty in a single state variable. In more complex problem domains, however, uncertainties exist in multiple state variables, and they affect the performance of the robot in different ways. The robot thus needs to have more sophisticated sensing strategies in order to decide which uncertainties to reduce, and to make the correct trade-offs. In this work, we apply a least squares reinforcement learning method to solve this problem. We implemented and tested the learning approach in the RoboCup domain, where the robot attempts to reach a ball and accurately kick it into the goal. We present experimental results that suggest our approach is able to learn highly effective sensing strategies. <s> BIB026 </s> Reinforcement learning in robotics: A survey <s> Approach <s> This article presents a general class of associative reinforcement learning algorithms for connectionist networks containing stochastic units. These algorithms, called REINFORCE algorithms, are shown to make weight adjustments in a direction that lies along the gradient of expected reinforcement in both immediate-reinforcement tasks and certain limited forms of delayed-reinforcement tasks, and they do this without explicitly computing gradient estimates or even storing information from which such estimates could be computed. Specific examples of such algorithms are presented, some of which bear a close relationship to certain existing algorithms while others are novel but potentially interesting in their own right. Also given are results that show how such algorithms can be naturally integrated with backpropagation. We close with a brief discussion of a number of additional issues surrounding the use of such algorithms, including what is known about their limiting behaviors as well as further considerations that might be used to help develop similar but potentially more powerful reinforcement learning algorithms. <s> BIB027 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Reinforcement learning (RL) methods provide a means for solving optimal control problems when accurate models are unavailable. For many such problems, however, RL alone is impractical and the associated learning problem must be structured somehow to take advantage of prior knowledge. In this paper we examine the use of such knowledge in the form of a stable controller that generates control inputs in parallel with an RL system. The controller acts as a supervisor that not only teaches the RL system about favorable control actions but also protects the learning system from risky behavior. We demonstrate the approach with a simulated robotic arm and a real seven-DOF manipulator. <s> BIB028 </s> Reinforcement learning in robotics: A survey <s> Approach <s> We present a learning system which is able to quickly and reliably acquire a robust feedback control policy for 3D dynamic walking from a blank-slate using only trials implemented on our physical robot. The robot begins walking within a minute and learning converges in approximately 20 minutes. This success can be attributed to the mechanics of our robot, which are modeled after a passive dynamic walker, and to a dramatic reduction in the dimensionality of the learning problem. We reduce the dimensionality by designing a robot with only 6 internal degrees of freedom and 4 actuators, by decomposing the control system in the frontal and sagittal planes, and by formulating the learning problem on the discrete return map dynamics. We apply a stochastic policy gradient algorithm to this reduced problem and decrease the variance of the update using a state-based estimate of the expected cost. This optimized learning system works quickly enough that the robot is able to continually adapt to the terrain as it walks. <s> BIB029 </s> Reinforcement learning in robotics: A survey <s> Approach <s> This paper presents a new reinforcement learning algorithm for accelerating acquisition of new skills by real mobile robots, without requiring simulation. It speeds up Q-learning by applying memory-based sweeping and enforcing the “adjoining property”, a technique that exploits the natural ordering of sensory state spaces in many robotic applications by only allowing transitions between neighbouring states. The algorithm is tested within an image-based visual servoing framework on a docking task, in which the robot has to position its gripper at a desired configuration relative to an object on a table. In experiments, we compare the performance of the new algorithm with a hand-designed linear controller and a scheme using the linear controller as a bias to further accelerate the learning. By analysis of the controllability and docking time, we show that the biased learner could improve on the performance of the linear controller, while requiring substantially lower training time than unbiased learning (less than 1 hour on the real robot). <s> BIB030 </s> Reinforcement learning in robotics: A survey <s> Approach <s> A combination of model-based and iterative learning control (ILC) is proposed as a method to achieve high-quality motion control of direct-drive robots in repetitive motion tasks. We include both model-based and learning components in the total control law, as their individual properties influence the performance of motion control. The model-based part of the controller compensates much of the nonlinear and coupled robot dynamics. A new procedure for estimating the parameters of the rigid body model, implemented in this part of the controller, is used. This procedure is based on a batch-adaptive control algorithm that estimates the model parameters online. Information about the dynamics not covered by the rigid body model, due to flexibilities, is acquired experimentally, by identification. The models of the flexibilities are used in the design of the iterative learning controllers for the individual joints. Use of the models facilitates quantitative prediction of performance improvement via ILC. The effectiveness of the combination of the model-based and the iterative learning controllers is demonstrated in experiments on a spatial serial direct-drive robot with revolute joints. <s> BIB031 </s> Reinforcement learning in robotics: A survey <s> Approach <s> This paper describes quasi-online reinforcement learning: while a robot is exploring its environment, in the background a probabilistic model of the environment is built on the fly as new experiences arrive; the policy is trained concurrently based on this model using an anytime algorithm. Prioritized sweeping, directed exploration, and transformed reward functions provide additional speed-ups. The robot quickly learns goal-directed policies from scratch, requiring few interactions with the environment and making efficient use of available computation time. From an outside perspective it learns the behavior online and in real time. We describe comparisons with standard methods and show the individual utility of each of the proposed techniques <s> BIB032 </s> Reinforcement learning in robotics: A survey <s> Approach <s> In this paper, we apply Reinforcement Learning (RL) to a real-world task. While complex problems have been solved by RL in simulated worlds, the costs of obtaining enough training examples often prohibits the use of plain RL in real-world scenarios. We propose three approaches to reduce training expenses for real-world RL. Firstly, we replace the random exploration of the huge search space, which plain RL uses, by guided exploration that imitates a teacher. Secondly, we use experiences not only once but store and reuse them later on when their value is easier to assess. Finally, we utilize function approximators in order to represent the experience in a way that balances between generalization and discrimination. We evaluate the performance of the combined extensions of plain RL using a humanoid robot in the RoboCup soccer domain. As we show in simulation and real-world experiments, our approach enables the robot to quickly learn fundamental soccer skills. <s> BIB033 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Autonomous approaching objects for an arm-hand robot is a very difficult problem because the possible arm-hand configurations are numerous. In this paper, we propose a modified reinforcement learning algorithm for a multifingerd hand approaching target objects. The proposed approach integrates the heuristic search information with the learning system, and solves the problem of how an arm-hand robot approaches objects before grasping. In addition, this method also overcomes the problem of time consuming of traditional reinforcement learning in the initial learning phase. The algorithm is applied to an arm-hand robot to approach objects before grasping, which can enable the robot to learn approaching skill by trial-and-error and plan its path by itself. The experimental results demonstrate the effectiveness of the proposed algorithm. <s> BIB034 </s> Reinforcement learning in robotics: A survey <s> Approach <s> In the model-based policy search approach to reinforcement learning (RL), policies are found using a model (or "simulator") of the Markov decision process. However, for high-dimensional continuous-state tasks, it can be extremely difficult to build an accurate model, and thus often the algorithm returns a policy that works in simulation but not in real-life. The other extreme, model-free RL, tends to require infeasibly large numbers of real-life trials. In this paper, we present a hybrid algorithm that requires only an approximate model, and only a small number of real-life trials. The key idea is to successively "ground" the policy evaluations using real-life trials, but to rely on the approximate model to suggest local changes. Our theoretical results show that this algorithm achieves near-optimal performance in the real system, even when the model is only approximate. Empirical results also demonstrate that---when given only a crude model and a small number of real-life trials---our algorithm can obtain near-optimal performance in the real system. <s> BIB035 </s> Reinforcement learning in robotics: A survey <s> Approach <s> This paper describes two experiments with supervised reinforcement learning (RL) on a real, mobile robot. Two types of experiments were preformed. One tests the robot's reliability in implementing a navigation task it has been taught by a supervisor. The other, in which new obstacles are placed along the previously learned path to the goal, measures the robot's robustness to changes in environment. Supervision consisted of human-guided, remote-controlled runs through a navigation task during the initial stages of reinforcement learning. The RL algorithms deployed enabled the robot to learn a path to a goal yet retain the ability to explore different solutions when confronted with a new obstacle. Experimental analysis was based on measurements of average time to reach the goal, the number of failed states encountered during an episode, and how closely the RL learner matched the supervisor's actions. <s> BIB036 </s> Reinforcement learning in robotics: A survey <s> Approach <s> The robot soccer game has been proposed as a benchmark problem for the artificial intelligence and robotic researches. Decision-making system is the most important part of the robot soccer system. As the environment is dynamic and complex, one of the reinforcement learning (RL) method named FNN-RL is employed in learning the decision-making strategy. The FNN-RL system consists of the fuzzy neural network (FNN) and RL. RL is used for structure identification and parameters tuning of FNN. On the other hand, the curse of dimensionality problem of RL can be solved by the function approximation characteristics of FNN. Furthermore, the residual algorithm is used to calculate the gradient of the FNN-RL method in order to guarantee the convergence and rapidity of learning. The complex decision-making task is divided into multiple learning subtasks that include dynamic role assignment, action selection, and action implementation. They constitute a hierarchical learning system. We apply the proposed FNN-RL method to the soccer agents who attempt to learn each subtask at the various layers. The effectiveness of the proposed method is demonstrated by the simulation and the real experiments. <s> BIB037 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Accurate and fast control of wheel speeds in the presence of noise and nonlinearities is one of the crucial requirements for building fast mobile robots, as they are required in the MiddleSize League of RoboCup. We will describe, how highly effective speed controllers can be learned from scratch on the real robot directly. The use of our recently developed neural fitted Q iteration scheme allows reinforcement learning of neural controllers with only a limited amount of training data seen. In the described application, less than 5 minutes of interaction with the real robot were sufficient, to learn fast and accurate control to arbitrary target speeds. <s> BIB038 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Gait optimization is a basic yet challenging problem for both quadrupedal and bipedal robots. Although techniques for automating the process exist, most involve local function optimization procedures that suffer from three key drawbacks. Local optimization techniques are naturally plagued by local optima, make no use of the expensive gait evaluations once a local step is taken, and do not explicitly model noise in gait evaluation. These drawbacks increase the need for a large number of gait evaluations, making optimization slow, data inefficient, and manually intensive. We present a Bayesian approach based on Gaussian process regression that addresses all three drawbacks. It uses a global search strategy based on a posterior model inferred from all of the individual noisy evaluations. We demonstrate the technique on a quadruped robot, using it to optimize two different criteria: speed and smoothness. We show in both cases our technique requires dramatically fewer gait evaluations than state-of-the-art local gradient approaches. <s> BIB039 </s> Reinforcement learning in robotics: A survey <s> Approach <s> This paper focused on the problem of the autonomous mobile robot navigation under the unknown and changing environment. The reinforcement learning (RL) is applied to learn behaviors of reactive robot. T-S fuzzy neural network and RL are integrated. T-S network is used to implement the mapping from the state space to Q values corresponding with action space of RL. The problem of continuous, infinite states and actions in RL is able to be solved through the function approximation of proposed method. Finally, the method of this paper is applied to learn behaviors for the reactive robot. The experiment shows that the algorithm can effectively solve the problem of navigation in a complicated unknown environment. <s> BIB040 </s> Reinforcement learning in robotics: A survey <s> Approach <s> We introduce a learning-based approach to manipulation in unstructured environments. This approach permits autonomous acquisition of manipulation expertise from interactions with the environment. The resulting expertise enables a robot to perform effective manipulation based on partial state information. The manipulation expertise is represented in a relational state representation and learned using relational reinforcement learning. The relational representation renders learning tractable by collapsing a large number of states onto a single, relational state. The relational state representation is carefully grounded in the perceptual and interaction skills of the robot. This ensures that symbolically learned knowledge remains meaningful in the physical world. We experimentally validate the proposed learning approach on the task of manipulating an articulated object to obtain a model of its kinematic structure. Our experiments demonstrate that the manipulation expertise acquired by the robot leads to substantial performance improvements. These improvements are maintained when experience is applied to previously unseen objects. <s> BIB041 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Autonomous learning is one of the hallmarks of human and animal behavior, and understanding the principles of learning will be crucial in order to achieve true autonomy in advanced machines like humanoid robots. In this paper, we examine learning of complex motor skills with human-like limbs. While supervised learning can offer useful tools for bootstrapping behavior, e.g., by learning from demonstration, it is only reinforcement learning that offers a general approach to the final trial-and-error improvement that is needed by each individual acquiring a skill. Neither neurobiological nor machine learning studies have, so far, offered compelling results on how reinforcement learning can be scaled to the high-dimensional continuous state and action spaces of humans or humanoids. Here, we combine two recent research developments on learning motor control in order to achieve this scaling. First, we interpret the idea of modular motor control by means of motor primitives as a suitable way to generate parameterized control policies for reinforcement learning. Second, we combine motor primitives with the theory of stochastic policy gradient learning, which currently seems to be the only feasible framework for reinforcement learning for humanoids. We evaluate different policy gradient methods with a focus on their applicability to parameterized motor primitives. We compare these algorithms in the context of motor primitive learning, and show that our most modern algorithm, the Episodic Natural Actor-Critic outperforms previous algorithms by at least an order of magnitude. We demonstrate the efficiency of this reinforcement learning method in the application of learning to hit a baseball with an anthropomorphic robot arm. <s> BIB042 </s> Reinforcement learning in robotics: A survey <s> Approach <s> One of the most general frameworks for phrasing control problems for complex, redundant robots is operational-space control. However, while this framework is of essential importance for robotics and well understood from an analytical point of view, it can be prohibitively hard to achieve accurate control in the face of modeling errors, which are inevitable in complex robots (e.g. humanoid robots). In this paper, we suggest a learning approach for operational-space control as a direct inverse model learning problem. A first important insight for this paper is that a physically correct solution to the inverse problem with redundant degrees of freedom does exist when learning of the inverse map is performed in a suitable piecewise linear way. The second crucial component of our work is based on the insight that many operational-space controllers can be understood in terms of a constrained optimal control problem. The cost function associated with this optimal control problem allows us to formulate a learning algorithm that automatically synthesizes a globally consistent desired resolution of redundancy while learning the operational-space controller. From the machine learning point of view, this learning problem corresponds to a reinforcement learning problem that maximizes an immediate reward. We employ an expectation-maximization policy search algorithm in order to solve this problem. Evaluations on a three degrees-of-freedom robot arm are used to illustrate the suggested approach. The application to a physically realistic simulator of the anthropomorphic SARCOS Master arm demonstrates feasibility for complex high degree-of-freedom robots. We also show that the proposed method works in the setting of learning resolved motion rate control on a real, physical Mitsubishi PA-10 medical robotics arm. <s> BIB043 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Many motor skills in humanoid robotics can be learned using parametrized motor primitives as done in imitation learning. However, most interesting motor learning problems are high-dimensional reinforcement learning problems often beyond the reach of current methods. In this paper, we extend previous work on policy learning from the immediate reward case to episodic reinforcement learning. We show that this results in a general, common framework also connected to policy gradient methods and yielding a novel algorithm for policy learning that is particularly well-suited for dynamic motor primitives. The resulting algorithm is an EM-inspired algorithm applicable to complex motor learning tasks. We compare this algorithm to several well-known parametrized policy search methods and show that it outperforms them. We apply it in the context of motor learning and show that it can learn a complex Ball-in-a-Cup task using a real Barrett WAM™ robot arm. <s> BIB044 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Automatically building maps from sensor data is a necessary and fundamental skill for mobile robots; as a result, considerable research attention has focused on the technical challenges inherent in the mapping problem. While statistical inference techniques have led to computationally efficient mapping algorithms, the next major challenge in robotic mapping is to automate the data collection process. In this paper, we address the problem of how a robot should plan to explore an unknown environment and collect data in order to maximize the accuracy of the resulting map. We formulate exploration as a constrained optimization problem and use reinforcement learning to find trajectories that lead to accurate maps. We demonstrate this process in simulation and show that the learned policy not only results in improved map building, but that the learned policy also transfers successfully to a real robot exploring on MIT campus. <s> BIB045 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Dynamic system-based motor primitives have enabled robots to learn complex tasks ranging from Tennis-swings to locomotion. However, to date there have been only few extensions which have incorporated perceptual coupling to variables of external focus, and, furthermore, these modifications have relied upon handcrafted solutions. Humans learn how to couple their movement primitives with external variables. Clearly, such a solution is needed in robotics. In this paper, we propose an augmented version of the dynamic systems motor primitives which incorporates perceptual coupling to an external variable. The resulting perceptually driven motor primitives include the previous primitives as a special case and can inherit some of their interesting properties. We show that these motor primitives can perform complex tasks such a Ball-in-a-Cup or Kendama task even with large variances in the initial conditions where a skilled human player would be challenged. For doing so, we initialize the motor primitives in the traditional way by imitation learning without perceptual coupling. Subsequently, we improve the motor primitives using a novel reinforcement learning method which is particularly well-suited for motor primitives. <s> BIB046 </s> Reinforcement learning in robotics: A survey <s> Approach <s> In this paper the problem of free gait generation and adaptability with reinforcement learning are addressed for a six-legged robot. Using the developed free gait generation algorithm the robot maintains to generate stable gaits according to the commanded velocity. The reinforcement learning scheme incorporated into the free gait generation makes the robot choose more stable states and develop a continuous walking pattern with a larger average stability margin. While walking in normal conditions with no external effects causing unstability, the robot is guaranteed to have stable walk, and the reinforcement learning only improves the stability. The adaptability of the learning scheme is tested also for the abnormal case of deficiency in one of the rear-legs. The robot gets a negative reinforcement when it falls, and a positive reinforcement when a stable transition is achieved. In this way the robot learns to achieve a continuous pattern of stable walk with five legs. The developed free gait generation with reinforcement learning is applied in real-time on the actual robot both for normal walking with different speeds and learning of five-legged walking in the abnormal case. <s> BIB047 </s> Reinforcement learning in robotics: A survey <s> Approach <s> We consider the task of learning to accurately follow a trajectory in a vehicle such as a car or helicopter. A number of dynamic programming algorithms such as Differential Dynamic Programming (DDP) and Policy Search by Dynamic Programming (PSDP), can efficiently compute non-stationary policies for these tasks --- such policies in general are well-suited to trajectory following since they can easily generate different control actions at different times in order to follow the trajectory. However, a weakness of these algorithms is that their policies are time-indexed, in that they apply different policies depending on the current time. This is problematic since 1) the current time may not correspond well to where we are along the trajectory and 2) the uncertainty over states can prevent these algorithms from finding any good policies at all. In this paper we propose a method for space-indexed dynamic programming that overcomes both these difficulties. We begin by showing how a dynamical system can be rewritten in terms of a spatial index variable (i.e., how far along the trajectory we are) rather than as a function of time. We then use these space-indexed dynamical systems to derive space-indexed version of the DDP and PSDP algorithms. Finally, we show that these algorithms perform well on a variety of control tasks, both in simulation and on real systems. <s> BIB048 </s> Reinforcement learning in robotics: A survey <s> Approach <s> When children learn to grasp a new object, they often know several possible grasping points from observing a parent's demonstration and subsequently learn better grasps by trial and error. From a machine learning point of view, this process is an active learning approach. In this paper, we present a new robot learning framework for reproducing this ability in robot grasping. For doing so, we chose a straightforward approach: first, the robot observes a few good grasps by demonstration and learns a value function for these grasps using Gaussian process regression. Subsequently, it chooses grasps which are optimal with respect to this value function using a mean-shift optimization approach, and tries them out on the real system. Upon every completed trial, the value function is updated, and in the following trials it is more likely to choose even better grasping points. This method exhibits fast learning due to the data-efficiency of the Gaussian process regression framework and the fact that the mean-shift method provides maxima of this cost function. Experiments were repeatedly carried out successfully on a real robot system. After less than sixty trials, our system has adapted its grasping policy to consistently exhibit successful grasps. <s> BIB049 </s> Reinforcement learning in robotics: A survey <s> Approach <s> General-purpose autonomous robots need to have the ability to sequence and adapt the available sensorimotor knowledge, which is often given in the form of movement primitives. In order to solve a given task in situations that were not considered during the initial learning, it is necessary to adapt trajectories contained in the library of primitive motions to new situations. In this paper we explore how to apply reinforcement learning to modify the subgoals of primitive movements involved in the given task. As the underlying sensorimotor representation we selected nonlinear dynamic systems, which provide a powerful machinery for the modification of motion trajectories. We propose a new formulation for dynamic systems, which ensures that consecutive primitive movements can be splined together in a continuous way (up to second order derivatives). <s> BIB050 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Autonomous helicopter flight is widely regarded to be a highly challenging control problem. As helicopters are highly unstable and exhibit complicated dynamical behavior, it is particularly difficult to design controllers that achieve high performance over a broad flight regime. While these aircraft are notoriously difficult to control, there are expert human pilots who are nonetheless capable of demonstrating a wide variety of maneuvers, including aerobatic maneuvers at the edge of the helicopter's performance envelope. In this paper, we present algorithms for modeling and control that leverage these demonstrations to build high-performance control systems for autonomous helicopters. More specifically, we detail our experiences with the Stanford Autonomous Helicopter, which is now capable of extreme aerobatic flight meeting or exceeding the performance of our own expert pilot. <s> BIB051 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Reinforcement learning (RL) algorithms have long been promising methods for enabling an autonomous robot to improve its behavior on sequential decision-making tasks. The obvious enticement is that the robot should be able to improve its own behavior without the need for detailed step-by-step programming. However, for RL to reach its full potential, the algorithms must be sample efficient: they must learn competent behavior from very few real-world trials. From this perspective, model-based methods, which use experiential data more efficiently than model-free approaches, are appealing. But they often require exhaustive exploration to learn an accurate model of the domain. In this paper, we present an algorithm, Reinforcement Learning with Decision Trees (RL-DT), that uses decision trees to learn the model by generalizing the relative effect of actions across states. The agent explores the environment until it believes it has a reasonable policy. The combination of the learning approach with the targeted exploration policy enables fast learning of the model. We compare RL-DT against standard model-free and model-based learning methods, and demonstrate its effectiveness on an Aldebaran Nao humanoid robot scoring goals in a penalty kick scenario. <s> BIB052 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Reinforcement learning in the high-dimensional, continuous spaces typical in robotics, remains a challenging problem. To overcome this challenge, a popular approach has been to use demonstrations to find an appropriate initialisation of the policy in an attempt to reduce the number of iterations needed to find a solution. Here, we present an alternative way to incorporate prior knowledge from demonstrations of individual postures into learning, by extracting the inherent problem structure to find an efficient state representation. In particular, we use probabilistic, nonlinear dimensionality reduction to capture latent constraints present in the data. By learning policies in the learnt latent space, we are able to solve the planning problem in a reduced space that automatically satisfies task constraints. As shown in our experiments, this reduces the exploration needed and greatly accelerates the learning. We demonstrate our approach for learning a bimanual reaching task on the 19-DOF KHR-1HV humanoid. <s> BIB053 </s> Reinforcement learning in robotics: A survey <s> Approach <s> While today's industrial robots achieve high performance in a wide range of tasks, the programming and adaptation of such robots to new tasks is still time-consuming and far from convenient. This shortcoming is a major obstacle for the deployment of robots in small to medium-size companies and also for service applications. Robot learning by imitation has been identified as an important paradigm to tackle this practicability issue. However, such approaches are only applicable in controlled static scenarios. To perform less constrained tasks, the robot needs the ability to generalize its skills and to adapt to changing conditions. It is thus a natural idea to formulate the problem as learning from experience and to incorporate demonstrations by an expert in the learning process. We present a new framework for learning of motion skills that seamlessly integrates demonstration learning from an expert teacher and further optimization of the demonstrated skill by own experience. Our method employs Gaussian Process regression to generalize a measure of performance to new skills that are close to already learned skills in the state-action space. We evaluate our approach for the task of grasping an object. <s> BIB054 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Grasping an object is a task that inherently needs to be treated in a hybrid fashion. The system must decide both where and how to grasp the object. While selecting where to grasp requires learning about the object as a whole, the execution only needs to reactively adapt to the context close to the grasp's location. We propose a hierarchical controller that reflects the structure of these two sub-problems, and attempts to learn solutions that work for both. A hybrid architecture is employed by the controller to make use of various machine learning methods that can cope with the large amount of uncertainty inherent to the task. The controller's upper level selects where to grasp the object using a reinforcement learner, while the lower level comprises an imitation learner and a vision-based reactive controller to determine appropriate grasping motions. The resulting system is able to quickly learn good grasps of a novel object in an unstructured environment, by executing smooth reaching motions and preshaping the hand depending on the object's geometry. The system was evaluated both in simulation and on a real robot. <s> BIB055 </s> Reinforcement learning in robotics: A survey <s> Approach <s> In the paper we evaluate two learning methods applied to the ball-in-a-cup game. The first approach is based on imitation learning. The captured trajectory was encoded with Dynamic motion primitives (DMP). The DMP approach allows simple adaptation of the demonstrated trajectory to the robot dynamics. In the second approach, we use reinforcement learning, which allows learning without any previous knowledge of the system or the environment. In contrast to the majority of the previous attempts, we used SASRA learning algorithm. Experimental results for both cases were performed on Mitsubishi PA10 robot arm. <s> BIB056 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Reinforcement learning (RL) is one of the most general approaches to learning control. Its applicability to complex motor systems, however, has been largely impossible so far due to the computational difficulties that reinforcement learning encounters in high dimensional continuous state-action spaces. In this paper, we derive a novel approach to RL for parameterized control policies based on the framework of stochastic optimal control with path integrals. While solidly grounded in optimal control theory and estimation theory, the update equations for learning are surprisingly simple and have no danger of numerical instabilities as neither matrix inversions nor gradient learning rates are required. Empirical evaluations demonstrate significant performance improvements over gradient-based policy learning and scalability to high-dimensional control problems. Finally, a learning experiment on a robot dog illustrates the functionality of our algorithm in a real-world scenario. We believe that our new algorithm, Policy Improvement with Path Integrals (PI2), offers currently one of the most efficient, numerically robust, and easy to implement algorithms for RL in robotics. <s> BIB057 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Policy search is a successful approach to reinforcement learning. However, policy improvements often result in the loss of information. Hence, it has been marred by premature convergence and implausible solutions. As first suggested in the context of covariant policy gradients (Bagnell and Schneider 2003), many of these problems may be addressed by constraining the information loss. In this paper, we continue this path of reasoning and suggest the Relative Entropy Policy Search (REPS) method. The resulting method differs significantly from previous policy gradient approaches and yields an exact update step. It works well on typical reinforcement learning benchmark problems. <s> BIB058 </s> Reinforcement learning in robotics: A survey <s> Approach <s> In the future, robotic surgical assistants may assist surgeons by performing specific subtasks such as retraction and suturing to reduce surgeon tedium and reduce the duration of some operations. We propose an apprenticeship learning approach that has potential to allow robotic surgical assistants to autonomously execute specific trajectories with superhuman performance in terms of speed and smoothness. In the first step, we record a set of trajectories using human-guided backdriven motions of the robot. These are then analyzed to extract a smooth reference trajectory, which we execute at gradually increasing speeds using a variant of iterative learning control. We evaluate this approach on two representative tasks using the Berkeley Surgical Robots: a figure eight trajectory and a two handed knot-tie, a tedious suturing sub-task required in many surgical procedures. Results suggest that the approach enables (i) rapid learning of trajectories, (ii) smoother trajectories than the human-guided trajectories, and (iii) trajectories that are 7 to 10 times faster than the best human-guided trajectories. <s> BIB059 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Synchronization is routinely required to coordinate the actions of the various sub-systems involved in process applications. This is commonly achieved through direct mechanical coupling, involving gears, drive belts and cams. Apart from the additional cost incurred, these components are subject to wear, constrain the layout of the plant and may have limited accuracy. It is shown in this paper that a mechanical linkage between two sub-systems may be replaced by instead implementing a control scheme comprising an iterative learning controller together with a supervisory control loop. To illustrate the approach, two types of iterative learning controller are first implemented on a gantry robot test facility to confirm the high levels of tracking accuracy that may be achieved. The supervisory control loop is then added to synchronize the ‘pick and place’ action of the robot with a conveyor system moving at constant velocity. Experimental results are provided to confirm both the accurate tracking performance produced by the iterative learning controller, and the high level of synchronization achieved by the overall scheme. <s> BIB060 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Reliable and efficient navigation with a humanoid robot is a difficult task. First, the motion commands are executed rather inaccurately due to backlash in the joints or foot slippage. Second, the observations are typically highly affected by noise due to the shaking behavior of the robot. Thus, the localization performance is typically reduced while the robot moves and the uncertainty about its pose increases. As a result, the reliable and efficient execution of a navigation task cannot be ensured anymore since the robot's pose estimate might not correspond to the true location. In this paper, we present a reinforcement learning approach to select appropriate navigation actions for a humanoid robot equipped with a camera for localization. The robot learns to reach the destination reliably and as fast as possible, thereby choosing actions to account for motion drift and trading off velocity in terms of fast walking movements against accuracy in localization. We present extensive simulated and practical experiments with a humanoid robot and demonstrate that our learned policy significantly outperforms a hand-optimized navigation strategy. <s> BIB061 </s> Reinforcement learning in robotics: A survey <s> Approach <s> We describe a robot system that autonomously acquires skills through interaction with its environment. The robot learns to sequence the execution of a set of innate controllers to solve a task, extracts and retains components of that solution as portable skills, and then transfers those skills to reduce the time required to learn to solve a second task. <s> BIB062 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Several categories of optimization problems suffer from expensive objective function evaluation, driving the need for smart selection of subsequent experiments. One such category of problems involves physical robotic systems, which often require significant time, effort, and monetary expenditure in order to run tests. To assist in the selection of the next experiment, there has been a focus on the idea of response surfaces in recent years. These surfaces interpolate the existing data and provide a measure of confidence in their error, serving as a low-fidelity surrogate function that can be used to more intelligently choose the next experiment. In this paper, we robustly implement a previous algorithm based on the response surface methodology with an expected improvement criteria. We apply this technique to optimize open-loop gait parameters for snake robots, and demonstrate improved locomotive capabilities. <s> BIB063 </s> Reinforcement learning in robotics: A survey <s> Approach <s> As this book demonstrates, the development of efficient probabilistic inference techniques has made considerable progress in recent years, in particular with respect to exploiting the structure (e.g., factored, hierarchical or relational) of discrete and continuous problem domains. In this chapter we show that these techniques can be used also for solving Markov Decision Processes (MDPs) or partial observable MDPs (POMDPs) when formulated in terms of a structured dynamic Bayesian network (DBN). The problems of planning in stochastic environments and inference in state space models are closely related, in particular in view of the challenges both of them face: scaling to large state spaces spanned by multiple state variables, or realizing planning (or inference) in continuous or mixed continuous-discrete state spaces. Both fields developed techniques to address these problems. For instance, in the field of planning, they include work on Factored Markov Decision Processes (Boutilier et al., 1995; Koller and Parr, 1999; Guestrin et al., 2003; Kveton and Hauskrecht, 2005), abstractions (Hauskrecht et al., 1998), and relational models of the environment (Zettlemoyer et al., 2005). On the other hand, recent advances in inference techniques show how structure can be exploited both for exact inference as well as making efficient approximations. Examples are message passing algorithms (loopy Belief Propagation, Expectation Propagation), variational approaches, approximate belief representations (particles, Assumed Density Filtering, Boyen-Koller) and arithmetic compilation (see, e.g., Minka, 2001; Murphy, 2002; Chavira et al., 2006). In view of these similarities one may ask whether existing techniques for probabilistic inference can directly be translated to solving stochastic planning problems. From a complexity theoretic point of view, the equivalence between inference and planning is well-known (see, e.g., Littman et al., 2001). Inference methods have been applied before to optimal decision making in Influence Diagrams (Cooper, 1988; Pearl, 1988; Shachter, 1988). However, contrary to MDPs, these methods focus on a finite number of decisions and a non-stationary policy, where optimal decisions are found by recursing backward starting from the last decision (see (Boutilier et al., 1999) and (Toussaint, 2009) for a discussion of MDPs versus Influence Diagrams). More recently, Bui et al. (2002) have used inference on Abstract Hidden Markov Models for policy recognition, i.e., for reasoning about executed behaviors, but do not address the problem of computing optimal policies from such inference. Attias <s> BIB064 </s> Reinforcement learning in robotics: A survey <s> Approach <s> The biomechanics community has recently made progress toward understanding the role of rapid arm movements in human stability recovery. However, comparatively little work has been done exploring this type of control in humanoid robots. We provide a summary of recent insights into the functional contributions of arm recovery motions in humans and experimentally demonstrate advantages of this behavior on a dynamically stable mobile manipulator. Using Bayesian optimization, the robot efficiently discovers policies that reduce total energy expenditure and recovery footprint, and increase ability to stabilize after large impacts. <s> BIB065 </s> Reinforcement learning in robotics: A survey <s> Approach <s> Reinforcement Learning (RL) is a paradigm for learning decision-making tasks that could enable robots to learn and adapt to their situation on-line. For an RL algorithm to be practical for robotic control tasks, it must learn in very few samples, while continually taking actions in real-time. Existing model-based RL methods learn in relatively few samples, but typically take too much time between each action for practical on-line learning. In this paper, we present a novel parallel architecture for model-based RL that runs in real-time by 1) taking advantage of sample-based approximate planning methods and 2) parallelizing the acting, model learning, and planning processes in a novel way such that the acting process is sufficiently fast for typical robot control cycles. We demonstrate that algorithms using this architecture perform nearly as well as methods using the typical sequential architecture when both are given unlimited time, and greatly out-perform these methods on tasks that require real-time actions such as controlling an autonomous vehicle. <s> BIB066 </s> Reinforcement learning in robotics: A survey <s> Approach <s> We describe CST, an online algorithm for constructing skill trees from demonstration trajectories. CST segments a demonstration trajectory into a chain of component skills, where each skill has a goal and is assigned a suitable abstraction from an abstraction library. These properties permit skills to be improved efficiently using a policy learning algorithm. Chains from multiple demonstration trajectories are merged into a skill tree. We show that CST can be used to acquire skills from human demonstration in a dynamic continuous domain, and from both expert demonstration and learned control sequences on the uBot-5 mobile manipulator. <s> BIB067 </s> Reinforcement learning in robotics: A survey <s> Approach <s> There has been a recent focus in reinforcement learning on addressing continuous state and action problems by optimizing parameterized policies. PI2 is a recent example of this approach. It combines a derivation from first principles of stochastic optimal control with tools from statistical estimation theory. In this paper, we consider PI2 as a member of the wider family of methods which share the concept of probability-weighted averaging to iteratively update parameters to optimize a cost function. We compare PI2 to other members of the same family - Cross-Entropy Methods and CMAES - at the conceptual level and in terms of performance. The comparison suggests the derivation of a novel algorithm which we call PI2-CMA for "Path Integral Policy Improvement with Covariance Matrix Adaptation". PI2-CMA's main advantage is that it determines the magnitude of the exploration noise automatically. <s> BIB068 </s> Reinforcement learning in robotics: A survey <s> Approach <s> A fundamental problem in control is to learn a model of a system from observations that is useful for controller synthesis. To provide good performance guarantees, existing methods must assume that the real system is in the class of models considered during learning. We present an iterative method with strong guarantees even in the agnostic case where the system is not in the class. In particular, we show that any no-regret online learning algorithm can be used to obtain a near-optimal policy, provided some model achieves low training error and access to a good exploration distribution. Our approach applies to both discrete and continuous domains. We demonstrate its efficacy and scalability on a challenging helicopter domain from the literature. <s> BIB069
Employed by… Model-based BIB032 , BIB052 BIB066 , BIB007 , Martínez-Marín and Duckett (2005), BIB006 , and Touzet (1997) Model-free BIB025 , BIB023 , BIB003 , BIB008 ,  , BIB053 , BIB036 , BIB037 BIB040 ,  , BIB018 , BIB054 , BIB038 ,  , BIB009 , BIB016 , BIB041 ,  , BIB010 , BIB062 BIB067 , BIB049 BIB055 , BIB026 , BIB033 , BIB002 , BIB011 , BIB019 , BIB050 BIB056 (2008), and  and multiple shooting approaches BIB022 . We may broadly break down policy-search methods into "black box" and "white box" methods. Black box methods are general stochastic optimization algorithms BIB024 using only the expected return of policies, estimated by sampling, and do not leverage any of the internal structure of the reinforcement learning problem. These may be very sophisticated techniques BIB063 that use response surface estimates and bandit-like strategies to achieve good performance. White box methods take advantage of some of additional structure within the reinforcement learning domain, including, for instance, the (approximate) Markov structure of problems, developing approximate models, value-function estimates when available BIB042 , or even simply the causal ordering of actions and rewards. A major open issue within the field is the relative merits of the these two approaches: in principle, white box methods leverage more information, but with the exception of models (which have been demonstrated repeatedly to often make tremendous performance improvements, see Section 6), the performance gains are traded-off with additional assumptions that may be violated and less mature optimization algorithms. Some recent work including BIB063 BIB068 suggest that much of the benefit of policy search is achieved by black-box methods. Some of the most popular white-box general reinforcement learning techniques that have translated particularly well into the domain of robotics include: (i) policy gradient approaches based on likelihood-ratio estimation BIB017 , (ii) policy updates inspired by expectationmaximization (EM) BIB064 , and (iii) the path integral methods  . Let us briefly take a closer look at gradient-based approaches first. The updates of the policy parameters are based on a hill-climbing approach, that is following the gradient of the expected return J for a defined step size α Different methods exist for estimating the gradient ∇ θ J and many algorithms require tuning of the step-size α. In finite difference gradients P perturbed policy parameters are evaluated to obtain an estimate of the gradient. Here are the individual perturbations, Ĵ p the estimate of their influence on the return, and J ref is a reference return, e.g. the return of the unperturbed parameters. The gradient can now be estimated by linear regression where the matrix contains all of the stacked samples of the perturbations θ p and Ĵ contains the corresponding Ĵ p . In order to estimate the gradient the number of perturbations needs to be at least as large as the number of parameters. The approach is very straightforward and even applicable to policies that are not differentiable. However, it is usually considered to be very noisy and inefficient. For the finite difference approach tuning the step size α for the update, the number of perturbations P, and the type and magnitude of perturbations are all critical tuning factors. Likelihood ratio methods rely on the insight that in an episodic setting where the episodes τ are generated according to the distribution P θ (τ ) = P (τ |θ ) with the return of an episode J τ = H h=1 R h and number of steps H the expected return for a set of policy parameter θ can be expressed as The gradient of the episode distribution can be written as 4 which is commonly known as the likelihood ratio or REINFORCE (REward Increment = Nonnegative Factor times Offset Reinforcement times Characteristic Eligibility) BIB027 trick. Combining Equations (5) and (6) we get the gradient of the expected return in the form If we have a stochastic policy π θ (s, a) that generates the episodes τ , we do not need to keep track of the probabilities of the episodes but can directly express the gradient in terms of the policy as ∇ θ log P θ (τ ) = H h=1 ∇ θ log π θ (s, a). Finally the gradient of the expected return with respect to the policy parameters can be estimated as If we now take into account that rewards at the beginning of an episode cannot be caused by actions taken at the end of an episode, we can replace the return of the episode J τ by the state-action value function Q π (s, a) and obtain BIB042 ) which is equivalent to the policy gradient theorem BIB017 . In practice, it is often advisable to subtract a reference J ref , also called a baseline, from the return of the episode J τ or the state-action value function Q π (s, a), respectively, to get better estimates, similar to the finite difference approach. In these settings, the exploration is automatically taken care of by the stochastic policy. Initial gradient-based approaches such as finite differences gradients or REINFORCE BIB027 have been rather slow. The weight perturbation algorithm is related to REINFORCE but can deal with non-Gaussian distributions which significantly improves the signal-tonoise ratio of the gradient . Recent natural policy gradient approaches (Peters and Schaal, 2008c,b) have allowed for faster convergence which may be advantageous for robotics as it reduces the learning time and required real-world interactions. A different class of safe and fast policy-search methods, that are inspired by EM, can be derived when the reward is treated as an improper probability distribution BIB012 . Some of these approaches have proven successful in robotics, e.g. reward-weighted regression BIB043 , policy learning by weighting exploration with the returns BIB044 ), Monte Carlo EM  , and cost-regularized kernel regression . Algorithms with closely related update rules can also be derived from different perspectives including policy improvements with path integrals BIB057 and relative entropy policy search BIB058 . Finally, the policy search by dynamic programming method is a general strategy that combines policy search with the principle of optimality. The approach learns a non-stationary policy backward in time like dynamic programming methods, but does not attempt to enforce the Bellman equation and the resulting approximation instabilities (see Section 2.4). The resulting approach provides some of the strongest guarantees that are currently known under function approximation and limited observability. It has been demonstrated in learning walking controllers and in finding near-optimal trajectories for map exploration BIB045 . The resulting method is more expensive than the value-function methods because it scales quadratically in the effective time horizon of the problem. Like DDP methods BIB013 , it is tied to a non-stationary (time-varying) policy. An overview of publications using policy-search methods is presented in Table 2 . One of the key open issues in the field is determining when it is appropriate to use each of these methods. Some approaches leverage significant structure specific to the reinforcement learning problem (e.g. BIB057 , including reward structure, Markovanity, causality of reward signals BIB027 , and value-function estimates when available BIB042 . Others embed policy search as a generic, black-box, problem of stochastic optimization BIB020 BIB039 BIB065 BIB063 . Significant open questions remain regarding which methods are best in which circumstances and further, at an even more basic level, how effective leveraging the kinds of problem structures mentioned above are in practice. Employed by… Hand-crafted BIB003 ,  , BIB026 at the same time, these approaches are hierarchical reinforcement learning approaches as discussed in Section 5.2. BIB062 BIB067 proposes an approach that constructs a skill tree from human demonstrations. Here, the skills correspond to options and are chained to learn a mobile manipulation skill. Employed by… Teacher  ,  , BIB053 , BIB036 , BIB054 , BIB046 ; Peters (2009), Latzke et al. (2007) , and Peters and Schaal (2008c,b ) Policy  , BIB047 , BIB030 , BIB028 , BIB014 , BIB029 ;  , and BIB034 Prior knowledge through task structuring Employed by… Iterative learning control BIB035 , BIB001 , BIB059 , BIB031 BIB060 , and Norrlöf (2002) Locally linear quadratic regulators  ; BIB013 , BIB051 BIB048 , BIB004 , and  Value-function methods with learned models BIB032 , BIB056 , and BIB015 Policy search with learned models BIB020 ;  , ; , , Ng et al. (2004a,b) , and BIB058 Peters et al. ( ) et al., 2003  BIB061 , obstacle avoidance  for a mobile robot, very basic robot soccer BIB037 and multi-legged robot locomotion BIB016 BIB021 . Nevertheless, simulation biases can be addressed by introducing stochastic models or distributions over models even if the system is very close to deterministic. Artificially adding a little noise will smooth model errors and avoid policy overfitting BIB005 BIB013 . On the downside, potentially very precise policies may be eliminated due to their fragility in the presence of noise. This technique can be beneficial in all of the approaches described in this section. Nevertheless, in recent work, BIB069 presented an approach with strong guarantees for learning the model and policy in an iterative fashion even if the true system is not in the model class, indicating that it may be possible to deal with simulation bias.