Reinforcement learning in robotics: A survey <s> Demonstrations by a teacher <s> This paper describes two experiments with supervised reinforcement learning (RL) on a real, mobile robot. Two types of experiments were preformed. One tests the robot's reliability in implementing a navigation task it has been taught by a supervisor. The other, in which new obstacles are placed along the previously learned path to the goal, measures the robot's robustness to changes in environment. Supervision consisted of human-guided, remote-controlled runs through a navigation task during the initial stages of reinforcement learning. The RL algorithms deployed enabled the robot to learn a path to a goal yet retain the ability to explore different solutions when confronted with a new obstacle. Experimental analysis was based on measurements of average time to reach the goal, the number of failed states encountered during an episode, and how closely the RL learner matched the supervisor's actions. <s> BIB001 </s> Reinforcement learning in robotics: A survey <s> Demonstrations by a teacher <s> The goal of developing algorithms for programming robots by demonstration is to create an easy way of programming robots that can be accomplished by everyone. When a demonstrator teaches a task to a robot, he/she shows some ways of fulfilling the task, but not all the possibilities. The robot must then be able to reproduce the task even when unexpected perturbations occur. In this case, it has to learn a new solution. In this paper, we describe a system to teach to the robot constrained reaching tasks. Our system is based on a dynamical system generator modulated with a learned speed trajectory and combined with a reinforcement learning module to allow the robot to adapt the trajectory when facing a new situation, such as avoiding obstacles. <s> BIB002 </s> Reinforcement learning in robotics: A survey <s> Demonstrations by a teacher <s> Dynamic system-based motor primitives have enabled robots to learn complex tasks ranging from Tennis-swings to locomotion. However, to date there have been only few extensions which have incorporated perceptual coupling to variables of external focus, and, furthermore, these modifications have relied upon handcrafted solutions. Humans learn how to couple their movement primitives with external variables. Clearly, such a solution is needed in robotics. In this paper, we propose an augmented version of the dynamic systems motor primitives which incorporates perceptual coupling to an external variable. The resulting perceptually driven motor primitives include the previous primitives as a special case and can inherit some of their interesting properties. We show that these motor primitives can perform complex tasks such a Ball-in-a-Cup or Kendama task even with large variances in the initial conditions where a skilled human player would be challenged. For doing so, we initialize the motor primitives in the traditional way by imitation learning without perceptual coupling. Subsequently, we improve the motor primitives using a novel reinforcement learning method which is particularly well-suited for motor primitives. <s> BIB003 </s> Reinforcement learning in robotics: A survey <s> Demonstrations by a teacher <s> Many motor skills in humanoid robotics can be learned using parametrized motor primitives as done in imitation learning. However, most interesting motor learning problems are high-dimensional reinforcement learning problems often beyond the reach of current methods. In this paper, we extend previous work on policy learning from the immediate reward case to episodic reinforcement learning. We show that this results in a general, common framework also connected to policy gradient methods and yielding a novel algorithm for policy learning that is particularly well-suited for dynamic motor primitives. The resulting algorithm is an EM-inspired algorithm applicable to complex motor learning tasks. We compare this algorithm to several well-known parametrized policy search methods and show that it outperforms them. We apply it in the context of motor learning and show that it can learn a complex Ball-in-a-Cup task using a real Barrett WAM™ robot arm. <s> BIB004 </s> Reinforcement learning in robotics: A survey <s> Demonstrations by a teacher <s> Autonomous helicopter flight is widely regarded to be a highly challenging control problem. As helicopters are highly unstable and exhibit complicated dynamical behavior, it is particularly difficult to design controllers that achieve high performance over a broad flight regime. While these aircraft are notoriously difficult to control, there are expert human pilots who are nonetheless capable of demonstrating a wide variety of maneuvers, including aerobatic maneuvers at the edge of the helicopter's performance envelope. In this paper, we present algorithms for modeling and control that leverage these demonstrations to build high-performance control systems for autonomous helicopters. More specifically, we detail our experiences with the Stanford Autonomous Helicopter, which is now capable of extreme aerobatic flight meeting or exceeding the performance of our own expert pilot. <s> BIB005 </s> Reinforcement learning in robotics: A survey <s> Demonstrations by a teacher <s> While today's industrial robots achieve high performance in a wide range of tasks, the programming and adaptation of such robots to new tasks is still time-consuming and far from convenient. This shortcoming is a major obstacle for the deployment of robots in small to medium-size companies and also for service applications. Robot learning by imitation has been identified as an important paradigm to tackle this practicability issue. However, such approaches are only applicable in controlled static scenarios. To perform less constrained tasks, the robot needs the ability to generalize its skills and to adapt to changing conditions. It is thus a natural idea to formulate the problem as learning from experience and to incorporate demonstrations by an expert in the learning process. We present a new framework for learning of motion skills that seamlessly integrates demonstration learning from an expert teacher and further optimization of the demonstrated skill by own experience. Our method employs Gaussian Process regression to generalize a measure of performance to new skills that are close to already learned skills in the state-action space. We evaluate our approach for the task of grasping an object. <s> BIB006 </s> Reinforcement learning in robotics: A survey <s> Demonstrations by a teacher <s> Reinforcement learning in the high-dimensional, continuous spaces typical in robotics, remains a challenging problem. To overcome this challenge, a popular approach has been to use demonstrations to find an appropriate initialisation of the policy in an attempt to reduce the number of iterations needed to find a solution. Here, we present an alternative way to incorporate prior knowledge from demonstrations of individual postures into learning, by extracting the inherent problem structure to find an efficient state representation. In particular, we use probabilistic, nonlinear dimensionality reduction to capture latent constraints present in the data. By learning policies in the learnt latent space, we are able to solve the planning problem in a reduced space that automatically satisfies task constraints. As shown in our experiments, this reduces the exploration needed and greatly accelerates the learning. We demonstrate our approach for learning a bimanual reaching task on the 19-DOF KHR-1HV humanoid. <s> BIB007 </s> Reinforcement learning in robotics: A survey <s> Demonstrations by a teacher <s> One of the hallmarks of the performance, versatility, and robustness of biological motor control is the ability to adapt the impedance of the overall biomechanical system to different task requirements and stochastic disturbances. A transfer of this principle to robotics is desirable, for instance to enable robots to work robustly and safely in everyday human environments. It is, however, not trivial to derive variable impedance controllers for practical high degree-of-freedom (DOF) robotic tasks. In this contribution, we accomplish such variable impedance control with the reinforcement learning (RL) algorithm PI2 (Policy Improvement with Path Integrals). PI2 is a model-free, sampling-based learning method derived from first principles of stochastic optimal control. The PI 2 algorithm requires no tuning of algorithmic parameters besides the exploration noise. The designer can thus fully focus on the cost function design to specify the task. From the viewpoint of robotics, a particular useful property of PI2 is that it can scale to problems of many DOFs, so that reinforcement learning on real robotic systems becomes feasible. We sketch the PI2 algorithm and its theoretical properties, and how it is applied to gain scheduling for variable impedance control. We evaluate our approach by presenting results on several simulated and real robots. We consider tasks involving accurate tracking through via points, and manipulation tasks requiring physical contact with the environment. In these tasks, the optimal strategy requires both tuning of a reference trajectory and the impedance of the end-effector. The results show that we can use path integral based reinforcement learning not only for planning but also to derive variable gain feedback controllers in realistic scenarios. Thus, the power of variable impedance control is made available to a wide variety of robotic systems and practical applications. <s> BIB008
Demonstrations by a teacher can be obtained in two different scenarios. In the first, the teacher demonstrates the task using his or her own body; in the second, the teacher controls the robot to do the task. The first scenario is limited by the embodiment issue, as the movement of a human teacher usually cannot be mapped directly to the robot due to different physical constraints and capabilities. For example, joint angles of a human demonstrator need to be adapted to account for the kinematic differences between the teacher and the robot. Often it is more advisable to only consider task-relevant information, such as the Cartesian positions and velocities of the end-effector and the object. Demonstrations obtained by motion-capture have been used to learn a pendulum swing-up  , ball in a cup BIB003 , and grasping BIB006 . The second scenario obtains demonstrations by a human teacher directly controlling the robot. Here the human teacher first has to learn how to achieve a task with the particular robot's hardware, adding valuable prior knowledge. For example, remotely controlling the robot initialized a Qtable for a navigation task BIB001 . If the robot is back-drivable, kinesthetic teach-in (i.e. by taking it by the hand and moving it) can be employed, which enables the teacher to interact more closely with the robot. This method has resulted in applications including T-ball batting (Peters and Schaal, 2008c,b) , reaching tasks BIB002 BIB007 , ball in a cup BIB004 , flipping a light switch BIB008 , playing pool and manipulating a box , and opening a door and picking up objects . A marble maze task can be learned using demonstrations by a human player . One of the more stunning demonstrations of the benefit of learning from a teacher is the helicopter airshows of BIB005 . This approach combines initial human demonstration of trajectories, machine learning to extract approximate models from multiple trajectories, and classical locally optimal control methods  to achieve state-of-the-art acrobatic flight.