Imitation Learning: A Survey of Learning Methods <s> Reinforcement Learning <s> Programming robots is a tedious task. So, there is growing interest in building robots which can learn by themselves. Self-improving, which involves trial and error, however, is often a slow process and could be hazardous in a hostile environment. By teaching robots how tasks can be achieved, learning time can be shortened and hazard can be minimized. This paper presents a general approach to making robots which can improve their performance from experiences as well as from being taught. Based on this proposed approach and other learning speedup techniques, a simulated learning robot was developed and could learn three moderately complex behaviors, which were then integrated in a subsumption style so that the robot could navigate and recharge itself. Interestingly, a real robot could actually use what was learned in the simulator to operate in the real world quite successfully. <s> BIB001 </s> Imitation Learning: A Survey of Learning Methods <s> Reinforcement Learning <s> ATR Human Information Science Laboratories, Kyoto, JapanEmail: ijspeert@usc.edu, jun@his.atr.co.jp, sschaal@usc.eduAbstractThis paper presents a new approach to the generationof rhythmic movement patterns with nonlinear dy-namical systems. Starting from a canonical limit cy-cle oscillator with well-de ned stability properties, wemodify the attractor landscape of the canonical sys-temby meansof statisticallearning methods toembedarbitrary smooth target patterns, however, withoutlosing the stability properties of the canonical sys-tem. In contrast to non-autonomous movement rep-resentations like splines, the learned pattern gener-ators remain autonomous dynamical systems whichrobustly cope with external perturbations that disruptthe time ow of the original pattern, and which canalso be modi ed on-line by additional perceptual vari-ables. A simple extension allows to cope with mul-tiple degrees-of-freedom (DOF) patterns, where allDOFs share the same fundamental frequency but,otherwise, can move in arbitrary phase and ampli-tude o sets to each other. We evaluate our meth-ods in learning from demonstration with an actual30 DOF humanoid robot. Figure-8 and drummingmovements are demonstrated by a human, recordedin joint angle space with an exoskeleton, and em-bedded in multi-dimensional rhythmic pattern gener-ators. The learned patterns can be used by the robotin various workspace locations and from arbitraryinitial conditions. Spatial and temporal invarianceof the pattern generators allow easy amplitude andspeed scaling without losing the qualitative signatureof a movement. This novel way of creating rhyth-mic patterns could tremendously facilitate rhythmicmovement generation, in particular in locomotion ofrobots and neural prosthetics in clinical applications. <s> BIB002 </s> Imitation Learning: A Survey of Learning Methods <s> Reinforcement Learning <s> To date, reinforcement learning has mostly been studied solving simple learning tasks. Reinforcement learning methods that have been studied so far typically converge slowly. The purpose of this work is thus two-fold: 1) to investigate the utility of reinforcement learning in solving much more complicated learning tasks than previously studied, and 2) to investigate methods that will speed up reinforcement learning. ::: ::: This paper compares eight reinforcement learning frameworks: adaptive heuristic critic (AHC) learning due to Sutton, Q-learning due to Watkins, and three extensions to both basic methods for speeding up learning. The three extensions are experience replay, learning action models for planning, and teaching. The frameworks were investigated using connectionism as an approach to generalization. To evaluate the performance of different frameworks, a dynamic environment was used as a testbed. The environment is moderately complex and nondeterministic. This paper describes these frameworks and algorithms in detail and presents empirical evaluation of the frameworks. <s> BIB003 </s> Imitation Learning: A Survey of Learning Methods <s> Reinforcement Learning <s> We consider learning in a Markov decision process where we are not explicitly given a reward function, but where instead we can observe an expert demonstrating the task that we want to learn to perform. This setting is useful in applications (such as the task of driving) where it may be difficult to write down an explicit reward function specifying exactly how different desiderata should be traded off. We think of the expert as trying to maximize a reward function that is expressible as a linear combination of known features, and give an algorithm for learning the task demonstrated by the expert. Our algorithm is based on using "inverse reinforcement learning" to try to recover the unknown reward function. We show that our algorithm terminates in a small number of iterations, and that even though we may never recover the expert's reward function, the policy output by the algorithm will attain performance close to that of the expert, where here performance is measured with respect to the expert's unknown reward function. <s> BIB004 </s> Imitation Learning: A Survey of Learning Methods <s> Reinforcement Learning <s> This paper presents a machine learning approach to optimizing a quadrupedal trot gait for forward speed. Given a parameterized walk designed for a specific robot, we propose using a form of policy gradient reinforcement learning to automatically search the set of possible parameters with the goal of finding the fastest possible walk. We implement and test our approach on a commercially available quadrupedal robot platform, namely the Sony Aibo robot. After about three hours of learning, all on the physical robots and with no human intervention other than to change the batteries, the robots achieved a gait faster than any previously known gait known for the Aibo, significantly outperforming a variety of existing hand-coded and learned solutions. <s> BIB005 </s> Imitation Learning: A Survey of Learning Methods <s> Reinforcement Learning <s> The goal of developing algorithms for programming robots by demonstration is to create an easy way of programming robots that can be accomplished by everyone. When a demonstrator teaches a task to a robot, he/she shows some ways of fulfilling the task, but not all the possibilities. The robot must then be able to reproduce the task even when unexpected perturbations occur. In this case, it has to learn a new solution. In this paper, we describe a system to teach to the robot constrained reaching tasks. Our system is based on a dynamical system generator modulated with a learned speed trajectory and combined with a reinforcement learning module to allow the robot to adapt the trajectory when facing a new situation, such as avoiding obstacles. <s> BIB006 </s> Imitation Learning: A Survey of Learning Methods <s> Reinforcement Learning <s> Autonomous learning is one of the hallmarks of human and animal behavior, and understanding the principles of learning will be crucial in order to achieve true autonomy in advanced machines like humanoid robots. In this paper, we examine learning of complex motor skills with human-like limbs. While supervised learning can offer useful tools for bootstrapping behavior, e.g., by learning from demonstration, it is only reinforcement learning that offers a general approach to the final trial-and-error improvement that is needed by each individual acquiring a skill. Neither neurobiological nor machine learning studies have, so far, offered compelling results on how reinforcement learning can be scaled to the high-dimensional continuous state and action spaces of humans or humanoids. Here, we combine two recent research developments on learning motor control in order to achieve this scaling. First, we interpret the idea of modular motor control by means of motor primitives as a suitable way to generate parameterized control policies for reinforcement learning. Second, we combine motor primitives with the theory of stochastic policy gradient learning, which currently seems to be the only feasible framework for reinforcement learning for humanoids. We evaluate different policy gradient methods with a focus on their applicability to parameterized motor primitives. We compare these algorithms in the context of motor primitive learning, and show that our most modern algorithm, the Episodic Natural Actor-Critic outperforms previous algorithms by at least an order of magnitude. We demonstrate the efficiency of this reinforcement learning method in the application of learning to hit a baseball with an anthropomorphic robot arm. <s> BIB007 </s> Imitation Learning: A Survey of Learning Methods <s> Reinforcement Learning <s> One of the hallmarks of the performance, versatility, and robustness of biological motor control is the ability to adapt the impedance of the overall biomechanical system to different task requirements and stochastic disturbances. A transfer of this principle to robotics is desirable, for instance to enable robots to work robustly and safely in everyday human environments. It is, however, not trivial to derive variable impedance controllers for practical high degree-of-freedom (DOF) robotic tasks. In this contribution, we accomplish such variable impedance control with the reinforcement learning (RL) algorithm PI2 (Policy Improvement with Path Integrals). PI2 is a model-free, sampling-based learning method derived from first principles of stochastic optimal control. The PI 2 algorithm requires no tuning of algorithmic parameters besides the exploration noise. The designer can thus fully focus on the cost function design to specify the task. From the viewpoint of robotics, a particular useful property of PI2 is that it can scale to problems of many DOFs, so that reinforcement learning on real robotic systems becomes feasible. We sketch the PI2 algorithm and its theoretical properties, and how it is applied to gain scheduling for variable impedance control. We evaluate our approach by presenting results on several simulated and real robots. We consider tasks involving accurate tracking through via points, and manipulation tasks requiring physical contact with the environment. In these tasks, the optimal strategy requires both tuning of a reference trajectory and the impedance of the end-effector. The results show that we can use path integral based reinforcement learning not only for planning but also to derive variable gain feedback controllers in realistic scenarios. Thus, the power of variable impedance control is made available to a wide variety of robotic systems and practical applications. <s> BIB008 </s> Imitation Learning: A Survey of Learning Methods <s> Reinforcement Learning <s> Learning complex motor skills for real world tasks is a hard problem in robotic manipulation that often requires painstaking manual tuning and design by a human expert. In this work, we present a Reinforcement Learning based approach to acquiring new motor skills from demonstration. Our approach allows the robot to learn fine manipulation skills and significantly improve its success rate and skill level starting from a possibly coarse demonstration. Our approach aims to incorporate task domain knowledge, where appropriate, by working in a space consistent with the constraints of a specific task. In addition, we also present an approach to using sensor feedback to learn a predictive model of the task outcome. This allows our system to learn the proprioceptive sensor feedback needed to monitor subsequent executions of the task online and abort execution in the event of predicted failure. We illustrate our approach using two example tasks executed with the PR2 dual-arm robot: a straight and accurate pool stroke and a box flipping task using two chopsticks as tools. <s> BIB009 </s> Imitation Learning: A Survey of Learning Methods <s> Reinforcement Learning <s> Learning new motor tasks autonomously from interaction with a human being is an important goal for both robotics and machine learning. However, when moving beyond basic skills, most monolithic machine learning approaches fail to scale. In this paper, we take the task of learning table tennis as an example and present a new framework which allows a robot to learn cooperative table tennis from interaction with a human. Therefore, the robot first learns a set of elementary table tennis hitting movements from a human teacher by kinesthetic teach-in, which is compiled into a set of dynamical system motor primitives (DMPs). Subsequently, the system generalizes these movements to a wider range of situations using our mixture of motor primitives (MoMP) approach. The resulting policy enables the robot to select appropriate motor primitives as well as to generalize between them. Finally, the robot plays with a human table tennis partner and learns online to improve its behavior. <s> BIB010 </s> Imitation Learning: A Survey of Learning Methods <s> Reinforcement Learning <s> Direct policy search can effectively scale to high-dimensional systems, but complex policies with hundreds of parameters often present a challenge for such methods, requiring numerous samples and often falling into poor local optima. We present a guided policy search algorithm that uses trajectory optimization to direct policy learning and avoid poor local optima. We show how differential dynamic programming can be used to generate suitable guiding samples, and describe a regularized importance sampled policy optimization that incorporates these samples into the policy search. We evaluate the method by learning neural network controllers for planar swimming, hopping, and walking, as well as simulated 3D humanoid running. <s> BIB011 </s> Imitation Learning: A Survey of Learning Methods <s> Reinforcement Learning <s> The combination of modern Reinforcement Learning and Deep Learning approaches holds the promise of making significant progress on challenging applications requiring both rich perception and policy-selection. The Arcade Learning Environment (ALE) provides a set of Atari games that represent a useful benchmark set of such applications. A recent breakthrough in combining model-free reinforcement learning with deep learning, called DQN, achieves the best real-time agents thus far. Planning-based approaches achieve far higher scores than the best model-free approaches, but they exploit information that is not available to human players, and they are orders of magnitude slower than needed for real-time play. Our main goal in this work is to build a better real-time Atari game playing agent than DQN. The central idea is to use the slow planning-based agents to provide training data for a deep-learning architecture capable of real-time play. We proposed new agents based on this idea and show that they outperform DQN. <s> BIB012 </s> Imitation Learning: A Survey of Learning Methods <s> Reinforcement Learning <s> Reinforcement learning describes how a learning agent can achieve optimal behaviour based on interactions with its environment and reward feedback. A limiting factor in reinforcement learning as employed in artificial intelligence is the need for an often prohibitively large number of environment samples before the agent reaches a desirable level of performance. Learning from demonstration is an approach that provides the agent with demonstrations by a supposed expert, from which it should derive suitable behaviour. Yet, one of the challenges of learning from demonstration is that no guarantees can be provided for the quality of the demonstrations, and thus the learned behavior. In this paper, we investigate the intersection of these two approaches, leveraging the theoretical guarantees provided by reinforcement learning, and using expert demonstrations to speed up this learning by biasing exploration through a process called reward shaping. This approach allows us to leverage human input without making an erroneous assumption regarding demonstration optimality. We show experimentally that this approach requires significantly fewer demonstrations, is more robust against suboptimality of demonstrations, and achieves much faster learning than the recently developed HAT algorithm. <s> BIB013 </s> Imitation Learning: A Survey of Learning Methods <s> Reinforcement Learning <s> Policy learning for partially observed control tasks requires policies that can remember salient information from past observations. In this paper, we present a method for learning policies with internal memory for high-dimensional, continuous systems, such as robotic manipulators. Our approach consists of augmenting the state and action space of the system with continuous-valued memory states that the policy can read from and write to. Learning general-purpose policies with this type of memory representation directly is difficult, because the policy must automatically figure out the most salient information to memorize at each time step. We show that, by decomposing this policy search problem into a trajectory optimization phase and a supervised learning phase through a method called guided policy search, we can acquire policies with effective memorization and recall strategies. Intuitively, the trajectory optimization phase chooses the values of the memory states that will make it easier for the policy to produce the right action in future states, while the supervised learning phase encourages the policy to use memorization actions to produce those memory states. We evaluate our method on tasks involving continuous control in manipulation and navigation settings, and show that our method can learn complex policies that successfully complete a range of tasks that require memory. <s> BIB014
Reinforcement learning (RL) learns a policy to solve a problem via trial and error. Definition 12. In RL, an agent is modeled as an MDP that learns to navigate in a state space. A finite MDP consists of a tuple (S, A, T , R), where S is a finite set of states, A is the set of possible actions, T is the set of state transition probabilities, and R is a reward function. TP sa contains a set of probabilities where P s a is the probability of arriving at state s given action a and where a ∈ A and s ∈ S. The reward function R(s k , a k , s k+1 ) returns an immediate reward for taking an action in a given state and ending up in a new state a k → s k+1 , where k is the time step. This reward is discounted over time by a discount factor γ ∈ [0, 1) and the goal of the agent is to maximize the expected discounted reward at each time step. RL starts off with a random policy and modifies its parameters based on rewards gained from executing this policy. Reinforcement learning can be used on its own to learn a policy for a variety of robotic applications. However, if a policy is learned from demonstration, reinforcement learning can be applied to fine-tune the parameters. Providing positive or negative examples to train a policy helps reinforcement learning by reducing the search space available . Enhancing the policy using RL is sometimes necessary if there are physical discrepancies between the teacher and the learner or to alleviate errors in acquiring the demonstrations. RL can also be useful to train the policy for unseen situations that are not covered in the demonstrations. Applying reinforcement learning to the learned policy instead of a random one can significantly speed up the RL process and avoids the risk of the policy from converging into a local minimum. Moreover, RL can find a policy to perform a task that does not look normal to the human observer. In applications where the learner interacts with a human, it is important for the user to intuitively recognize the agent's actions. This is common in cases where robots are introduced into established environments (such as homes and offices) to interact with untrained human users . By applying reinforcement learning to a policy learned from human demonstrations, the problem of unfamiliar behavior can be avoided. In imitation learning methods, reinforcement learning is often combined with learning from demonstrations to improve a learned policy when the fitness of the performed task can be evaluated. In early research, teaching using demonstrations of successful actions was used to improve and speed up reinforcement learning. In BIB003 , reinforcement learning is used to learn a policy to play a game in a 2D dynamic environment. Different methods for enhancing the RL policy are examined. The results demonstrate that teaching the learner with demonstrations improves its score and helps prevent the learner from falling in local minima. It is also noted that the improvement from teaching increases with the difficulty of the task. Solutions to simple problems can be easily inferred without requiring demonstrations from an expert. But as the complexity of the task increases, the advantage of learning from demonstrations becomes more significant, and even necessary for successful learning in more difficult tasks BIB001 ]. In BIB006 , GMR is used to train a robot on an object grasping task. Since unseen scenarios such as obstacles and the variable location of the object are expected in this application, reinforcement learning is used to explore new ways to perform the task. The trained system is a dynamic system that performs damping on the imitated trajectories. This allows the robot to smoothly achieve its target and prevents reinforcement learning from resulting in oscillations. Using damping in dynamic systems is a common approach when combining imitation learning and reinforcement learning  ]. An impressive application of imitation and reinforcement learning is training an agent to play the board game "Go" that rivals human experts . A deep convolutional neural network is trained using past games. Then reinforcement learning is used to refine the weights of the network and improve the policy. A different approach to combine learning from demonstrations with reinforcement learning is employed in BIB013 . Rather than using the demonstrations to train an initial policy, they are used to derive prior knowledge for reward shaping  . A reward function is used to encourage subachievements in the task, such as milestones reached in the demonstrations. This reward function is combined with the primary reward function to supply the agent with the cost of its actions. This paradigm of using expert demonstrations to derive a reward function is similar to inverse reinforcement learning approaches BIB004 . Policy search methods are a subset of reinforcement learning that lend themselves naturally to robotic applications as they scale to high-dimensional MDPs . Therefore, policy search methods are a good fit to integrate with imitation learning methods. A policy gradient method is used in BIB005 to improve an existing policy that can be created through supervised learning or explicit programming. A similar approach BIB007 ] is used within a dynamic system that was previously used for supervised learning from demonstrations BIB002 . This led to a series of work that utilizes the dynamic system in Ijspeert et al. [2002b] to learn from demonstrations and subsequently use reinforcement learning for self-improvement Peters 2010, 2014; BIB008 BIB009 ]. This framework is used to teach robotic arms a number of applications such as ball in cup, ball paddling Peters 2009c, 2010] , and playing table tennis BIB010 . Rather than using reinforcement learning to refine a policy trained from demonstrations, demonstrations can be used to guide the policy search. In BIB011 , differential dynamic programming is used to generate guiding samples from human demonstrations. These guiding samples help the policy search explore high reward regions of the policy space. Recurrent neural networks are incorporated into guided policy search in BIB014 to facilitate dealing with partially observed problems. Past memories are augmented to the state space and are considered when predicting the next action. A supervised approach uses demonstrated trajectories to decide which memories to store, while reinforcement learning is used to optimize the policy including the memory state values. A different way to utilize reinforcement learning in imitation learning is to use RL to provide demonstrations for direct imitation. This approach does not need a human teacher as the policy is learned from scratch using trial and error and then used to generate demonstrations for training. One reason for generating demonstrations and training a supervised model rather than using the RL policy directly is that the RL method does not act in real time BIB012 . Another situation is when the RL policy is learned in a controlled environment. In  , reinforcement learning is used to learn a variety of robotic tasks in a controlled environment. Information such as the position of target objects is available during this phase. A deep convolutional neural network is then trained using demonstrations from the RL policy. The neural network learns to map visual input to actions and thus learns to perform the tasks without the information needed in the RL phase. This mimics human demonstrations as humans utilize expert knowledge-that is not incorporated in the training process-to provide demonstrations. For a comprehensive survey of reinforcement learning in robotics, the reader is referred to .