Imitation Learning: A Survey of Learning Methods <s> Regression <s> ATR Human Information Science Laboratories, Kyoto, JapanEmail: ijspeert@usc.edu, jun@his.atr.co.jp, sschaal@usc.eduAbstractThis paper presents a new approach to the generationof rhythmic movement patterns with nonlinear dy-namical systems. Starting from a canonical limit cy-cle oscillator with well-de ned stability properties, wemodify the attractor landscape of the canonical sys-temby meansof statisticallearning methods toembedarbitrary smooth target patterns, however, withoutlosing the stability properties of the canonical sys-tem. In contrast to non-autonomous movement rep-resentations like splines, the learned pattern gener-ators remain autonomous dynamical systems whichrobustly cope with external perturbations that disruptthe time ow of the original pattern, and which canalso be modi ed on-line by additional perceptual vari-ables. A simple extension allows to cope with mul-tiple degrees-of-freedom (DOF) patterns, where allDOFs share the same fundamental frequency but,otherwise, can move in arbitrary phase and ampli-tude o sets to each other. We evaluate our meth-ods in learning from demonstration with an actual30 DOF humanoid robot. Figure-8 and drummingmovements are demonstrated by a human, recordedin joint angle space with an exoskeleton, and em-bedded in multi-dimensional rhythmic pattern gener-ators. The learned patterns can be used by the robotin various workspace locations and from arbitraryinitial conditions. Spatial and temporal invarianceof the pattern generators allow easy amplitude andspeed scaling without losing the qualitative signatureof a movement. This novel way of creating rhyth-mic patterns could tremendously facilitate rhythmicmovement generation, in particular in locomotion ofrobots and neural prosthetics in clinical applications. <s> BIB001 </s> Imitation Learning: A Survey of Learning Methods <s> Regression <s> Many control problems take place in continuous state-action spaces, e.g., as in manipulator robotics, where the control objective is often defined as finding a desired trajectory that reaches a particular goal state. While reinforcement learning offers a theoretical framework to learn such control policies from scratch, its applicability to higher dimensional continuous state-action spaces remains rather limited to date. Instead of learning from scratch, in this paper we suggest to learn a desired complex control policy by transforming an existing simple canonical control policy. For this purpose, we represent canonical policies in terms of differential equations with well-defined attractor properties. By nonlinearly transforming the canonical attractor dynamics using techniques from nonparametric regression, almost arbitrary new nonlinear policies can be generated without losing the stability properties of the canonical system. We demonstrate our techniques in the context of learning a set of movement skills for a humanoid robot from demonstrations of a human teacher. Policies are acquired rapidly, and, due to the properties of well formulated differential equations, can be re-used and modified on-line under dynamic changes of the environment. The linear parameterization of nonparametric regression moreover lends itself to recognize and classify previously learned movement skills. Evaluations in simulations and on an actual 30 degree-of-freedom humanoid robot exemplify the feasibility and robustness of our approach. <s> BIB002 </s> Imitation Learning: A Survey of Learning Methods <s> Regression <s> In this paper, we introduce a framework for learning biped locomotion using dynamical movement primitives based on non-linear oscillators. Our ultimate goal is to establish a design principle of a controller in order to achieve natural human-like locomotion. We suggest dynamical movement primitives as a central pattern generator (CPG) of a biped robot, an approach we have previously proposed for learning and encoding complex human movements. Demonstrated trajectories are learned through movement primitives by locally weighted regression, and the frequency of the learned trajectories is adjusted automatically by a novel frequency adaptation algorithm based on phase resetting and entrainment of coupled oscillators. Numerical simulations and experimental implementation on a physical robot demonstrate the effectiveness of the proposed locomotion controller. © 2004 Elsevier B.V. All rights reserved. <s> BIB003 </s> Imitation Learning: A Survey of Learning Methods <s> Regression <s> In the past, computational motor control has been approached from at least two major frameworks: the dynamic systems approach and the viewpoint of optimal control. The dynamic system approach emphasizes motor control as a process of self-organization between an animal and its environment. Nonlinear differential equations that can model entrainment and synchronization behavior are among the most favorable tools of dynamic systems modelers. In contrast, optimal control approaches view motor control as the evolutionary or development result of a nervous system that tries to optimize rather general organizational principles, e.g., energy consumption or accurate task achievement. Optimal control theory is usually employed to develop appropriate theories. Interestingly, there is rather little interaction between dynamic systems and optimal control modelers as the two approaches follow rather different philosophies and are often viewed as diametrically opposing. In this paper, we develop a computational approach to motor control that offers a unifying modeling framework for both dynamic systems and optimal control approaches. In discussions of several behavioral experiments and some theoretical and robotics studies, we demonstrate how our computational ideas allow both the representation of self-organizing processes and the optimization of movement based on reward criteria. Our modeling framework is rather simple and general, and opens opportunities to revisit many previous modeling results from this novel unifying view. <s> BIB004 </s> Imitation Learning: A Survey of Learning Methods <s> Regression <s> Many motor skills in humanoid robotics can be learned using parametrized motor primitives as done in imitation learning. However, most interesting motor learning problems are high-dimensional reinforcement learning problems often beyond the reach of current methods. In this paper, we extend previous work on policy learning from the immediate reward case to episodic reinforcement learning. We show that this results in a general, common framework also connected to policy gradient methods and yielding a novel algorithm for policy learning that is particularly well-suited for dynamic motor primitives. The resulting algorithm is an EM-inspired algorithm applicable to complex motor learning tasks. We compare this algorithm to several well-known parametrized policy search methods and show that it outperforms them. We apply it in the context of motor learning and show that it can learn a complex Ball-in-a-Cup task using a real Barrett WAM™ robot arm. <s> BIB005 </s> Imitation Learning: A Survey of Learning Methods <s> Regression <s> Dynamic system-based motor primitives have enabled robots to learn complex tasks ranging from Tennis-swings to locomotion. However, to date there have been only few extensions which have incorporated perceptual coupling to variables of external focus, and, furthermore, these modifications have relied upon handcrafted solutions. Humans learn how to couple their movement primitives with external variables. Clearly, such a solution is needed in robotics. In this paper, we propose an augmented version of the dynamic systems motor primitives which incorporates perceptual coupling to an external variable. The resulting perceptually driven motor primitives include the previous primitives as a special case and can inherit some of their interesting properties. We show that these motor primitives can perform complex tasks such a Ball-in-a-Cup or Kendama task even with large variances in the initial conditions where a skilled human player would be challenged. For doing so, we initialize the motor primitives in the traditional way by imitation learning without perceptual coupling. Subsequently, we improve the motor primitives using a novel reinforcement learning method which is particularly well-suited for motor primitives. <s> BIB006 </s> Imitation Learning: A Survey of Learning Methods <s> Regression <s> The acquisition and self-improvement of novel motor skills is among the most important problems in robotics. Motor primitives offer one of the most promising frameworks for the application of machine learning techniques in this context. Employing an improved form of the dynamic systems motor primitives originally introduced by Ijspeert et al. [2], we show how both discrete and rhythmic tasks can be learned using a concerted approach of both imitation and reinforcement learning. For doing so, we present both learning algorithms and representations targeted for the practical application in robotics. Furthermore, we show that it is possible to include a start-up phase in rhythmic primitives. We show that two new motor skills, i.e., Ball-in-a-Cup and Ball-Paddling, can be learned on a real Barrett WAM robot arm at a pace similar to human learning while achieving a significantly more reliable final performance. <s> BIB007 </s> Imitation Learning: A Survey of Learning Methods <s> Regression <s> Learning by imitation in humanoids is challenging due to the unpredictable environments these robots have to face during reproduction. Two sets of tools are relevant for this purpose: 1) probabilistic machine learning methods that can extract and exploit the regularities and important features of the task; and 2) dynamical systems that can cope with perturbation in real-time without having to replan the whole movement. We present a learning by imitation approach combining the two benefits. It is based on a superposition of virtual spring-damper systems to drive a humanoid robot's movement. The method relies on a statistical description of the springs attractor points acting in different candidate frames of reference. It extends dynamic movement primitives models by formulating the dynamical systems parameters estimation problem as a Gaussian mixture regression problem with projection in different coordinate systems. The robot exploits local variability information extracted from multiple demonstrations of movements to determine which frames are relevant for the task, and how the movement should be modulated with respect to these frames. The approach is tested on the new prototype of the COMAN compliant humanoid with time-based and time-invariant movements, including bimanual coordination skills. <s> BIB008 </s> Imitation Learning: A Survey of Learning Methods <s> Regression <s> Learning new motor tasks autonomously from interaction with a human being is an important goal for both robotics and machine learning. However, when moving beyond basic skills, most monolithic machine learning approaches fail to scale. In this paper, we take the task of learning table tennis as an example and present a new framework which allows a robot to learn cooperative table tennis from interaction with a human. Therefore, the robot first learns a set of elementary table tennis hitting movements from a human teacher by kinesthetic teach-in, which is compiled into a set of dynamical system motor primitives (DMPs). Subsequently, the system generalizes these movements to a wider range of situations using our mixture of motor primitives (MoMP) approach. The resulting policy enables the robot to select appropriate motor primitives as well as to generalize between them. Finally, the robot plays with a human table tennis partner and learns online to improve its behavior. <s> BIB009 </s> Imitation Learning: A Survey of Learning Methods <s> Regression <s> Human-robot collaboration seeks to have humans and robots closely interacting in everyday situations. For some tasks, physical contact between the user and the robot may occur, originating significant challenges at safety, cognition, perception and control levels, among others. This paper focuses on robot motion adaptation to parameters of a collaborative task, extraction of the desired robot behavior, and variable impedance control for human-safe interaction. We propose to teach a robot cooperative behaviors from demonstrations, which are probabilistically encoded by a task-parametrized formulation of a Gaussian mixture model. Such encoding is later used for specifying both the desired state of the robot, and an optimal feedback control law that exploits the variability in position, velocity and force spaces observed during the demonstrations. The whole framework allows the robot to modify its movements as a function of parameters of the task, while showing different impedance behaviors. Tests were successfully carried out in a scenario where a 7 DOF backdrivable manipulator learns to cooperate with a human to transport an object. <s> BIB010 </s> Imitation Learning: A Survey of Learning Methods <s> Regression <s> Predictive modeling of human or humanoid movement becomes increasingly complex as the dimensionality of those movements grows. Dynamic Movement Primitives (DMP) have been shown to be a powerful method of representing such movements, but do not generalize well when used in configuration or task space. To solve this problem we propose a model called autoencoded dynamic movement primitive (AE-DMP) which uses deep autoencoders to find a representation of movement in a latent feature space, in which DMP can optimally generalize. The architecture embeds DMP into such an autoencoder and allows the whole to be trained as a unit. To further improve the model for multiple movements, sparsity is added for the feature layer neurons; therefore, various movements can be observed clearly in the feature space. After training, the model finds a single hidden neuron from the sparsity that can efficiently generate new movements. Our experiments clearly demonstrate the efficiency of missing data imputation using 50-dimensional human movement data. <s> BIB011
Regression methods are used to learn actions in a continuous space. Unlike classification, regression methods map the input state to a numeric output that represents an action. Thus, they are suitable for low-level motor actions rather than higher-level decisions, especially when actions are represented as continuous values, such as input from a joystick . The regressor I(x) maps an independent sample x to a continuous value y rather than a set of classes, where y ∈ R, the set of real numbers. Similarly, the regressor is trained using a set of labeled samples (x (i) , y (i) ), where A commonly used technique is locally weighted regression (LWR). LWR is suitable for learning trajectories, as these motions are made up of sequences of continuous values. Examples of such motions are batting tasks BIB005 BIB001 , where the agent is required to execute a motion trajectory in order to pass by a point and hit a target, and walking BIB003 , where the agent needs to produce a trajectory that results in smooth stable motion. A more comprehensive application is table tennis. BIB009 use Linear Bayesian Regression to teach a robot arm to play a continuous game of table tennis. The agent is required to move with precision in a continuous 3D space in different situations, such as when hitting the ball, recovering position after a hit, and preparing for the opponent's next move. Another paradigm commonly used for regression is artificial neural networks (ANNs). Neural networks differ from other regression techniques in that they are demanding in training time and training samples. Neural network approaches are often inspired by biology and neuroscience studies and attempt to emulate the learning and imitation process in humans and animals . The use of regression with a dynamic system of motor primitives has produced a number of applications for learning discrete and rhythmic motions BIB002 BIB004 BIB006 , though most approaches focus on direct imitation without further optimization BIB007 . In such applications, a dynamic system represents a single DOF as each DOF has a different goal and constraints BIB004 . Dynamic systems can be combined with probabilistic machine-learning methods to reap the benefits of both approaches. This allows the extraction of patterns that are important to a given task and generalization to different scenarios while maintaining the ability to adapt and correct movement trajectories in real time BIB008 . In BIB008 , the estimation of dynamical systems' parameters is represented as a Gaussian mixture regression (GMR) problem. This approach has an advantage over LWR-based approaches as it allows learning of the activation functions along with the motor actions. The proposed method is used to learn time-based and time-invariant movement. In BIB010 , a similar GMM-based method is used in a task-parameterized framework that allows shaping the robot's motion as a function of the task parameters. Human demonstrations are encoded to reflect parameters that are relevant to the task at hand and identify the position, velocity, and force constraints of the task. This encoding allows the framework to derive the state in which the robot should be and optimize the movement of the robot accordingly. This approach is used in a Human Robot Collaboration (HRC) context and aims to optimize human intervention as well as robot effort. Deep learning is combined with dynamical systems in BIB011 . Dynamic movement primitives (DMPs) are embedded into autoencoders that learn representations of movement from demonstrated data. Autoencoders nonlinearly map features to a lower-dimensional latent space in the hidden layer. However, in this approach, the hidden units are constrained to DMPs to limit the hidden layer into representing the dynamics of the system. In both classification and regression methods, a design decision can be made regarding the learning models' resources. Lazy learners such as kNN and LWR do not need training but need to retain all training samples when performing predictions. On the other hand, trained models such as ANN and SVM require training time, but once a model is created, the training samples are no longer needed and only the model is stored, which saves memory. These models can also result in very short prediction times. 