Imitation Learning: A Survey of Learning Methods <s> Handling Demonstrations <s> This paper presents a method to learn discrete robot motions from a set of demonstrations. We model a motion as a nonlinear autonomous (i.e., time-invariant) dynamical system (DS) and define sufficient conditions to ensure global asymptotic stability at the target. We propose a learning method, which is called Stable Estimator of Dynamical Systems (SEDS), to learn the parameters of the DS to ensure that all motions closely follow the demonstrations while ultimately reaching and stopping at the target. Time-invariance and global asymptotic stability at the target ensures that the system can respond immediately and appropriately to perturbations that are encountered during the motion. The method is evaluated through a set of robot experiments and on a library of human handwriting motions. <s> BIB001 </s> Imitation Learning: A Survey of Learning Methods <s> Handling Demonstrations <s> Robot Learning from Demonstration (RLfD) seeks to enable lay users to encode desired robot behaviors as autonomous controllers. Current work uses a human’s demonstration of the target task to initialize the robot’s policy, and then improves its performance either through practice (with a known reward function), or additional human interaction. In this article, we focus on the initialization step and consider what can be learned when the humans do not provide successful examples. We develop probabilistic approaches that avoid reproducing observed failures while leveraging the variance across multiple attempts to drive exploration. Our experiments indicate that failure data do contain information that can be used to discover successful means to accomplish tasks. However, in higher dimensions, additional information from the user will most likely be necessary to enable efficient failure-based learning. <s> BIB002 </s> Imitation Learning: A Survey of Learning Methods <s> Handling Demonstrations <s> Humans are known to manage postural movements in a very elegant manner. In the task of standing up from a chair, a humanoid robot can benefit from the variability of human demonstrations. In this paper we propose a novel method for humanoid robots to imitate a dynamic postural movement demonstrated by humans. Since the kinematics of human participants and the humanoid robot used in this experiment are different, we solve the correspondence problem by making comparisons in a common reward space defined by a multimodal reward function composed of balance and effort terms. We fitted a fully actuated triple inverted pendulum to model both human and robot. We used Differential Evolution to find the optimal articular trajectory that minimizes the Kullback-Leibler difference between the human's and robot's reward profile subject to constraints. <s> BIB003 </s> Imitation Learning: A Survey of Learning Methods <s> Handling Demonstrations <s> We propose a Learning from Demonstration (LfD) algorithm which leverages expert data, even if they are very few or inaccurate. We achieve this by using both expert data, as well as reinforcement signals gathered through trial-and-error interactions with the environment. The key idea of our approach, Approximate Policy Iteration with Demonstration (APID), is that expert's suggestions are used to define linear constraints which guide the optimization performed by Approximate Policy Iteration. We prove an upper bound on the Bellman error of the estimate computed by APID at each iteration. Moreover, we show empirically that APID outperforms pure Approximate Policy Iteration, a state-of-the-art LfD algorithm, and supervised learning in a variety of scenarios, including when very few and/or suboptimal demonstrations are available. Our experiments include simulations as well as a real robot path-finding task. <s> BIB004 </s> Imitation Learning: A Survey of Learning Methods <s> Handling Demonstrations <s> This paper proposes a framework of generating behavior sequences for robots, especially for humanoid robots, to perform complex tasks. This framework provides a method for the robot to generalize common features of demonstrated behaviors, to store the learned behaviors in the memory system, to construct a behavior graph to describe relationships among learned behaviors, to find and assemble a behavior sequence, and to generate similar motion trajectories of basic behaviors when it is placed in a similar but slightly different task-relevant situation. Additionally, we successfully use behavior graph to describe the dynamic relationship among behaviors and we successfully apply shortest path searching methods in behavior sequence generation, which provides a novel solution to associate knowledge representation with behavior generation. Simulation and experiments are carried on a humanoid robot to validate our proposed framework. <s> BIB005 </s> Imitation Learning: A Survey of Learning Methods <s> Handling Demonstrations <s> Existing techniques for motion imitation often suffer a certain level of latency due to their computational overhead or a large set of correspondence samples to search. To achieve real-time imitation with small latency, we present a framework in this paper to reconstruct motion on humanoids based on sparsely sampled correspondence. The imitation problem is formulated as finding the projection of a point from the configuration space of a human's poses into the configuration space of a humanoid. An optimal projection is defined as the one that minimizes a back-projected deviation among a group of candidates, which can be determined in a very efficient way. Benefited from this formulation, effective projections can be obtained by using sparse correspondence. Methods for generating these sparse correspondence samples have also been introduced. Our method is evaluated by applying the human's motion captured by a RGB-D sensor to a humanoid in real-time. Continuous motion can be realized and used in the example application of tele-operation. <s> BIB006 </s> Imitation Learning: A Survey of Learning Methods <s> Handling Demonstrations <s> A sequence of 3-D motion images of a human facing an imitator [i.e., a humanoid robot (HR)] is captured using a stereo-vision system installed on the HR. After acquiring enough motion sequences via image ratio and background registration, the estimation of seven feature points (i.e., the head, two arm tips, two leg tips, and two elbows) of the human are obtained. Using five feature points (i.e., the head, two arm tips, and two leg tips), the extraction of key posture frames from the recorded videos is achieved. To ensure a stable imitation motion of the HR, the human motions of the lower body (LB) and the upper body (UB) are imitated separately. All 11 stable motions for the LB are classified by the motion direction, the motion state, and the support phase, whereas only the two arm tips are employed to imitate the motions of the UB. Due to the disadvantages of traditional inverse kinematics (IK), neural-network-based IK is developed to reduce the computation time and improve the accuracy of the time response of the system. Finally, the combined motions of the LB and the UB with suitable interpolation and time intervals are applied to imitate the 3-D motion of a human. The corresponding experimental results confirm the efficacy of the proposed technique. <s> BIB007
Even before feature extraction stages, dealing with demonstrations poses a number of challenges. A major issue is the correspondence problem introduced in Section 1. Creating correspondence mappings between teacher and learner can be computationally intensive, but some methods attempt to create such correspondence in real time. In BIB006 , a projection of the teacher's behavior to the agent's motion space is developed online by sparsely sampling corresponding features. Neural networks can also be utilized to improve the response time of inverse kinematics (IK)-based methods BIB007 and find trajectories appropriate for the learner's motion space based on the desired state of end-effectors. However, IK methods place no further restrictions on the agent's behavior as long as the end effector is in the demonstrated position  . This poses a problem for trajectory imitation applications such as gesture imitation. To alleviate this limitation,  propose a closed-form solution to the correspondence problem based on optimizing external (mapping between observed demonstrations and expected behavior of learner) and learner mapping (mapping between state of the learner and its observed behavior). While most approaches store learned behaviors after mapping to the learner's motion space, in Bandera  , human gestures are captured, identified, and learned in the motion space of the teacher. While that requires learning a model for the teacher's motion, it allows perception to be totally independent of the learner's constraints and facilitates human motion recognition. The learned motions are finally translated to the robot's motion space before execution. A different approach is to address correspondence in reward space where corresponding reward profiles for the demonstrators and the robot are built BIB003 . The difference between them is optimized with respect to the robots' internal constraints to ensure the feasibility of the developed trajectory. This enables learning from demonstrations by multiple human teachers with different physical characteristics. Another challenge concerning demonstrations is incomplete or inaccurate demonstrations. Statistical models can deal with sensor error or inaccurate demonstrations; however, incomplete demonstrations can result in suboptimal behavior [KhansariZadeh and Billard 2011] . BIB001 note that a robot provided only with demonstrations starting from the right of the target will start by moving to the familiar position if initialized in a different position. In BIB004 , a method for learning from limited and inaccurate data is proposed, where the demonstrations are used to constrain a reinforcement learning algorithm that learns from trial and error. Combining learning from demonstrations and experience is extensively investigated in Section 5.1. As an alternative way to cope with the lack of good demonstrations, Grollman and BIB002 learn a policy from failed demonstrations where the agent is trained to avoid repeating unsuccessful behavior. Demonstrations need not be provided by a dedicated teacher but can instead be observed from another agent. In this case, an important question is what to imitate from the continuous behavior being demonstrated. In  , learning from unplanned demonstrations is addressed by segmenting actions from the demonstrations and estimating the significance of the behavior in these segments to the investigated task according to three criteria: (1) change detection is used to discover significant regions of the demonstrations, (2) constrained motif discovery identifies recurring behaviors, and (3) change causality explores the causality between the demonstrated behavior and changes in the environment. Similarly, in BIB005 , acquired recordings of a human hand are segmented into basic behaviors before extracting relevant features and learning behavior generation. The feature extraction and behavior generation are performed with respect to three attributes: (1) preconditions, which are environmental conditions required for the task; (2) internal constraints, which are characteristics of the agent that restrict its behavior; and (3) post results, which represent the consequences of a behavior. Regardless of the source of the signal, captured data may be represented in different ways. We categorize representations as raw features, designed or engineered features, and extracted features. Figure 2 shows the relations between different feature representations.