Imitation Learning: A Survey of Learning Methods <s> Quantitative Evaluation <s> Movement imitation requires a complex set of mechanisms that map an observed movement of a teacher onto one's own movement apparatus. Relevant problems include movement recognition, pose estimation, pose tracking, body correspondence, coordinate transformation from external to egocentric space, matching of observed against previously learned movement, resolution of redundant degrees-of-freedom that are unconstrained by the observation, suitable movement representations for imitation, modularization of motor control, etc. All of these topics by themselves are active research problems in computational and neurobiological sciences, such that their combination into a complete imitation system remains a daunting undertaking-indeed, one could argue that we need to understand the complete perception-action loop. As a strategy to untangle the complexity of imitation, this paper will examine imitation purely from a computational point of view, i.e. we will review statistical and mathematical approaches that have been suggested for tackling parts of the imitation problem, and discuss their merits, disadvantages and underlying principles. Given the focus on action recognition of other contributions in this special issue, this paper will primarily emphasize the motor side of imitation, assuming that a perceptual system has already identified important features of a demonstrated movement and created their corresponding spatial information. Based on the formalization of motor control in terms of control policies and their associated performance criteria, useful taxonomies of imitation learning can be generated that clarify different approaches and future research directions. <s> BIB001 </s> Imitation Learning: A Survey of Learning Methods <s> Quantitative Evaluation <s> We consider learning in a Markov decision process where we are not explicitly given a reward function, but where instead we can observe an expert demonstrating the task that we want to learn to perform. This setting is useful in applications (such as the task of driving) where it may be difficult to write down an explicit reward function specifying exactly how different desiderata should be traded off. We think of the expert as trying to maximize a reward function that is expressible as a linear combination of known features, and give an algorithm for learning the task demonstrated by the expert. Our algorithm is based on using "inverse reinforcement learning" to try to recover the unknown reward function. We show that our algorithm terminates in a small number of iterations, and that even though we may never recover the expert's reward function, the policy output by the algorithm will attain performance close to that of the expert, where here performance is measured with respect to the expert's unknown reward function. <s> BIB002 </s> Imitation Learning: A Survey of Learning Methods <s> Quantitative Evaluation <s> We present a method for transferring knowledge learned in one task to a related task. Our problem solvers employ reinforcement learning to acquire a model for one task. We then transform that learned model into advice for a new task. A human teacher provides a mapping from the old task to the new task to guide this knowledge transfer. Advice is incorporated into our problem solver using a knowledge-based support vector regression method that we previously developed. This advice-taking approach allows the problem solver to refine or even discard the transferred knowledge based on its subsequent experiences. We empirically demonstrate the effectiveness of our approach with two games from the RoboCup soccer simulator: KeepAway and BreakAway. Our results demonstrate that a problem solver learning to play BreakAway using advice extracted from KeepAway outperforms a problem solver learning without the benefit of such advice. <s> BIB003 </s> Imitation Learning: A Survey of Learning Methods <s> Quantitative Evaluation <s> Many motor skills in humanoid robotics can be learned using parametrized motor primitives as done in imitation learning. However, most interesting motor learning problems are high-dimensional reinforcement learning problems often beyond the reach of current methods. In this paper, we extend previous work on policy learning from the immediate reward case to episodic reinforcement learning. We show that this results in a general, common framework also connected to policy gradient methods and yielding a novel algorithm for policy learning that is particularly well-suited for dynamic motor primitives. The resulting algorithm is an EM-inspired algorithm applicable to complex motor learning tasks. We compare this algorithm to several well-known parametrized policy search methods and show that it outperforms them. We apply it in the context of motor learning and show that it can learn a complex Ball-in-a-Cup task using a real Barrett WAM™ robot arm. <s> BIB004
7.1.1. Error Calculation. These are tasks where the score of an executed policy can be directly calculated. For instance, a ball-in-cup task ultimately depends on the ball falling into the cup BIB004 . The loss in the performance is the distance of the ball from the cup. Another example is quantifying progress in a Mario game as the distance traveled from left to right . In cyclic tasks where the agent is required to maintain an action, time can be inherently used as a score. For example, in Keepaway BIB003 ], a robocup soccer simulator task, one team is required to keep the ball away from the other team. The game ends if the ball is intercepted or goes out of bounds. Therefore, the duration that the ball is kept until the game ends reflects the performance of the keeping team. Another measure for evaluating cyclic actions is the amount of training needed to obtain a stable performance BIB004 . The error could also be calculated as deviation from the teacher's actions. This is possible if the motor primitives are the same for the teacher and test samples can be reproduced BIB001 ]. However, these methods need to account for the fact that small errors could substantially change the action trajectory. 7.1.2. Quantitative Scores. In many applications, the error cannot be explicitly calculated. However, some criteria can be used to quantify the quality of an executed policy. The performance is usually quantified by counting the number of desired (or unwanted) statuses achieved during the execution of a policy over a sequence. For example, in a driving simulator, an undesired action is hitting an object. Thus, the number of collisions could be used as a criterion for quantifying a driving task BIB002 .