Imitation Learning: A Survey of Learning Methods <s> INTRODUCTION <s> By now it is widely accepted that learning a task from scratch, i.e., without any prior knowledge, is a daunting undertaking. Humans, however, rarely attempt to learn from scratch. They extract initial biases as well as strategies how to approach a learning problem from instructions and/or demonstrations of other humans. For teaming control, this paper investigates how learning from demonstration can be applied in the context of reinforcement learning. We consider priming the Q-function, the value function, the policy, and the model of the task dynamics as possible areas where demonstrations can speed up learning. In general nonlinear learning problems, only model-based reinforcement learning shows significant speed-up after a demonstration, while in the special case of linear quadratic regulator (LQR) problems, all methods profit from the demonstration. In an implementation of pole balancing on a complex anthropomorphic robot arm, we demonstrate that, when facing the complexities of real signal processing, model-based reinforcement learning offers the most robustness for LQR problems. Using the suggested methods, the robot learns pole balancing in just a single trial after a 30 second long demonstration of the human instructor. <s> BIB001 </s> Imitation Learning: A Survey of Learning Methods <s> INTRODUCTION <s> A belt grinding machine which is equipped with an elastically flexible load beam extending over the entire operative length of an endless grinding belt for applying the necessary grinding pressure to the belt. The effective pressure-applying surface of said beam is suspended for elevational control from support elements distributed at intervals along the length of the beam and attached to an elevationally movable carrier preferably in the form of transverse beams mounted on push rods operable by a pressure medium. The push rods are associated with constant delay mechanism adjustable to the difference in time between the instant of response of a sensing element to the passage of the leading end of a workpiece and the instant the grinding action begins, on the one hand, and between the instant of response of a sensing element to the passage of the trailing end of the workpiece and the instant the grinding action terminates, on the other hand. <s> BIB002 </s> Imitation Learning: A Survey of Learning Methods <s> INTRODUCTION <s> Summary This review investigates two recent developments in artificial intelligence and neural computation: learning from imitation and the development of humanoid robots. It will be postulated that the study of imitation learning offers a promising route to gain new insights into mechanisms of perceptual motor control that could ultimately lead to the creation of autonomous humanoid robots. Imitation learning focuses on three important issues: efficient motor learning, the connection between action and perception, and modular motor control in form of movement primitives. It will be reviewed how research on representations of, and functional connections between action and perception have contributed to our understanding of motor acts of other beings. The recent discovery that some areas in the primate brain are active during both movement perception and execution has provided a hypothetical neural basis of imitation. Computational approaches to imitation learning will also be described, initially from the perspective of traditional AI and robotics, but also from the perspective of neural network models and statistical learning research. Parallels and differences between biological and computational approaches to imitation will be highlighted and an overview of current projects that actually employ imitation learning for humanoid robots will be given. <s> BIB003 </s> Imitation Learning: A Survey of Learning Methods <s> INTRODUCTION <s> A behavior-based approach to structuring and controlling complex robotic systems is described that uses imitation for interaction and learning. The use of basis behaviors, or primitives, lets humanoid robots dance the Macarena, do aerobics and throw objects. <s> BIB004 </s> Imitation Learning: A Survey of Learning Methods <s> INTRODUCTION <s> Movement imitation requires a complex set of mechanisms that map an observed movement of a teacher onto one's own movement apparatus. Relevant problems include movement recognition, pose estimation, pose tracking, body correspondence, coordinate transformation from external to egocentric space, matching of observed against previously learned movement, resolution of redundant degrees-of-freedom that are unconstrained by the observation, suitable movement representations for imitation, modularization of motor control, etc. All of these topics by themselves are active research problems in computational and neurobiological sciences, such that their combination into a complete imitation system remains a daunting undertaking-indeed, one could argue that we need to understand the complete perception-action loop. As a strategy to untangle the complexity of imitation, this paper will examine imitation purely from a computational point of view, i.e. we will review statistical and mathematical approaches that have been suggested for tackling parts of the imitation problem, and discuss their merits, disadvantages and underlying principles. Given the focus on action recognition of other contributions in this special issue, this paper will primarily emphasize the motor side of imitation, assuming that a perceptual system has already identified important features of a demonstrated movement and created their corresponding spatial information. Based on the formalization of motor control in terms of control policies and their associated performance criteria, useful taxonomies of imitation learning can be generated that clarify different approaches and future research directions. <s> BIB005 </s> Imitation Learning: A Survey of Learning Methods <s> INTRODUCTION <s> Helicopters have highly stochastic, nonlinear, dynamics, and autonomous helicopter flight is widely regarded to be a challenging control problem. As helicopters are highly unstable at low speeds, it is particularly difficult to design controllers for low speed aerobatic maneuvers. In this paper, we describe a successful application of reinforcement learning to designing a controller for sustained inverted flight on an autonomous helicopter. Using data collected from the helicopter in flight, we began by learning a stochastic, nonlinear model of the helicopter’s dynamics. Then, a reinforcement learning algorithm was applied to automatically learn a controller for autonomous inverted hovering. Finally, the resulting controller was successfully tested on our autonomous helicopter platform. <s> BIB006 </s> Imitation Learning: A Survey of Learning Methods <s> INTRODUCTION <s> Imitation is a powerful mechanism the human brain applies to extend its repertoire of solutions and behaviors suitable to solve problems of various kinds. From an abstract point of view, the major advantage of this strategy is that it reduces the search space of apropriate solutions. In this contribution, we discuss if and how the principle of imitation learning can facilitate the programing of life-like computer game charecters. We present different algorithms that learn from human generated training data and we show that machine learning can be applied on different levels of cognitive abstraction. <s> BIB007 </s> Imitation Learning: A Survey of Learning Methods <s> INTRODUCTION <s> In this paper, we deal with imitation learning of arm movements in humanoid robots. Hidden Markov Models (HMM) are used to generalize movements demonstrated to a robot multiple times. They are trained with the characteristic features (key points) of each demonstration. Using the same HMM, key points that are common to all demonstrations are identified; only those are considered when reproducing a movement. We also show how HMM can be used to detect temporal dependencies between both arms in dual-arm tasks. We created a model of the human upper body to simulate the reproduction of dual-arm movements and generate natural-looking joint configurations from tracked hand paths. Results are presented and discussed. <s> BIB008 </s> Imitation Learning: A Survey of Learning Methods <s> INTRODUCTION <s> Autonomous helicopter flight is widely regarded to be a highly challenging control problem. This paper presents the first successful autonomous completion on a real RC helicopter of the following four aerobatic maneuvers: forward flip and sideways roll at low speed, tail-in funnel, and nose-in funnel. Our experimental results significantly extend the state of the art in autonomous helicopter flight. We used the following approach: First we had a pilot fly the helicopter to help us find a helicopter dynamics model and a reward (cost) function. Then we used a reinforcement learning (optimal control) algorithm to find a controller that is optimized for the resulting model and reward function. More specifically, we used differential dynamic programming (DDP), an extension of the linear quadratic regulator (LQR). <s> BIB009 </s> Imitation Learning: A Survey of Learning Methods <s> INTRODUCTION <s> The Maximum Margin Planning (MMP) (Ratliff et al., 2006) algorithm solves imitation learning problems by learning linear mappings from features to cost functions in a planning domain. The learned policy is the result of minimum-cost planning using these cost functions. These mappings are chosen so that example policies (or trajectories) given by a teacher appear to be lower cost (with a lossscaled margin) than any other policy for a given planning domain. We provide a novel approach, MMPBOOST , based on the functional gradient descent view of boosting (Mason et al., 1999; Friedman, 1999a) that extends MMP by “boosting” in new features. This approach uses simple binary classification or regression to improve performance of MMP imitation learning, and naturally extends to the class of structured maximum margin prediction problems. (Taskar et al., 2005) Our technique is applied to navigation and planning problems for outdoor mobile robots and robotic legged locomotion. <s> BIB010 </s> Imitation Learning: A Survey of Learning Methods <s> INTRODUCTION <s> We contribute an approach for interactive policy learning through expert demonstration that allows an agent to actively request and effectively represent demonstration examples. In order to address the inherent uncertainty of human demonstration, we represent the policy as a set of Gaussian mixture models (GMMs), where each model, with multiple Gaussian components, corresponds to a single action. Incrementally received demonstration examples are used as training data for the GMM set. We then introduce our confident execution approach, which focuses learning on relevant parts of the domain by enabling the agent to identify the need for and request demonstrations for specific parts of the state space. The agent selects between demonstration and autonomous execution based on statistical analysis of the uncertainty of the learned Gaussian mixture set. As it achieves proficiency at its task and gains confidence in its actions, the agent operates with increasing autonomy, eliminating the need for unnecessary demonstrations of already acquired behavior, and reducing both the training time and the demonstration workload of the expert. We validate our approach with experiments in simulated and real robot domains. <s> BIB011 </s> Imitation Learning: A Survey of Learning Methods <s> INTRODUCTION <s> Driving in unknown natural outdoor terrain is a challenge for autonomous ground vehicles. It can be difficult for a robot to discern obstacles and other hazards in its environment, and characteristics of this high cost terrain may change from one environment to another, or even with different lighting conditions. One successful approach to this problem is for a robot to learn from a demonstration by a human operator. In this paper, we describe an approach to calculating terrain costs from Bayesian estimates using feature vectors measured during a short teleoperated training run in similar terrain and conditions. We describe the theory, its implementation on two different robotic systems, and results of several independently conducted field tests. <s> BIB012 </s> Imitation Learning: A Survey of Learning Methods <s> INTRODUCTION <s> We present an approach to teach incrementally human gestures to a humanoid robot. By using active teaching methods that puts the human teacher "in the loop" of the robot's learning, we show that the essential characteristics of a gesture can be efficiently transferred by interacting socially with the robot. In a first phase, the robot observes the user demonstrating the skill while wearing motion sensors. The motion of his/her two arms and head are recorded by the robot, projected in a latent space of motion and encoded bprobabilistically in a Gaussian Mixture Model (GMM). In a second phase, the user helps the robot refine its gesture by kinesthetic teaching, i.e. by grabbing and moving its arms throughout the movement to provide the appropriate scaffolds. To update the model of the gesture, we compare the performance of two incremental training procedures against a batch training procedure. We present experiments to show that different modalities can be combined efficiently to teach incrementally basketball officials' signals to a HOAP-3 humanoid robot. <s> BIB013 </s> Imitation Learning: A Survey of Learning Methods <s> INTRODUCTION <s> Learning by demonstration can be a powerful and natural tool for developing robot control policies. That is, instead of tedious hand-coding, a robot may learn a control policy by interacting with a teacher. In this work we present an algorithm for learning by demonstration in which the teacher operates in two phases. The teacher first demonstrates the task to the learner. The teacher next critiques learner performance of the task. This critique is used by the learner to update its control policy. In our implementation we utilize a 1-Nearest Neighbor technique which incorporates both training dataset and teacher critique. Since the teacher critiques performance only, they do not need to guess at an effective critique for the underlying algorithm. We argue that this method is particularly well-suited to human teachers, who are generally better at assigning credit to performances than to algorithms. We have applied this algorithm to the simulated task of a robot intercepting a ball. Our results demonstrate improved performance with teacher critiquing, where performance is measured by both execution success and efficiency. <s> BIB014 </s> Imitation Learning: A Survey of Learning Methods <s> INTRODUCTION <s> We present a comprehensive survey of robot Learning from Demonstration (LfD), a technique that develops policies from example state to action mappings. We introduce the LfD design choices in terms of demonstrator, problem space, policy derivation and performance, and contribute the foundations for a structure in which to categorize LfD research. Specifically, we analyze and categorize the multiple ways in which examples are gathered, ranging from teleoperation to imitation, as well as the various techniques for policy derivation, including matching functions, dynamics models and plans. To conclude we discuss LfD limitations and related promising areas for future research. <s> BIB015 </s> Imitation Learning: A Survey of Learning Methods <s> INTRODUCTION <s> A faucet includes an inner tube inserted in an upper opening formed in between a horizontal spout portion of the faucet and a downcomer spout portion having a lower tube hole of the inner tube poked into the downcomer spout portion and positioned under an overflow level of the horizontal spout portion, whereby upon a shielding or closing of a discharge outlet formed in a lowermost end of the downcomer spout portion, a water stream, as supplied by an opened faucet, will be ejected upwardly through the inner tube for rinsing or drinking purpose. <s> BIB016 </s> Imitation Learning: A Survey of Learning Methods <s> INTRODUCTION <s> This paper presents a physics based simulator that allows kinesthetic interactions between a human and a robot to be recorded, and later used for imitation learning. We argue that kinesthetic interac- tion can be a very important tool for programming robots, if properly supported by a simulation engine. For this, we propose a new scheme for robot motion learning based on kinesthetic bootstrapping. Interactions with the robot are used to create a low-dimensional posture space. The posture space together with the presented simulation engine allow for fast imitation learning of behaviors. Early results of this approach, using Genetic Algorithms as learning technique, will be presented. <s> BIB017 </s> Imitation Learning: A Survey of Learning Methods <s> INTRODUCTION <s> Despite a history of games-based research, academia has generally regarded ::: commercial games as a distraction from the serious business of AI, rather than as an ::: opportunity to leverage this existing domain to the advancement of our knowledge. ::: Similarly, the computer game industry still relies on techniques that were developed ::: several decades ago, and has shown little interest in adopting more progressive ::: academic approaches. In recent times, however, these attitudes have begun to change; ::: under- and post-graduate games development courses are increasingly common, ::: while the industry itself is slowly but surely beginning to recognise the potential ::: offered by modern machine-learning approaches, though games which actually ::: implement said approaches on more than a token scale remain scarce. ::: One area which has not yet received much attention from either academia or industry ::: is imitation learning, which seeks to expedite the learning process by exploiting data ::: harvested from demonstrations of a given task. While substantial work has been done ::: in developing imitation techniques for humanoid robot movement, there has been ::: very little exploration of the challenges posed by interactive computer games. Given ::: that such games generally encode reasoning and decision-making behaviours which ::: are inherently more complex and potentially more interesting than limb motion data, ::: that they often provide inbuilt facilities for recording human play, that the generation ::: and collection of training samples is therefore far easier than in robotics, and that ::: many games have vast pre-existing libraries of these recorded demonstrations, it is ::: fair to say that computer games represent an extremely fertile domain for imitation ::: learning research. ::: In this thesis, we argue in favour of using modern, commercial computer games to ::: study, model and reproduce humanlike behaviour. We provide an overview of the ::: biological and robotic imitation literature as well as the current status of game AI, highlighting techniques which may be adapted for the purposes of game-based ::: imitation. We then proceed to describe our contributions to the field of imitation ::: learning itself, which encompass three distinct categories: theory, implementation ::: and evaluation. ::: We first describe the development of a fully-featured Java API - the Quake2 Agent ::: Simulation Environment (QASE) - designed to facilitate both research and education ::: in imitation and general machine-learning, using the game Quake 2 as a testbed. We ::: outline our motivation for developing QASE, discussing the shortcomings of existing ::: APIs and the steps which we have taken to circumvent them. We describe QASE’s ::: network layer, which acts as an interface between the local AI routines and the ::: Quake 2 server on which the game environment is maintained, before detailing the ::: API’s agent architecture, which includes an interface to the MatLab programming ::: environment and the ability to parse and analyse full recordings of game sessions. ::: We conclude the chapter with a discussion of QASE’s adoption by numerous ::: universities as both an undergraduate teaching tool and research platform. ::: We then proceed to describe the various imitative mechanisms which we have ::: developed using QASE and its MatLab integration facilities. We first outline a ::: behaviour model based on a well-known psychological model of human planning. ::: Drawing upon previous research, we also identify a set of believability criteria - ::: elements of agent behaviour which are of particular importance in determining the ::: “humanness” of its in-game appearance. We then detail a reinforcement-learning ::: approach to imitating the human player’s navigation of his environment, centred ::: upon his pursuit of items as strategic goals. In the subsequent section, we describe ::: the integration of this strategic system with a Bayesian mechanism for the imitation ::: of tactical and motion-modelling behaviours. Finally, we outline a model for the ::: imitation of reactive combat behaviours; specifically, weapon-selection and aiming. Experiments are presented in each case to demonstrate the imitative mechanisms’ ::: ability to accurately reproduce observed behaviours. ::: Finally, we criticise the lack of any existing methodology to formally gauge the ::: believability of game agents, and observe that the few previous attempts have been ::: extremely ad-hoc and informal. We therefore propose a generalised approach to such ::: testing; the Bot-Oriented Turing Test (BOTT). This takes the form of an anonymous ::: online questionnaire, an accompanying protocol to which examiners should adhere, ::: and the formulation of a believability index which numerically expresses each agent’s ::: humanness as indicated by its observers, weighted by their experience and the ::: accuracy with which the agents were identified. To both validate the survey approach ::: and to determine the efficacy of our imitative models, we present a series of ::: experiments which use the believability test to evaluate our own imitation agents ::: against both human players and traditional artificial bots. We demonstrate that our ::: imitation agents perform substantially better than even a highly-regarded rule-based ::: agent, and indeed approach the believability of actual human players. ::: Some suggestions for future directions in our research, as well as a broader ::: discussion of open questions, conclude this thesis. <s> BIB018 </s> Imitation Learning: A Survey of Learning Methods <s> INTRODUCTION <s> Developing coordination among multiple agents and enabling them to exhibit teamwork is a challenging yet exciting task that can benefit many of the complex real-life problems. This research uses imitation to learn collaborative strategies for a team of agents. Imitation based learning involves learning from an expert by observing him/her demonstrating a task and then replicating it. The key idea is to involve multiple human experts during demonstration to teach autonomous agents how to work in coordination. The effectiveness of the proposed methodology has been assessed in a goal defending scenario of the RoboCup Soccer Simulation 3D league. The process involves multiple human demonstrators controlling soccer agents via game controllers and demonstrating them how to play soccer in coordination. The data gathered during this phase is used as training data to learn a classification model which is later used by the soccer agents to make autonomous decisions during actual matches. Different performance evaluation metrics are derived to compare the performance of imitating agent with that of the human-driven agent and hand-coded (if-then-else rules) agent. <s> BIB019 </s> Imitation Learning: A Survey of Learning Methods <s> INTRODUCTION <s> Close physical interaction between robots and humans is a particularly challenging aspect of robot development. For successful interaction and cooperation, the robot must have the ability to adapt its behavior to the human counterpart. Based on our earlier work, we present and evaluate a computationally efficient machine learning algorithm that is well suited for such close-contact interaction scenarios. We show that this algorithm helps to improve the quality of the interaction between a robot and a human caregiver. To this end, we present two human-in-the-loop learning scenarios that are inspired by human parenting behavior, namely, an assisted standing-up task and an assisted walking task. <s> BIB020 </s> Imitation Learning: A Survey of Learning Methods <s> INTRODUCTION <s> Nonlinear dynamical systems have been used in many disciplines to model complex behaviors, including biological motor control, robotics, perception, economics, traffic prediction, and neuroscience. While often the unexpected emergent behavior of nonlinear systems is the focus of investigations, it is of equal importance to create goal-directed behavior e.g., stable locomotion from a system of coupled oscillators under perceptual guidance. Modeling goal-directed behavior with nonlinear systems is, however, rather difficult due to the parameter sensitivity of these systems, their complex phase transitions in response to subtle parameter changes, and the difficulty of analyzing and predicting their long-term behavior; intuition and time-consuming parameter tuning play a major role. This letter presents and reviews dynamical movement primitives, a line of research for modeling attractor behaviors of autonomous nonlinear dynamical systems with the help of statistical learning techniques. The essence of our approach is to start with a simple dynamical system, such as a set of linear differential equations, and transform those into a weakly nonlinear system with prescribed attractor dynamics by means of a learnable autonomous forcing term. Both point attractors and limit cycle attractors of almost arbitrary complexity can be generated. We explain the design principle of our approach and evaluate its properties in several example applications in motor control and robotics. <s> BIB021 </s> Imitation Learning: A Survey of Learning Methods <s> INTRODUCTION <s> Learning new motor tasks autonomously from interaction with a human being is an important goal for both robotics and machine learning. However, when moving beyond basic skills, most monolithic machine learning approaches fail to scale. In this paper, we take the task of learning table tennis as an example and present a new framework which allows a robot to learn cooperative table tennis from interaction with a human. Therefore, the robot first learns a set of elementary table tennis hitting movements from a human teacher by kinesthetic teach-in, which is compiled into a set of dynamical system motor primitives (DMPs). Subsequently, the system generalizes these movements to a wider range of situations using our mixture of motor primitives (MoMP) approach. The resulting policy enables the robot to select appropriate motor primitives as well as to generalize between them. Finally, the robot plays with a human table tennis partner and learns online to improve its behavior. <s> BIB022
In recent years, the demand for intelligent agents capable of mimicking human behavior has grown substantially. Advancements in robotics and communication technology have given rise to many potential applications that need artificial intelligence that can not only make intelligent decisions but also is able to perform motor actions realistically in a variety of situations. Many future directions in technology rely on the ability of artificial intelligence agents to behave as a human would when presented with the same situation. Examples of such fields are self-driving vehicles, assistive robots, and human-computer interaction. For the latter especially, opportunities for new applications are growing due to recent interest in consumer virtual reality and motion capture systems. 1 In these applications and many robotics tasks, we are faced with the problem of executing an action given the current state of the agent and its surroundings. The number of possible scenarios in a complex application is too large to cover by explicit programming, and so a successful agent must be able to handle unseen scenarios. While such a task may be formulated as an optimization problem, it has become widely accepted that having prior knowledge provided by an expert is more effective and efficient than searching for a solution from scratch BIB003 BIB001 BIB002 . In addition, optimization through trial and error requires reward functions that are designed specifically for each task. One can imagine that even for simple tasks, the number of possible sequences of actions an agent can take grows exponentially. Defining rewards for such problems is difficult, and in many cases unknown. One of the more natural and intuitive ways of imparting knowledge by an expert is to provide demonstrations for the desired behavior that the learner is required to emulate. It is much easier for the human teacher to transfer his or her knowledge through demonstration than to articulate it in a way that the learner will understand BIB019 . This article reviews the methods used to teach artificial agents to perform complex sequences of actions through imitation. Imitation learning is an interdisciplinary field of research. Existing surveys focus on different challenges and perspectives of tackling this problem. Early surveys review the history of imitation learning and early attempts to learn from demonstration BIB003 BIB005 ]. In , learning approaches are categorized as engineering-oriented and biologically oriented methods. BIB021 focus on learning methods from the viewpoint of dynamical systems, while BIB015 address different challenges in the process of imitation such as acquiring demonstrations, physical and sensory issues, and learning techniques. However, due to recent advancements in the field and a surge in potential applications, it is important at this time to conduct a survey that focuses on the computational methods used to learn from demonstrated behavior. More specifically, we review artificial intelligence methods that are used to learn policies that solve problems according to human demonstrations. By focusing on learning methods, this survey addresses learning for any intelligent agent, whether it manifests itself as a physical robot or a software agent (such as games, simulations, planning, etc.). The reviewed literature addresses various applications; however, many of the methods used are generic and can be applied to general motion learning tasks. The learning process is categorized into creating feature representations, direct imitation, and indirect learning. The methods and sources of learning for each process are reviewed as well as evaluation metrics and applications suitable for these methods. Imitation learning refers to an agent's acquisition of skills or behaviors by observing a teacher demonstrating a given task. With inspiration and basis stemmed in neuroscience, imitation learning is an important part of machine intelligence and humancomputer interaction, and has from an early point been viewed as an integral part in the future of robotics BIB003 . Another popular paradigm is learning through trial and error; however, providing good examples to learn from expedites the process of finding a suitable action model and prevents the agent from falling into local minima. Moreover, a learner could very well arrive on its own at a suitable solution, that is, one that achieves a certain quantifiable goal but which differs significantly from the way a human would approach the task. It is sometimes important for the learner's actions to be believable and appear natural. This is necessary in many robotic domains as well as human-computer interaction where the performance of the learner is only as good as a human observer's perception of it. It is therefore favorable to teach a learner the desired behavior from a set of collected instances. However, it is often the case that direct imitation of the expert's motion doesn't suffice due to variations in the task such as the positions of objects or inadequate demonstrations. Therefore, imitation learning techniques need to be able to learn a policy from the given demonstrations that can generalize to unseen scenarios. As such, the agent learns to perform the task rather than deterministically copying the teacher. The field of imitation learning draws its importance from its relevance to a variety of applications such as human-computer interaction and assistive robots. It is being used to teach robots of varying skeletons and degrees of freedom (DOF) to perform an array of different tasks. Some examples are navigational problems, which typically employ vehicle-like robots, with relatively lower degrees of freedom. These include flying vehicles  BIB009 BIB006 ] and ground vehicles BIB016 BIB010 BIB011 BIB012 ]. Other applications focus on robots with higher degrees of freedom such as humanoid robots BIB004 BIB008 BIB013 or robotic arms Peters 2009b, 2010; BIB022 . High DOF humanoid robots can learn discrete actions such as standing up and cyclic tasks such as walking BIB017 . Although the majority of applications target robotics, imitation learning applies to simulations BIB017 BIB014 and is even used in computer games BIB007 BIB018 . Imitation learning works by extracting information about the behavior of the teacher and the surrounding environment including any manipulated objects, and learning a mapping between the situation and demonstrated behavior. Traditional machinelearning algorithms do not scale to high-dimensional agents with high degrees of freedom  . Specialized algorithms are therefore needed to create adequate representations and predictions to be able to emulate motor functions in humans. Similar to traditional supervised learning where examples represent pairs of features and labels, in imitation learning the examples demonstrate pairs of states and actions, where the state represents the current pose of the agent, including the position and velocities of relevant joints and the status of a target object if one exists (such as position, velocity, geometric information, etc.). Therefore, Markov decision processes (MDPs) lend themselves naturally to imitation learning problems and are commonly used to represent expert demonstrations. The Markov property dictates that the next state is only dependent on the previous state and action, which alleviates the need to include earlier states in the state representation . A typical imitation learning workflow starts by acquiring sample demonstrations from an expert, which are then encoded as state-action pairs. These examples are then used to train a policy. However, learning a direct mapping between state and action is often not enough to achieve the required behavior. This can happen due to a number of issues such as errors in acquiring the demonstrations, variance in the skeletons of the teacher and learner (correspondence problem), or insufficient demonstrations. Moreover, the task performed by the learner may slightly vary from the demonstrated task due to changes in the environment, obstacles, or targets. Therefore, imitation learning frequently involves another step that requires the learner to perform the learned action and reoptimize the learned policy according to its performance of the task. This selfimprovement can be achieved with respect to a quantifiable reward or learned from examples. Many of these approaches fall under the wide umbrella of reinforcement learning. Figure 1 shows a workflow of an imitation learning process. The process starts by capturing actions to learn from, which can be achieved via different sensing methods. The data from the sensors is then processed to extract features that describe the state and surroundings of the performer. The features are used to learn a policy to mimic the demonstrated behavior. Finally, the policy can be enhanced by allowing the agent to act out the policy and refine it based on its performance. This step may or may not require the input of a teacher. It might be intuitive to think of policy refinement as a postlearning step, but in many cases it occurs in conjunction with learning from demonstrations. Imitation learning applications face a number of diverse challenges due to their interdisciplinary nature: -Starting with the process of acquiring demonstrations, whether capturing data from sensors on the learner or the teacher or using visual information, the captured signals are prone to noise and sensor errors. Similar problems arise during execution, when the agent is sensing the environment. Noisy or unreliable sensing will result in erroneous behavior even if the model is adequately trained. BIB015 survey the different methods of gathering demonstrations and the challenges in each approach. -Another issue concerning demonstration is the correspondence problem  . Correspondence is the matching of the learner's capabilities, skeleton, and degrees of freedom to that of the teacher. Any discrepancies in the size or structure between the teacher and learner need to be compensated for during training. Often in this case, a learner can learn the shape of the movement from demonstrations, then refine the model through trial and error to achieve its goal. -A related challenge is the problem of observability where the kinematics of the teacher are not known to the learner BIB005 . If the demonstrations are not provided by a designated teacher, the learner may not be aware of the capabilities and possible actions of the teacher; it can only observe the effects of the teacher's actions and attempt to replicate them using its own kinematics. -The learning process also faces practical problems as traditional machine-learning techniques do not scale well to high degrees of freedom  . Due to the real-time nature of many imitation learning applications, the learning algorithms are restricted by computing power and memory limitations, especially in robotic applications that require on-board computers to perform the real-time processing. -Moreover, complex behaviors can often be viewed as a trajectory of dependent micro actions that violates the independent and identically distributed (i.i.d.) assumption adopted in most machine-learning practices. The learned policy must be able to adapt its behavior based on previous actions and make corrections if necessary. -The policy must also be able to adapt to variations in the task and the surrounding environment. The complex nature of imitation learning applications dictates that agents must be able to reliably perform the task even under circumstances that vary from the training demonstration. -Tasks that entail human-robot interaction pose a new set of challenges. Naturally safety is a chief concern in such applications  BIB020 , and measures need to be taken to prevent injury of the human partners and ensure their safety. Moreover, other challenges concern the mechanics of the robot, such as its ability to react to the humans' force and adapt to their actions. In the next section, we formally present the problem of imitation learning and discuss different ways to formulate the problem. Section 3 describes the different methods for creating feature representations. Section 4 reviews direct imitation methods for learning from demonstrations. Section 5 surveys indirect learning techniques and presents the different approaches to improve learned policies through optimizing reward functions and teachers' behaviors. The paradigms for improving direct imitation through indirect learning are also discussed. Section 6 reviews the use of imitation learning in multiagent scenarios. Section 7 describes different evaluation approaches, and Section 8 shows the potential applications to utilize imitation learning. Finally, we present a conclusion and discuss future directions in Section 9.