Imitation Learning: A Survey of Learning Methods <s> PROBLEM FORMULATION <s> Many control problems take place in continuous state-action spaces, e.g., as in manipulator robotics, where the control objective is often defined as finding a desired trajectory that reaches a particular goal state. While reinforcement learning offers a theoretical framework to learn such control policies from scratch, its applicability to higher dimensional continuous state-action spaces remains rather limited to date. Instead of learning from scratch, in this paper we suggest to learn a desired complex control policy by transforming an existing simple canonical control policy. For this purpose, we represent canonical policies in terms of differential equations with well-defined attractor properties. By nonlinearly transforming the canonical attractor dynamics using techniques from nonparametric regression, almost arbitrary new nonlinear policies can be generated without losing the stability properties of the canonical system. We demonstrate our techniques in the context of learning a set of movement skills for a humanoid robot from demonstrations of a human teacher. Policies are acquired rapidly, and, due to the properties of well formulated differential equations, can be re-used and modified on-line under dynamic changes of the environment. The linear parameterization of nonparametric regression moreover lends itself to recognize and classify previously learned movement skills. Evaluations in simulations and on an actual 30 degree-of-freedom humanoid robot exemplify the feasibility and robustness of our approach. <s> BIB001 </s> Imitation Learning: A Survey of Learning Methods <s> PROBLEM FORMULATION <s> Movement imitation requires a complex set of mechanisms that map an observed movement of a teacher onto one's own movement apparatus. Relevant problems include movement recognition, pose estimation, pose tracking, body correspondence, coordinate transformation from external to egocentric space, matching of observed against previously learned movement, resolution of redundant degrees-of-freedom that are unconstrained by the observation, suitable movement representations for imitation, modularization of motor control, etc. All of these topics by themselves are active research problems in computational and neurobiological sciences, such that their combination into a complete imitation system remains a daunting undertaking-indeed, one could argue that we need to understand the complete perception-action loop. As a strategy to untangle the complexity of imitation, this paper will examine imitation purely from a computational point of view, i.e. we will review statistical and mathematical approaches that have been suggested for tackling parts of the imitation problem, and discuss their merits, disadvantages and underlying principles. Given the focus on action recognition of other contributions in this special issue, this paper will primarily emphasize the motor side of imitation, assuming that a perceptual system has already identified important features of a demonstrated movement and created their corresponding spatial information. Based on the formalization of motor control in terms of control policies and their associated performance criteria, useful taxonomies of imitation learning can be generated that clarify different approaches and future research directions. <s> BIB002 </s> Imitation Learning: A Survey of Learning Methods <s> PROBLEM FORMULATION <s> We present a comprehensive survey of robot Learning from Demonstration (LfD), a technique that develops policies from example state to action mappings. We introduce the LfD design choices in terms of demonstrator, problem space, policy derivation and performance, and contribute the foundations for a structure in which to categorize LfD research. Specifically, we analyze and categorize the multiple ways in which examples are gathered, ranging from teleoperation to imitation, as well as the various techniques for policy derivation, including matching functions, dynamics models and plans. To conclude we discuss LfD limitations and related promising areas for future research. <s> BIB003
In this section, we formalize the problem of imitation learning and introduce some preliminaries and definitions. Definition 1. The process of imitation learning is one by which an agent uses instances of performed actions to learn a policy that solves a given task. Definition 2. An agent is defined as an entity that autonomously interacts within an environment toward achieving or optimizing a goal  ]. An agent can be thought of as a software robot; it receives information from the environment by sensing or communication and acts upon the environment using a set of actuators. Definition 3. A policy is a function that maps a state (a description of the agent, such as pose, positions, and velocities of various parts of the skeleton, and its relevant surrounding) to an action. It is what the agent uses to decide which action to execute when presented with a situation. Policies can be learned from demonstration or experience. The demonstrations may come from a designated teacher or another agent; the experience may be the agent's own or another's. The difference between the two types of training instances is that demonstrations provide the optimal action to a given state, and so the agent learns to reproduce this behavior in similar situations. This makes demonstrations suited for direct policy learning such as supervised learning methods. On the other hand, experience shows the performed action, which may not be optimal, but also provides the reward (or cost) of performing that action given the current state, the agent learns to act in a manner that maximizes its overall reward. Therefore, reinforcement learning is mainly used to learn from experience. More formally, demonstrations and experiences can be defined as follows. Definition 4. A demonstration is presented as a pair of input and output (x, y) , where x is a vector of features describing the state at that instant and y is the action performed by the demonstrator. Definition 5. An experience is presented as a tuple (s, a, r, s ), where s is the state, a is the action taken at state s, r is the reward received for performing action a, and s is the new state resulting from that action. It is clear from this formulation that learning from demonstration doesn't require the learner to know the cost function optimized by the teacher. It can simply optimize the error of deviating from the demonstrated output such as the least square error in supervised learning. More formally, from a set of demonstrations D = (x i , y i ), an agent learns a policy π such that where u is the predicted action, x is the feature vector, t is the time, and α is the set of policy parameters that are changed through learning. While the time parameter t is used to specify an instance of input and output, it is also input to the policy π as a separate parameter. Definition 6. A policy that uses t in learning the parameters of the policy is called a nonstationary policy (also known as nonautonomous BIB002 ; that is, the policy takes into consideration at what stage of the task the agent is currently acting. Definition 7. A stationary policy (autonomous) neglects the time parameter and learns one policy for all steps in an action sequence. One advantage of stationary policies is the ability to learn tasks where the horizon (the time limit for actions) is large or unknown . Nonstationary policies are more naturally situated to learn motor trajectories, that is, actions that occur over a period of time, and are composed of multiple motor primitive executed sequentially. However, these policies are difficult to adapt to unseen scenarios and changes in the parameters of the task BIB002 ]. Moreover, this failure to adapt to new scenarios, at one point in the trajectory, can result in compounded errors as the agent continues to perform the remainder of the action. In light of these drawbacks, methods for learning trajectories using stationary policies are motivated. An example is the use of structured predictions where the training demonstrations are aggregated with labeled instances at different time steps in the trajectory-so time is encoded in the state. Alternatively, BIB001 learn attractor landscapes from the demonstrations, creating a stationary policy that is attracted to the demonstrated trajectory. This avoids compounded errors as the current state is considered by the policy before executing each state of the trajectory. Learning from experience is commonly formulated as an MDP. MDPs lend themselves naturally to motor actions, as they represent a state-action network and are therefore suitable for reinforcement learning. In addition, the Markov property dictates that the next state is only dependent on the previous state and action, regardless of earlier states. This timeless property promotes stationary policies. There are different methods to learn from experience through reinforcement learning that are out of the scope of this article. For a survey and formal definitions of reinforcement learning methods for intelligent agents, the reader is referred to . Note that both learning paradigms are similarly formulated with the exception of the cost function; the feature vector x(t) corresponds to s, u(t) corresponds to a, and x(t + 1) corresponds to the resulting state s . It is therefore not uncommon (especially in more recent research) to combine learning from demonstrations and experience to perform a task. We now consider the predicted action u(t) in Equation (1). Definition 8. An action u(t) can often represent a vector rather than a single value. This means that the action is composed of more than one decision executed simultaneously, such as pressing multiple buttons on a controller or moving multiple joints in a robot. Actions can also represent different levels of motor control: low-level actions, motor primitives, and high-level macro actions BIB003 ]. Definition 9. Low-level actions are those that execute simple commands such as move forward and turn in navigation tasks, or jump and shoot in games. These low-level actions can be directly predicted using a supervised classifier. Lowlevel actions also extend to regression when the predicted actions have continuous values rather than a discrete set of actions (see learning motion). Definition 10. Motor primitives are simple building blocks that are executed in sequence to perform complex motions. An action is broken down into basic unit actions (often concerning one degree of freedom or actuator) that can be used to make up any action that needs to be performed in the given problem. These primitives are then learned by the policy. In addition to being useful in building complex actions from a discrete set of primitives, motor primitives can represent a desired move in state space, since they can be used to reach any possible state. As in MDPs described earlier, the transition from one state to another state based on which action is taken is easily tracked when using motor primitives. In this case, the output of the policy in Equation (1) can represent the change in the current state BIB002 ] as follows:ẋ Definition 11. High-level macro actions are decisions that determine the immediate plan of the agent. It is then broken down to lower-level action sequences. Examples of high-level decisions are grasp object or perform forehand. For a thorough formalization of learning from demonstrations, we refer the reader to BIB002 .