Imitation Learning: A Survey of Learning Methods <s> Classification <s> We contribute an approach for interactive policy learning through expert demonstration that allows an agent to actively request and effectively represent demonstration examples. In order to address the inherent uncertainty of human demonstration, we represent the policy as a set of Gaussian mixture models (GMMs), where each model, with multiple Gaussian components, corresponds to a single action. Incrementally received demonstration examples are used as training data for the GMM set. We then introduce our confident execution approach, which focuses learning on relevant parts of the domain by enabling the agent to identify the need for and request demonstrations for specific parts of the state space. The agent selects between demonstration and autonomous execution based on statistical analysis of the uncertainty of the learned Gaussian mixture set. As it achieves proficiency at its task and gains confidence in its actions, the agent operates with increasing autonomy, eliminating the need for unnecessary demonstrations of already acquired behavior, and reducing both the training time and the demonstration workload of the expert. We validate our approach with experiments in simulated and real robot domains. <s> BIB001 </s> Imitation Learning: A Survey of Learning Methods <s> Classification <s> The Maximum Margin Planning (MMP) (Ratliff et al., 2006) algorithm solves imitation learning problems by learning linear mappings from features to cost functions in a planning domain. The learned policy is the result of minimum-cost planning using these cost functions. These mappings are chosen so that example policies (or trajectories) given by a teacher appear to be lower cost (with a lossscaled margin) than any other policy for a given planning domain. We provide a novel approach, MMPBOOST , based on the functional gradient descent view of boosting (Mason et al., 1999; Friedman, 1999a) that extends MMP by “boosting” in new features. This approach uses simple binary classification or regression to improve performance of MMP imitation learning, and naturally extends to the class of structured maximum margin prediction problems. (Taskar et al., 2005) Our technique is applied to navigation and planning problems for outdoor mobile robots and robotic legged locomotion. <s> BIB002 </s> Imitation Learning: A Survey of Learning Methods <s> Classification <s> We present a comprehensive survey of robot Learning from Demonstration (LfD), a technique that develops policies from example state to action mappings. We introduce the LfD design choices in terms of demonstrator, problem space, policy derivation and performance, and contribute the foundations for a structure in which to categorize LfD research. Specifically, we analyze and categorize the multiple ways in which examples are gathered, ranging from teleoperation to imitation, as well as the various techniques for policy derivation, including matching functions, dynamics models and plans. To conclude we discuss LfD limitations and related promising areas for future research. <s> BIB003 </s> Imitation Learning: A Survey of Learning Methods <s> Classification <s> Developing coordination among multiple agents and enabling them to exhibit teamwork is a challenging yet exciting task that can benefit many of the complex real-life problems. This research uses imitation to learn collaborative strategies for a team of agents. Imitation based learning involves learning from an expert by observing him/her demonstrating a task and then replicating it. The key idea is to involve multiple human experts during demonstration to teach autonomous agents how to work in coordination. The effectiveness of the proposed methodology has been assessed in a goal defending scenario of the RoboCup Soccer Simulation 3D league. The process involves multiple human demonstrators controlling soccer agents via game controllers and demonstrating them how to play soccer in coordination. The data gathered during this phase is used as training data to learn a classification model which is later used by the soccer agents to make autonomous decisions during actual matches. Different performance evaluation metrics are derived to compare the performance of imitating agent with that of the human-driven agent and hand-coded (if-then-else rules) agent. <s> BIB004 </s> Imitation Learning: A Survey of Learning Methods <s> Classification <s> Robots assisting disabled or elderly people in the performance of activities of daily living need to perform complex manipulation tasks which are highly dependent on the environment and preferences of the user. In addition, these environments and users are not suitable for the collection of massive amounts of training data, as the manipulated objects can be fragile, and the wheelchair-bound users might have difficulty recovering from a failed manipulation task. ::: In this paper, we propose an end-to-end learning mechanism for the type of complex robot arm trajectories used in manipulation tasks for assistive robots. The trajectory is learned using a recurrent neural network that can generate the trajectory in real-time based on the current situation of the end-effector, the objects in the environment and the preferences of the user. The learning data is acquired from a simulation environment where the human can demonstrate the task in a simulation closely modeling his or her own environment. Experiments using two different manipulation tasks show that the robot can learn the manipulation planning as well the ability to recover from failure. <s> BIB005
Classification is a popular task in machine learning where observations are automatically categorized into a finite set of classes. A classifier h(x) is used to predict the class y to which an independent observation x belongs, where y ∈ Y , Y = {y 1 , y 2 . . . y p } is a finite set of classes, and x = {x 1 , x 2 . . . x m } is a vector of m features. In supervised classification, h(x) is trained using a dataset of n labeled samples ( Classification approaches are used when the learner's actions can be categorized into discrete classes BIB003 . This is suitable for applications where the action can be viewed as a decision such as navigation BIB001 and flight simulators  . In BIB001 , a Gaussian mixture model (GMM) is trained to predict navigational decisions. Meta-classifiers are used in to learn a policy to play computer games. The base classifier used in this article is a neural network. In the kart racing game, the analog joystick commands are discretized into 15 buckets, reducing the problem to a 15-class classification problem. So the neural network used had 15 output nodes. The Mario Bros game uses a discrete controller. Actions are selected by pressing one or more of four buttons. So in the neural network, the action for a frame is represented by four output nodes. This enables the classifier to choose multiple classes for the same instance. Although the results are promising, it is argued that using an Inverse Optimal Control (IOC) technique BIB002 ] as the base classifier might be beneficial. In , the experiments are repeated this time using regression (see regression section) to learn the analog input in Super Tux Kart. For Mario Bros, four Support Vector Machine (SVM) classifiers replace the neural network to predict the value of each of the four binary classes. Classification can also be used to make decisions that entail lower-level actions. In BIB004 , high-level decisions are predicted by the classifier in a multiagent soccer simulation. Decisions such as "Approach ball" and "Dribble toward goal" can then be deterministically executed using lower-level actions. An empirical study is conducted to evaluate which classifiers are best suited for the imitation learning task. Four different classifiers are compared with respect to accuracy and learning time. The results show that a number of classifiers can perform predictions with comparable accuracy; however, the learning time relative to the number of demonstrations can vary greatly BIB004 . Recurrent neural networks (RNNs) are used in BIB005 to learn trajectories for object manipulation from demonstrations. RNNs incorporate memory of past actions when considering the next action. Storing memory enables the network to learn corrective behavior such as recovery from failure given that the teacher demonstrates such a scenario.