Imitation Learning: A Survey of Learning Methods <s> INDIRECT LEARNING <s> Evolutionary algorithms are commonly used to create high-performing strategies or agents for computer games. In this paper, we instead choose to evolve the racing tracks in a car racing game. An evolvable track representation is devised, and a multiobjective evolutionary algorithm maximises the entertainment value of the track relative to a particular human player. This requires a way to create accurate models of players' driving styles, as well as a tentative definition of when a racing track is fun, both of which are provided. We believe this approach opens up interesting new research questions and is potentially applicable to commercial racing games. <s> BIB001 </s> Imitation Learning: A Survey of Learning Methods <s> INDIRECT LEARNING <s> We present a comprehensive survey of robot Learning from Demonstration (LfD), a technique that develops policies from example state to action mappings. We introduce the LfD design choices in terms of demonstrator, problem space, policy derivation and performance, and contribute the foundations for a structure in which to categorize LfD research. Specifically, we analyze and categorize the multiple ways in which examples are gathered, ranging from teleoperation to imitation, as well as the various techniques for policy derivation, including matching functions, dynamics models and plans. To conclude we discuss LfD limitations and related promising areas for future research. <s> BIB002 </s> Imitation Learning: A Survey of Learning Methods <s> INDIRECT LEARNING <s> This paper presents a physics based simulator that allows kinesthetic interactions between a human and a robot to be recorded, and later used for imitation learning. We argue that kinesthetic interac- tion can be a very important tool for programming robots, if properly supported by a simulation engine. For this, we propose a new scheme for robot motion learning based on kinesthetic bootstrapping. Interactions with the robot are used to create a low-dimensional posture space. The posture space together with the presented simulation engine allow for fast imitation learning of behaviors. Early results of this approach, using Genetic Algorithms as learning technique, will be presented. <s> BIB003
In this section, we discuss indirect ways to learn policies that can complement or replace direct imitation. The policy can be refined from demonstrations, experience, or observation to be more accurate or to be more general and robust against unseen circumstances. It is often the case that direct imitation on its own is not adequate to reproduce robust, human-like behavior in intelligent agents. This limitation can be attributed to two main factors: (1) errors in demonstration and (2) poor generalization. Due to limitations of data acquisition techniques, such as the correspondence problem, sensor error, and physical influences in kinesthetic demonstrations BIB002 ], direct imitation can lead to inaccurate or unstable performance, especially in tasks that require precise motion in continuous space such as reaching or batting. For example, in BIB003 , a robot attempting to walk by directly mimicking the demonstrations would fall because the demonstrations do not accurately take into consideration the physical properties involved in the task, such as the robot's weight and center of mass. However, refinement of the policy through trial and error would take these factors into account and produce a stable motion. While generalization is an important issue in all machine-learning practices, a special case of generalization is highlighted in imitation learning applications. It is common that human demonstrations are provided as a sequence of actions. The dependence of each action on the previous part of the sequence violates the "iid" assumption of training samples that is integral to generalization in supervised learning . Moreover, since human experts provide only correct examples, the learner is unequipped to handle errors in the trajectory. If the learner deviates from the optimal performance at any point in the trajectory (which is expected in any machine-learning task), it would be presented with an unseen situation that the model is not trained to accommodate for. A clear example is provided in BIB001 , where supervised learning was used to learn a policy to drive a car. Given that human demonstrations contained only "good driving" with no crashes or close calls, when error occurs and the car deviates from demonstrated trajectories, the learner does not know how to recover.