Imitation Learning: A Survey of Learning Methods <s> APPLICATIONS <s> The many degrees-of-freedom and distributed sensing capability of dextrous robot hands permit the use of control programs that rely on qualitative changes in sensor feedback. One way of designing such a control program is to have the robot learn the qualitative control characteristics from examples. These examples may be provided via teleoperation. Results are presented for recognizing and segmenting manipulation primitives from a teleoperated task by analysis of features in sensor feedback. k-nearest quantized pattern vectors determine potential classifications. A hidden Markov model provides task context for the final segmentation. The illustrative task is picking up a plastic egg with a spatula. > <s> BIB001 </s> Imitation Learning: A Survey of Learning Methods <s> APPLICATIONS <s> Autonomous navigation is a difficult problem for traditional vision and robotic techniques, primarily because of the noise and variability associated with real world scenes. Autonomous navigation systems based on traditional image processing and pattern recognition techniques often perform well under certain conditions, but have problems with others. Part of the difficulty stems from the fact that the processing performed by these systems remains fixed across various environments. <s> BIB002 </s> Imitation Learning: A Survey of Learning Methods <s> APPLICATIONS <s> Abstract The formulation and optimization of joint trajectories for humanoid robots is quite different from this same task for standard robots because of the complexity of humanoid robots’ kinematics and dynamics. In this paper we exploit the similarity between human motion and humanoid robot motion to generate joint trajectories for humanoids. In particular, we show how to transform human motion information captured by an optical tracking device into a high dimensional trajectory for a humanoid robot. We propose an automatic approach to relate humanoid robot kinematic parameters to the kinematic parameters of a human performer. Based on this relationship we infer the desired trajectories in robot joint space. B-spline wavelets are utilized to efficiently represent the trajectories. The density of the basis functions on the time axis is selected automatically. Large-scale optimization techniques are employed to solve the underlying computational problems efficiently. We applied our method to the task of teaching a humanoid robot how to make various naturally looking movements. <s> BIB003 </s> Imitation Learning: A Survey of Learning Methods <s> APPLICATIONS <s> Abstract This paper evaluates a model of human imitation of abstract, two-arm movements. The model consists of a hierarchy of artificial neural networks, which are abstractions of brain regions involved in visuo-motor control. The model is validated in a biomechanical simulation of a 37 degree of freedom (DOF) humanoid. Input to the model are data from human arm movements recorded using video and marker-based tracking systems. Results show a high qualitative and quantitative agreement with human data. The model’s reproduction is better or comparable to that of human subjects imitating the same movements. <s> BIB004 </s> Imitation Learning: A Survey of Learning Methods <s> APPLICATIONS <s> ATR Human Information Science Laboratories, Kyoto, JapanEmail: ijspeert@usc.edu, jun@his.atr.co.jp, sschaal@usc.eduAbstractThis paper presents a new approach to the generationof rhythmic movement patterns with nonlinear dy-namical systems. Starting from a canonical limit cy-cle oscillator with well-de ned stability properties, wemodify the attractor landscape of the canonical sys-temby meansof statisticallearning methods toembedarbitrary smooth target patterns, however, withoutlosing the stability properties of the canonical sys-tem. In contrast to non-autonomous movement rep-resentations like splines, the learned pattern gener-ators remain autonomous dynamical systems whichrobustly cope with external perturbations that disruptthe time ow of the original pattern, and which canalso be modi ed on-line by additional perceptual vari-ables. A simple extension allows to cope with mul-tiple degrees-of-freedom (DOF) patterns, where allDOFs share the same fundamental frequency but,otherwise, can move in arbitrary phase and ampli-tude o sets to each other. We evaluate our meth-ods in learning from demonstration with an actual30 DOF humanoid robot. Figure-8 and drummingmovements are demonstrated by a human, recordedin joint angle space with an exoskeleton, and em-bedded in multi-dimensional rhythmic pattern gener-ators. The learned patterns can be used by the robotin various workspace locations and from arbitraryinitial conditions. Spatial and temporal invarianceof the pattern generators allow easy amplitude andspeed scaling without losing the qualitative signatureof a movement. This novel way of creating rhyth-mic patterns could tremendously facilitate rhythmicmovement generation, in particular in locomotion ofrobots and neural prosthetics in clinical applications. <s> BIB005 </s> Imitation Learning: A Survey of Learning Methods <s> APPLICATIONS <s> Mirror neurons within a monkey's premotor area F5 fire not only when the monkey performs a certain class of actions but also when the monkey observes another monkey (or the experimenter) perform a similar action. It has thus been argued that these neurons are crucial for understanding of actions by others. We offer the hand-state hypothesis as a new explanation of the evolution of this capability: the basic functionality of the F5 mirror system is to elaborate the appropriate feedback – what we call the hand state– for opposition-space based control of manual grasping of an object. Given this functionality, the social role of the F5 mirror system in understanding the actions of others may be seen as an exaptation gained by generalizing from one's own hand to an other's hand. In other words, mirror neurons first evolved to augment the “canonical” F5 neurons (active during self-movement based on observation of an object) by providing visual feedback on “hand state,” relating the shape of the hand to the shape of the object. We then introduce the MNS1 (mirror neuron system 1) model of F5 and related brain regions. The existing Fagg–Arbib–Rizzolatti–Sakata model represents circuitry for visually guided grasping of objects, linking the anterior intraparietal area (AIP) with F5 canonical neurons. The MNS1 model extends the AIP visual pathway by also modeling pathways, directed toward F5 mirror neurons, which match arm–hand trajectories to the affordances and location of a potential target object. We present the basic schemas for the MNS1 model, then aggregate them into three “grand schemas”– visual analysis of hand state, reach and grasp, and the core mirror circuit – for each of which we present a useful implementation (a non-neural visual processing system, a multijoint 3-D kinematics simulator, and a learning neural network, respectively). With this implementation we show how the mirror system may learnto recognize actions already in the repertoire of the F5 canonical neurons. We show that the connectivity pattern of mirror neuron circuitry can be established through training, and that the resultant network can exhibit a range of novel, physiologically interesting behaviors during the process of action recognition. We train the system on the basis of final grasp but then observe the whole time course of mirror neuron activity, yielding predictions for neurophysiological experiments under conditions of spatial perturbation, altered kinematics, and ambiguous grasp execution which highlight the importance of the timingof mirror neuron activity. <s> BIB006 </s> Imitation Learning: A Survey of Learning Methods <s> APPLICATIONS <s> Among humans, teaching various tasks is a complex process which relies on multiple means for interaction and learning, both on the part of the teacher and of the learner. Used together, these modalities lead to effective teaching and learning approaches, respectively. In the robotics domain, task teaching has been mostly addressed by using only one or very few of these interactions. In this paper we present an approach for teaching robots that relies on the key features and the general approach people use when teaching each other: first give a demonstration, then allow the learner to refine the acquired capabilities by practicing under the teacher's supervision, involving a small number of trials. Depending on the quality of the learned task, the teacher may either demonstrate it again or provide specific feedback during the learner's practice trial for further refinement. Also, as people do during demonstrations, the teacher can provide simple instructions and informative cues, increasing the performance of learning. Thus, instructive demonstrations, generalization over multiple demonstrations and practice trials are essential features for a successful human-robot teaching approach. We implemented a system that enables all these capabilities and validated these concepts with a Pioneer 2DX mobile robot learning tasks from multiple demonstrations and teacher feedback. <s> BIB007 </s> Imitation Learning: A Survey of Learning Methods <s> APPLICATIONS <s> We consider learning in a Markov decision process where we are not explicitly given a reward function, but where instead we can observe an expert demonstrating the task that we want to learn to perform. This setting is useful in applications (such as the task of driving) where it may be difficult to write down an explicit reward function specifying exactly how different desiderata should be traded off. We think of the expert as trying to maximize a reward function that is expressible as a linear combination of known features, and give an algorithm for learning the task demonstrated by the expert. Our algorithm is based on using "inverse reinforcement learning" to try to recover the unknown reward function. We show that our algorithm terminates in a small number of iterations, and that even though we may never recover the expert's reward function, the policy output by the algorithm will attain performance close to that of the expert, where here performance is measured with respect to the expert's unknown reward function. <s> BIB008 </s> Imitation Learning: A Survey of Learning Methods <s> APPLICATIONS <s> We present a computational approach to learning by observation (LBO) that allows users to program mobile robots by demonstrating a task. Unlike previous approaches, our system incorporates statistical-learning techniques and concepts from control theory to reduce the amount of domain knowledge needed to infer the intent of the user. To improve the generalization ability of the system, the user can demonstrate the task multiple times. We extract task subgoals from these demonstrations and automatically associate them with objects in the environment. As these objects move, the subgoals are updated accordingly. This gives our system the ability to learn from demonstrations performed in different environments. In this paper, we present the concepts used in our LBO system as well as experimental laboratory results in learning motor-skill tasks. <s> BIB009 </s> Imitation Learning: A Survey of Learning Methods <s> APPLICATIONS <s> Imitation is a powerful mechanism the human brain applies to extend its repertoire of solutions and behaviors suitable to solve problems of various kinds. From an abstract point of view, the major advantage of this strategy is that it reduces the search space of apropriate solutions. In this contribution, we discuss if and how the principle of imitation learning can facilitate the programing of life-like computer game charecters. We present different algorithms that learn from human generated training data and we show that machine learning can be applied on different levels of cognitive abstraction. <s> BIB010 </s> Imitation Learning: A Survey of Learning Methods <s> APPLICATIONS <s> This paper defines the research area of socially assistive robotics, focusing on assisting people through social interaction. While much attention has been paid to robots that provide assistance to people through physical contact (which we call contact assistive robotics), and to robots that entertain through social interaction (social interactive robotics), so far there is no clear definition of socially assistive robotics. We summarize active social assistive research projects and classify them by target populations, application domains, and interaction methods. While distinguishing these from socially interactive robotics endeavors, we discuss challenges and opportunities that are specific to the growing field of socially assistive robotics. <s> BIB011 </s> Imitation Learning: A Survey of Learning Methods <s> APPLICATIONS <s> Imitation is a powerful mechanism for transferring knowledge from an instructor to a naive observer, one that is deeply contingent on a state of shared attention between these two agents. In this paper we present Bayesian algorithms that implement the core of an imitation learning framework. We use gaze imitation, coupled with task-dependent saliency learning, to build a state of shared attention between the instructor and observer. We demonstrate the performance of our algorithms in a gaze following and saliency learning task implemented on an active vision robotic head. Our results suggest that the ability to follow gaze and learn instructor-and task-specific saliency models could play a crucial role in building systems capable of complex forms of human-robot interaction. <s> BIB012 </s> Imitation Learning: A Survey of Learning Methods <s> APPLICATIONS <s> Programming robots to carry out useful tasks is both a complex and non-trivial exercise. A simple and intuitive method to allow humans to train and shape robot behaviour is clearly a key goal in making this task easier. This paper describes an approach to this problem based on studies of social animals where two teaching strategies are applied to allow a human teacher to train a robot by moulding its actions within a carefully scaffolded environment. Within these enviroments sets of competences can be built by building stateslash action memory maps of the robot's interaction within that environment. These memory maps are then polled using a k-nearest neighbour based algorithm to provide a generalised competence. We take a novel approach in building the memory models by allowing the human teacher to construct them in a hierarchical manner. This mechanism allows a human trainer to build and extend an action-selection mechanism into which new skills can be added to the robot's repertoire of existing competencies. These techniques are implemented on physical Khepera miniature robots and validated on a variety of tasks. <s> BIB013 </s> Imitation Learning: A Survey of Learning Methods <s> APPLICATIONS <s> Tying suture knots is a time-consuming task performed frequently during minimally invasive surgery (MIS). Automating this task could greatly reduce total surgery time for patients. Current solutions to this problem replay manually programmed trajectories, but a more general and robust approach is to use supervised machine learning to smooth surgeon-given training trajectories and generalize from them. Since knottying generally requires a controller with internal memory to distinguish between identical inputs that require different actions at different points along a trajectory, it would be impossible to teach the system using traditional feedforward neural nets or support vector machines. Instead we exploit more powerful, recurrent neural networks (RNNs) with adaptive internal states. Results obtained using LSTM RNNs trained by the recent Evolino algorithm show that this approach can significantly increase the efficiency of suture knot tying in MIS over preprogrammed control <s> BIB014 </s> Imitation Learning: A Survey of Learning Methods <s> APPLICATIONS <s> We contribute an approach for interactive policy learning through expert demonstration that allows an agent to actively request and effectively represent demonstration examples. In order to address the inherent uncertainty of human demonstration, we represent the policy as a set of Gaussian mixture models (GMMs), where each model, with multiple Gaussian components, corresponds to a single action. Incrementally received demonstration examples are used as training data for the GMM set. We then introduce our confident execution approach, which focuses learning on relevant parts of the domain by enabling the agent to identify the need for and request demonstrations for specific parts of the state space. The agent selects between demonstration and autonomous execution based on statistical analysis of the uncertainty of the learned Gaussian mixture set. As it achieves proficiency at its task and gains confidence in its actions, the agent operates with increasing autonomy, eliminating the need for unnecessary demonstrations of already acquired behavior, and reducing both the training time and the demonstration workload of the expert. We validate our approach with experiments in simulated and real robot domains. <s> BIB015 </s> Imitation Learning: A Survey of Learning Methods <s> APPLICATIONS <s> Robot programming by demonstration (RPD) covers methods by which a robot learns new skills through human guidance. We present an interactive, multimodal RPD framework using active teaching methods that places the human teacher in the robot's learning loop. Two experiments are presented in which observational learning is first used to demonstrate a manipulation skill to a HOAP-3 humanoid robot by using motion sensors attached to the teacher's body. Then, putting the robot through the motion, the teacher incrementally refines the robot's skill by moving its arms manually, providing the appropriate scaffolds to reproduce the action. An incremental teaching scenario is proposed based on insights from various fields addressing developmental, psychological, and social issues related to teaching mechanisms in humans. Based on this analysis, different benchmarks are suggested to evaluate the setup further. <s> BIB016 </s> Imitation Learning: A Survey of Learning Methods <s> APPLICATIONS <s> Recent research has shown the benefit of framing problems of imitation learning as solutions to Markov Decision Problems. This approach reduces learning to the problem of recovering a utility function that makes the behavior induced by a near-optimal policy closely mimic demonstrated behavior. In this work, we develop a probabilistic approach based on the principle of maximum entropy. Our approach provides a well-defined, globally normalized distribution over decision sequences, while providing the same performance guarantees as existing methods. ::: ::: We develop our technique in the context of modeling real-world navigation and driving behaviors where collected data is inherently noisy and imperfect. Our probabilistic approach enables modeling of route preferences as well as a powerful new approach to inferring destinations and routes based on partial trajectories. <s> BIB017 </s> Imitation Learning: A Survey of Learning Methods <s> APPLICATIONS <s> This paper presents a physics based simulator that allows kinesthetic interactions between a human and a robot to be recorded, and later used for imitation learning. We argue that kinesthetic interac- tion can be a very important tool for programming robots, if properly supported by a simulation engine. For this, we propose a new scheme for robot motion learning based on kinesthetic bootstrapping. Interactions with the robot are used to create a low-dimensional posture space. The posture space together with the presented simulation engine allow for fast imitation learning of behaviors. Early results of this approach, using Genetic Algorithms as learning technique, will be presented. <s> BIB018 </s> Imitation Learning: A Survey of Learning Methods <s> APPLICATIONS <s> This paper is an initial approach to create a controller for the game TORCS by learning how another controller or humans play the game. We used data obtained from two controllers and from one human player. The first controller is the winner of the WCCI 2008 Simulated Car Racing Competition, and the second one is a hand coded controller that performs a complete lap in all tracks. First, each kind of controller is imitated separately, then a mix of the data is used to create new controllers. The imitation is performed by means of training a feed forward neural network with the data, using the backpropagation algorithm for learning. <s> BIB019 </s> Imitation Learning: A Survey of Learning Methods <s> APPLICATIONS <s> In this paper, we apply imitation learning to develop drivers for The Open Racing Car Simulator (TORCS). Our approach can be classified as a direct method in that it applies supervised learning to learn car racing behaviors from the data collected from other drivers. In the literature, this approach is known to have led to extremely poor performance with drivers capable of completing only very small parts of a track. In this paper we show that, by using high-level information about the track ahead of the car and by predicting high-level actions, it is possible to develop drivers with performances that in some cases are only 15% lower than the performance of the fastest driver available in TORCS. Our experimental results suggest that our approach can be effective in developing drivers with good performance in non-trivial tracks using a very limited amount of data and computational resources. We analyze the driving behavior of the controllers developed using our approach and identify perceptual aliasing as one of the factors which can limit performance of our approach. <s> BIB020 </s> Imitation Learning: A Survey of Learning Methods <s> APPLICATIONS <s> Currently the 2 percent growth rate for the world's older population exceeds the 1.2 rate for the world's population as a whole. By 2050, the number of individuals over the age 85 is projected to be three times more than there is today. Most of these individuals will need physical, emotional, and cognitive assistance. In this paper, we present a new adaptive robotic system based on the socially assistive robotics (SAR) technology that tries to provide a customized help protocol through motivation, encouragements, and companionship to users suffering from cognitive changes related to aging and/or Alzheimer's disease. Our results show that this approach can engage the patients and keep them interested in interacting with the robot, which, in turn, increases their positive behavior. <s> BIB021 </s> Imitation Learning: A Survey of Learning Methods <s> APPLICATIONS <s> Despite a history of games-based research, academia has generally regarded ::: commercial games as a distraction from the serious business of AI, rather than as an ::: opportunity to leverage this existing domain to the advancement of our knowledge. ::: Similarly, the computer game industry still relies on techniques that were developed ::: several decades ago, and has shown little interest in adopting more progressive ::: academic approaches. In recent times, however, these attitudes have begun to change; ::: under- and post-graduate games development courses are increasingly common, ::: while the industry itself is slowly but surely beginning to recognise the potential ::: offered by modern machine-learning approaches, though games which actually ::: implement said approaches on more than a token scale remain scarce. ::: One area which has not yet received much attention from either academia or industry ::: is imitation learning, which seeks to expedite the learning process by exploiting data ::: harvested from demonstrations of a given task. While substantial work has been done ::: in developing imitation techniques for humanoid robot movement, there has been ::: very little exploration of the challenges posed by interactive computer games. Given ::: that such games generally encode reasoning and decision-making behaviours which ::: are inherently more complex and potentially more interesting than limb motion data, ::: that they often provide inbuilt facilities for recording human play, that the generation ::: and collection of training samples is therefore far easier than in robotics, and that ::: many games have vast pre-existing libraries of these recorded demonstrations, it is ::: fair to say that computer games represent an extremely fertile domain for imitation ::: learning research. ::: In this thesis, we argue in favour of using modern, commercial computer games to ::: study, model and reproduce humanlike behaviour. We provide an overview of the ::: biological and robotic imitation literature as well as the current status of game AI, highlighting techniques which may be adapted for the purposes of game-based ::: imitation. We then proceed to describe our contributions to the field of imitation ::: learning itself, which encompass three distinct categories: theory, implementation ::: and evaluation. ::: We first describe the development of a fully-featured Java API - the Quake2 Agent ::: Simulation Environment (QASE) - designed to facilitate both research and education ::: in imitation and general machine-learning, using the game Quake 2 as a testbed. We ::: outline our motivation for developing QASE, discussing the shortcomings of existing ::: APIs and the steps which we have taken to circumvent them. We describe QASE’s ::: network layer, which acts as an interface between the local AI routines and the ::: Quake 2 server on which the game environment is maintained, before detailing the ::: API’s agent architecture, which includes an interface to the MatLab programming ::: environment and the ability to parse and analyse full recordings of game sessions. ::: We conclude the chapter with a discussion of QASE’s adoption by numerous ::: universities as both an undergraduate teaching tool and research platform. ::: We then proceed to describe the various imitative mechanisms which we have ::: developed using QASE and its MatLab integration facilities. We first outline a ::: behaviour model based on a well-known psychological model of human planning. ::: Drawing upon previous research, we also identify a set of believability criteria - ::: elements of agent behaviour which are of particular importance in determining the ::: “humanness” of its in-game appearance. We then detail a reinforcement-learning ::: approach to imitating the human player’s navigation of his environment, centred ::: upon his pursuit of items as strategic goals. In the subsequent section, we describe ::: the integration of this strategic system with a Bayesian mechanism for the imitation ::: of tactical and motion-modelling behaviours. Finally, we outline a model for the ::: imitation of reactive combat behaviours; specifically, weapon-selection and aiming. Experiments are presented in each case to demonstrate the imitative mechanisms’ ::: ability to accurately reproduce observed behaviours. ::: Finally, we criticise the lack of any existing methodology to formally gauge the ::: believability of game agents, and observe that the few previous attempts have been ::: extremely ad-hoc and informal. We therefore propose a generalised approach to such ::: testing; the Bot-Oriented Turing Test (BOTT). This takes the form of an anonymous ::: online questionnaire, an accompanying protocol to which examiners should adhere, ::: and the formulation of a believability index which numerically expresses each agent’s ::: humanness as indicated by its observers, weighted by their experience and the ::: accuracy with which the agents were identified. To both validate the survey approach ::: and to determine the efficacy of our imitative models, we present a series of ::: experiments which use the believability test to evaluate our own imitation agents ::: against both human players and traditional artificial bots. We demonstrate that our ::: imitation agents perform substantially better than even a highly-regarded rule-based ::: agent, and indeed approach the believability of actual human players. ::: Some suggestions for future directions in our research, as well as a broader ::: discussion of open questions, conclude this thesis. <s> BIB022 </s> Imitation Learning: A Survey of Learning Methods <s> APPLICATIONS <s> Abstract Artiﬁcial general intelligence (AGI) refers to research aimed at tackling the full problemof artiﬁcial intelligence, that is, create truly intelligent agents. This sets it apart from mostAI research which aims at solving relatively narrow domains, such as character recognition,motion planning, or increasing player satisfaction in games. But how do we know when anagent is truly intelligent? A common point of reference in the AGI community is Legg andHutter’s formal deﬁnition of universal intelligence, which has the appeal of simplicity andgenerality but is unfortunately incomputable.Games of various kinds are commonly used as benchmarks for “narrow” AI research,as they are considered to have many important properties. We argue that many of theseproperties carry over to the testing of general intelligence as well. We then sketch how suchtesting could practically be carried out. The central part of this sketch is an extension ofuniversal intelligence to deal with ﬁnite time, and the use of sampling of the space of gamesexpressed in a suitably biased game description language.Keywords: measure of intelligence, games <s> BIB023 </s> Imitation Learning: A Survey of Learning Methods <s> APPLICATIONS <s> Abstract The ongoing development of robotics on the one hand and, on the other hand, the foreseen relative growth in number of elderly individuals suffering from dementia, raises the question of which contribution robotics could have to rationalize and maintain, or even improve the quality of care. The objective of this review was to assess the published effects and effectiveness of robot interventions aiming at social assistance in elderly care. We searched, using Medical Subject Headings terms and free words, in the CINAHL, MEDLINE, Cochrane, BIOMED, PUBMED, PsycINFO, and EMBASE databases. Also the IEEE Digital Library was searched. No limitations were applied for the date of publication. Only articles written in English were taken into account. Collected publications went through a selection process. In the first step, publications were collected from major databases using a search query. In the second step, 3 reviewers independently selected publications on their title, using predefined selection criteria. In the third step, publications were judged based on their abstracts by the same reviewers, using the same selection criteria. In the fourth step, one reviewer made the final selection of publications based on complete content. Finally, 41 publications were included in the review, describing 17 studies involving 4 robot systems. Most studies reported positive effects of companion-type robots on (socio)psychological (eg, mood, loneliness, and social connections and communication) and physiological (eg, stress reduction) parameters. The methodological quality of the studies was, mostly, low. Although positive effects were reported, the scientific value of the evidence was limited. The positive results described, however, prompt further effectiveness research in this field. <s> BIB024 </s> Imitation Learning: A Survey of Learning Methods <s> APPLICATIONS <s> Close physical interaction between robots and humans is a particularly challenging aspect of robot development. For successful interaction and cooperation, the robot must have the ability to adapt its behavior to the human counterpart. Based on our earlier work, we present and evaluate a computationally efficient machine learning algorithm that is well suited for such close-contact interaction scenarios. We show that this algorithm helps to improve the quality of the interaction between a robot and a human caregiver. To this end, we present two human-in-the-loop learning scenarios that are inspired by human parenting behavior, namely, an assisted standing-up task and an assisted walking task. <s> BIB025 </s> Imitation Learning: A Survey of Learning Methods <s> APPLICATIONS <s> We share our modern world with bots chatbots to converse with, roombots to clean our houses, spambots to fill our e-mail inboxes, and medibots to assist our surgeons. This book is about computer game bots, virtual companions who accompany us in virtual worlds or sharpen our fighting skills. These bots must be believable, that is human players should believe they are interacting with entities operating at a human level bots are more fun if they behave like we do. This book shows how to create believable bots that play computer games, and it discusses the implications of making them appear human. The chapters in this book present the state of the art in research on and development of game bots, and they also look beyond the design aspects to address deep questions: Is a bot that plays like a person intelligent? Does it have emotions? Is it conscious? The topic is inherently interdisciplinary, and the work draws from research and practice in many fields, such as design, creativity, entertainment, and graphics; learning, psychology, and sociology; artificial intelligence, embodiment, agents, machine learning, robotics, humancomputer interaction, and artificial life; cognition and neuroscience; and evolutionary computing. The contributing authors are among the leading researchers and developers in this field, and most of the examples and case studies involve analysis of commercial products. The book will be of value to graduate students and academic researchers in artificial intelligence, and to engineers charged with the design of entertaining games. <s> BIB026 </s> Imitation Learning: A Survey of Learning Methods <s> APPLICATIONS <s> Abstract We describe and compare several methods for generating game character controllers that mimic the playing style of a particular human player, or of a population of human players, across video game levels. Similarity in playing style is measured through an evaluation framework, that compares the play trace of one or several human players with the punctuated play trace of an AI player. The methods that are compared are either hand-coded, direct (based on supervised learning) or indirect (based on maximising a similarity measure). We find that a method based on neuroevolution performs best both in terms of the instrumental similarity measure and in phenomenological evaluation by human spectators. A version of the classic platform game “Super Mario Bros” is used as the testbed game in this study but the methods are applicable to other games that are based on character movement in space. <s> BIB027 </s> Imitation Learning: A Survey of Learning Methods <s> APPLICATIONS <s> JVRB, 11(2014), no. 1. - Imitation learning is a promising approach for generating life-like behaviors of virtual humans and humanoid robots. So far, however, imitation learning has been mostly restricted to single agent settings where observed motions are adapted to new environment conditions but not to the dynamic behavior of interaction partners. In this paper, we introduce a new imitation learning approach that is based on the simultaneous motion capture of two human interaction partners. From the observed interactions, low-dimensional motion models are extracted and a mapping between these motion models is learned. This interaction model allows the real-time generation of agent behaviors that are responsive to the body movements of an interaction partner. The interaction model can be applied both to the animation of virtual characters as well as to the behavior generation for humanoid robots. <s> BIB028
Advances in imitation learning open the door for a variety of potential applications. Though commercialization and finished products of such applications may not be realized yet, we present some of the directions taken in the literature and their potential applications. Autonomous vehicles have been a popular concept in AI from its early days. And recently self-driving cars have been gaining a lot of attention from car manufacturers and tech companies alike. The advancement in sensors and onboard computers in modern cars has rekindled the interest in producing driverless vehicles commercially. Early research in imitation learning focused on this problem, proposing a method for learning to fly an aircraft from demonstrations provided via remote control  and self-driving road vehicles BIB002 . Since then, many researchers have directed imitation learning research to navigational or driving tasks BIB008 BIB015 BIB009 BIB013 BIB019 BIB020 ], not only low-level control car route selection and planning BIB017 . Assistive robotics aims to provide intelligent robots that can help elderly or recovering individuals in their day-to-day activities. Generalization is necessary in most applications as the human partner is usually untrained, so the robot must be able to behave robustly in unseen situations. The human-robot interaction is not limited to physical assistance. Socially assistive robots can offer help with sociological and mental problems BIB011 BIB024 BIB021 ]. For robots to be effective in such a social context, their behavior must be human-like to be intuitively recognized by the human partner. Therefore, imitation learning has the potential to be an integral part in assistive robots. The same argument can be made for teaching infants using interactive robots. BIB025 incorporate interaction in the training process to account for the changing behavior of the human partner. The robot, therefore, can adapt to the human's reactions, as the human naturally makes similar adaptations. Electronic games are a multi-billion-dollar industry, and one in which realism and immersion are important factors. There is an ever-growing demand for believable artificial intelligence to enhance immersion in games BIB026 ]. Similar to human-robot interaction, the possibilities are too great to consider with explicit programming  ], especially in modern games where the player has more freedom and control and the environments are becoming increasingly complex. A number of studies investigate the effectiveness of imitation learning in video games such as first-person shooters  BIB022 BIB010 , platformers BIB027 , and racing BIB019 . These examples are applied to relatively simple problems, with limited action options and few features to consider from the environment; however, they show promising performances from imitation agents that could be extended to more complex games in the future. For instance, the study in BIB022 showed that not only are imitating agents subjectively believable but also they outperform other hand-crafted agents. Electronic games provide a friendly platform for advancing imitation learning as they do not require additional systems such as sensors and hardware, and the cost of faults is low compared to applications where faults might endanger people or property. They can also be used as a testbed for artificial general intelligence BIB023 . Humanoid robots are one of the domains most associated with artificial intelligence. They are one of the most relatable domains, because many of their potential applications are quite obvious. The premise of humanoid robots is to replace some of the workload that humans do. These problems range from specific tasks such as physical chores and housework to generic problem solving. Since most of the required tasks are already performed by humans, humanoid robot applications are inherently suitable for the concept of learning by imitation. Humanoid robots can learn to perform actions that utilize only part of their skeleton BIB004 BIB005 BIB028 or the entire body BIB018 BIB016 BIB003 . Since humanoid robots are commonly used to interact with humans in a social context, it is important not only to learn how to move but also how to direct their attention to important occurrences. In BIB012 , a robot learns where to look by imitating a teacher's gaze and learns to estimate saliency based on the teacher's preferences. Another field of robotic applications is automation. It is different from the aforementioned domains in that automation is more relevant to industrial tasks while the other domains aim to create AI for personal products and services. However, automation can still be useful in domestic applications BIB013 ]. While automation is not a new concept, learning by imitation introduces generalization and adaptability to automated tasks BIB007 . This means that the robot can act robustly in unseen situations, such as the introduction of obstacles or changes in the shape or position of relevant objects. Generalization diminishes the need for supervision and human intervention between small tasks. Imitation learning research done in this direction focuses on object manipulation, such as sorting and assembly tasks BIB013 BIB001 BIB006 BIB016 BIB007 . Another example of automation is medical procedures. In BIB014 , a robot is trained to automate part of a surgical procedure from surgeons' demonstrations.