Imitation Learning: A Survey of Learning Methods <s> EVALUATION <s> Imitation is a powerful mechanism the human brain applies to extend its repertoire of solutions and behaviors suitable to solve problems of various kinds. From an abstract point of view, the major advantage of this strategy is that it reduces the search space of apropriate solutions. In this contribution, we discuss if and how the principle of imitation learning can facilitate the programing of life-like computer game charecters. We present different algorithms that learn from human generated training data and we show that machine learning can be applied on different levels of cognitive abstraction. <s> BIB001 </s> Imitation Learning: A Survey of Learning Methods <s> EVALUATION <s> The Robosoccer simulator is a challenging environment, where a human introduces a team of agents into a football virtual environment. Typically, agents are programmed by hand, but it would be a great advantage to transfer human experience into football agents. The first aim of this paper is to use machine learning techniques to obtain models of humans playing Robosoccer. These models can be used later to control a Robosoccer agent. However, models did not play as smoothly and optimally as the human. To solve this problem, the second goal of this paper is to incrementally correct models by means of evolutionary techniques, and to adapt them against more difficult opponents than the ones beatable by the human. <s> BIB002 </s> Imitation Learning: A Survey of Learning Methods <s> EVALUATION <s> Despite a history of games-based research, academia has generally regarded ::: commercial games as a distraction from the serious business of AI, rather than as an ::: opportunity to leverage this existing domain to the advancement of our knowledge. ::: Similarly, the computer game industry still relies on techniques that were developed ::: several decades ago, and has shown little interest in adopting more progressive ::: academic approaches. In recent times, however, these attitudes have begun to change; ::: under- and post-graduate games development courses are increasingly common, ::: while the industry itself is slowly but surely beginning to recognise the potential ::: offered by modern machine-learning approaches, though games which actually ::: implement said approaches on more than a token scale remain scarce. ::: One area which has not yet received much attention from either academia or industry ::: is imitation learning, which seeks to expedite the learning process by exploiting data ::: harvested from demonstrations of a given task. While substantial work has been done ::: in developing imitation techniques for humanoid robot movement, there has been ::: very little exploration of the challenges posed by interactive computer games. Given ::: that such games generally encode reasoning and decision-making behaviours which ::: are inherently more complex and potentially more interesting than limb motion data, ::: that they often provide inbuilt facilities for recording human play, that the generation ::: and collection of training samples is therefore far easier than in robotics, and that ::: many games have vast pre-existing libraries of these recorded demonstrations, it is ::: fair to say that computer games represent an extremely fertile domain for imitation ::: learning research. ::: In this thesis, we argue in favour of using modern, commercial computer games to ::: study, model and reproduce humanlike behaviour. We provide an overview of the ::: biological and robotic imitation literature as well as the current status of game AI, highlighting techniques which may be adapted for the purposes of game-based ::: imitation. We then proceed to describe our contributions to the field of imitation ::: learning itself, which encompass three distinct categories: theory, implementation ::: and evaluation. ::: We first describe the development of a fully-featured Java API - the Quake2 Agent ::: Simulation Environment (QASE) - designed to facilitate both research and education ::: in imitation and general machine-learning, using the game Quake 2 as a testbed. We ::: outline our motivation for developing QASE, discussing the shortcomings of existing ::: APIs and the steps which we have taken to circumvent them. We describe QASE’s ::: network layer, which acts as an interface between the local AI routines and the ::: Quake 2 server on which the game environment is maintained, before detailing the ::: API’s agent architecture, which includes an interface to the MatLab programming ::: environment and the ability to parse and analyse full recordings of game sessions. ::: We conclude the chapter with a discussion of QASE’s adoption by numerous ::: universities as both an undergraduate teaching tool and research platform. ::: We then proceed to describe the various imitative mechanisms which we have ::: developed using QASE and its MatLab integration facilities. We first outline a ::: behaviour model based on a well-known psychological model of human planning. ::: Drawing upon previous research, we also identify a set of believability criteria - ::: elements of agent behaviour which are of particular importance in determining the ::: “humanness” of its in-game appearance. We then detail a reinforcement-learning ::: approach to imitating the human player’s navigation of his environment, centred ::: upon his pursuit of items as strategic goals. In the subsequent section, we describe ::: the integration of this strategic system with a Bayesian mechanism for the imitation ::: of tactical and motion-modelling behaviours. Finally, we outline a model for the ::: imitation of reactive combat behaviours; specifically, weapon-selection and aiming. Experiments are presented in each case to demonstrate the imitative mechanisms’ ::: ability to accurately reproduce observed behaviours. ::: Finally, we criticise the lack of any existing methodology to formally gauge the ::: believability of game agents, and observe that the few previous attempts have been ::: extremely ad-hoc and informal. We therefore propose a generalised approach to such ::: testing; the Bot-Oriented Turing Test (BOTT). This takes the form of an anonymous ::: online questionnaire, an accompanying protocol to which examiners should adhere, ::: and the formulation of a believability index which numerically expresses each agent’s ::: humanness as indicated by its observers, weighted by their experience and the ::: accuracy with which the agents were identified. To both validate the survey approach ::: and to determine the efficacy of our imitative models, we present a series of ::: experiments which use the believability test to evaluate our own imitation agents ::: against both human players and traditional artificial bots. We demonstrate that our ::: imitation agents perform substantially better than even a highly-regarded rule-based ::: agent, and indeed approach the believability of actual human players. ::: Some suggestions for future directions in our research, as well as a broader ::: discussion of open questions, conclude this thesis. <s> BIB003 </s> Imitation Learning: A Survey of Learning Methods <s> EVALUATION <s> Imitation Learning has been shown to be successful in solving many challenging real-world problems. Some recent approaches give strong performance guarantees by training the policy iteratively. However, it is important to note that these guarantees depend on how well the policy we found can imitate the oracle on the training data. When there is a substantial difference between the oracle's ability and the learner's policy space, we may fail to find a policy that has low error on the training set. In such cases, we propose to use a coach that demonstrates easy-to-learn actions for the learner and gradually approaches the oracle. By a reduction of learning by demonstration to online learning, we prove that coaching can yield a lower regret bound than using the oracle. We apply our algorithm to cost-sensitive dynamic feature selection, a hard decision problem that considers a user-specified accuracy-cost trade-off. Experimental results on UCI datasets show that our method outperforms state-of-the-art imitation learning methods in dynamic feature selection and two static feature selection methods. <s> BIB004 </s> Imitation Learning: A Survey of Learning Methods <s> EVALUATION <s> Developing coordination among multiple agents and enabling them to exhibit teamwork is a challenging yet exciting task that can benefit many of the complex real-life problems. This research uses imitation to learn collaborative strategies for a team of agents. Imitation based learning involves learning from an expert by observing him/her demonstrating a task and then replicating it. The key idea is to involve multiple human experts during demonstration to teach autonomous agents how to work in coordination. The effectiveness of the proposed methodology has been assessed in a goal defending scenario of the RoboCup Soccer Simulation 3D league. The process involves multiple human demonstrators controlling soccer agents via game controllers and demonstrating them how to play soccer in coordination. The data gathered during this phase is used as training data to learn a classification model which is later used by the soccer agents to make autonomous decisions during actual matches. Different performance evaluation metrics are derived to compare the performance of imitating agent with that of the human-driven agent and hand-coded (if-then-else rules) agent. <s> BIB005 </s> Imitation Learning: A Survey of Learning Methods <s> EVALUATION <s> The theory of reinforcement learning provides a normative account, deeply rooted in psychological and neuroscientific perspectives on animal behaviour, of how agents may optimize their control of an environment. To use reinforcement learning successfully in situations approaching real-world complexity, however, agents are confronted with a difficult task: they must derive efficient representations of the environment from high-dimensional sensory inputs, and use these to generalize past experience to new situations. Remarkably, humans and other animals seem to solve this problem through a harmonious combination of reinforcement learning and hierarchical sensory processing systems, the former evidenced by a wealth of neural data revealing notable parallels between the phasic signals emitted by dopaminergic neurons and temporal difference reinforcement learning algorithms. While reinforcement learning agents have achieved some successes in a variety of domains, their applicability has previously been limited to domains in which useful features can be handcrafted, or to domains with fully observed, low-dimensional state spaces. Here we use recent advances in training deep neural networks to develop a novel artificial agent, termed a deep Q-network, that can learn successful policies directly from high-dimensional sensory inputs using end-to-end reinforcement learning. We tested this agent on the challenging domain of classic Atari 2600 games. We demonstrate that the deep Q-network agent, receiving only the pixels and the game score as inputs, was able to surpass the performance of all previous algorithms and achieve a level comparable to that of a professional human games tester across a set of 49 games, using the same algorithm, network architecture and hyperparameters. This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks. <s> BIB006
Imitation learning is a relatively young field, and evaluation of novel algorithms is challenging due to the lack of standard evaluation criteria and benchmark tasks. It is common that experiments are conducted on specialized robots or simulators. This makes comparison to other approaches difficult for two reasons: first, because the designed algorithms are often specialized for a specific task and setup, and second, because reproducing the results of an experiment needs special hardware or software that may not be available to other researchers. One track of imitation learning uses structured prediction datasets as a benchmark for comparison with other techniques BIB004  . However, structured prediction tasks do not always correspond to real robotic applications. The dependency of a state on previous states can be limited. Furthermore, simple Hamming loss is usually employed; that is, if a mistake is made in step t, the best action for t + 1 remains unchanged. This makes such benchmarks less suitable for testing robustness to errors and the ability to recover. Although not widely used to compare different techniques, some tools are readily available to perform and evaluate imitation learning, such as video games BIB001 BIB003 BIB006  , AI simulators BIB005 BIB002 , and reinforcement learning benchmarks   ]. Methods for evaluating imitation learning applications can be quantitative or qualitative.