Imitation Learning: A Survey of Learning Methods <s> Qualitative Evaluation <s> A general game player is an agent capable of taking as input a description of a game's rules in a formal language and proceeding to play without any subsequent human input. To do well, an agent should learn from experience with past games and transfer the learned knowledge to new problems. We introduce a graph-based method for identifying previously encountered games and prove its robustness formally. We then describe how the same basic approach can be used to identify similar but non-identical games. We apply this technique to automate domain mapping for value function transfer and speed up reinforcement learning on variants of previously played games. Our approach is fully implemented with empirical results in the general game playing system. <s> BIB001 </s> Imitation Learning: A Survey of Learning Methods <s> Qualitative Evaluation <s> Despite a history of games-based research, academia has generally regarded ::: commercial games as a distraction from the serious business of AI, rather than as an ::: opportunity to leverage this existing domain to the advancement of our knowledge. ::: Similarly, the computer game industry still relies on techniques that were developed ::: several decades ago, and has shown little interest in adopting more progressive ::: academic approaches. In recent times, however, these attitudes have begun to change; ::: under- and post-graduate games development courses are increasingly common, ::: while the industry itself is slowly but surely beginning to recognise the potential ::: offered by modern machine-learning approaches, though games which actually ::: implement said approaches on more than a token scale remain scarce. ::: One area which has not yet received much attention from either academia or industry ::: is imitation learning, which seeks to expedite the learning process by exploiting data ::: harvested from demonstrations of a given task. While substantial work has been done ::: in developing imitation techniques for humanoid robot movement, there has been ::: very little exploration of the challenges posed by interactive computer games. Given ::: that such games generally encode reasoning and decision-making behaviours which ::: are inherently more complex and potentially more interesting than limb motion data, ::: that they often provide inbuilt facilities for recording human play, that the generation ::: and collection of training samples is therefore far easier than in robotics, and that ::: many games have vast pre-existing libraries of these recorded demonstrations, it is ::: fair to say that computer games represent an extremely fertile domain for imitation ::: learning research. ::: In this thesis, we argue in favour of using modern, commercial computer games to ::: study, model and reproduce humanlike behaviour. We provide an overview of the ::: biological and robotic imitation literature as well as the current status of game AI, highlighting techniques which may be adapted for the purposes of game-based ::: imitation. We then proceed to describe our contributions to the field of imitation ::: learning itself, which encompass three distinct categories: theory, implementation ::: and evaluation. ::: We first describe the development of a fully-featured Java API - the Quake2 Agent ::: Simulation Environment (QASE) - designed to facilitate both research and education ::: in imitation and general machine-learning, using the game Quake 2 as a testbed. We ::: outline our motivation for developing QASE, discussing the shortcomings of existing ::: APIs and the steps which we have taken to circumvent them. We describe QASE’s ::: network layer, which acts as an interface between the local AI routines and the ::: Quake 2 server on which the game environment is maintained, before detailing the ::: API’s agent architecture, which includes an interface to the MatLab programming ::: environment and the ability to parse and analyse full recordings of game sessions. ::: We conclude the chapter with a discussion of QASE’s adoption by numerous ::: universities as both an undergraduate teaching tool and research platform. ::: We then proceed to describe the various imitative mechanisms which we have ::: developed using QASE and its MatLab integration facilities. We first outline a ::: behaviour model based on a well-known psychological model of human planning. ::: Drawing upon previous research, we also identify a set of believability criteria - ::: elements of agent behaviour which are of particular importance in determining the ::: “humanness” of its in-game appearance. We then detail a reinforcement-learning ::: approach to imitating the human player’s navigation of his environment, centred ::: upon his pursuit of items as strategic goals. In the subsequent section, we describe ::: the integration of this strategic system with a Bayesian mechanism for the imitation ::: of tactical and motion-modelling behaviours. Finally, we outline a model for the ::: imitation of reactive combat behaviours; specifically, weapon-selection and aiming. Experiments are presented in each case to demonstrate the imitative mechanisms’ ::: ability to accurately reproduce observed behaviours. ::: Finally, we criticise the lack of any existing methodology to formally gauge the ::: believability of game agents, and observe that the few previous attempts have been ::: extremely ad-hoc and informal. We therefore propose a generalised approach to such ::: testing; the Bot-Oriented Turing Test (BOTT). This takes the form of an anonymous ::: online questionnaire, an accompanying protocol to which examiners should adhere, ::: and the formulation of a believability index which numerically expresses each agent’s ::: humanness as indicated by its observers, weighted by their experience and the ::: accuracy with which the agents were identified. To both validate the survey approach ::: and to determine the efficacy of our imitative models, we present a series of ::: experiments which use the believability test to evaluate our own imitation agents ::: against both human players and traditional artificial bots. We demonstrate that our ::: imitation agents perform substantially better than even a highly-regarded rule-based ::: agent, and indeed approach the believability of actual human players. ::: Some suggestions for future directions in our research, as well as a broader ::: discussion of open questions, conclude this thesis. <s> BIB002 </s> Imitation Learning: A Survey of Learning Methods <s> Qualitative Evaluation <s> Abstract We describe and compare several methods for generating game character controllers that mimic the playing style of a particular human player, or of a population of human players, across video game levels. Similarity in playing style is measured through an evaluation framework, that compares the play trace of one or several human players with the punctuated play trace of an AI player. The methods that are compared are either hand-coded, direct (based on supervised learning) or indirect (based on maximising a similarity measure). We find that a method based on neuroevolution performs best both in terms of the instrumental similarity measure and in phenomenological evaluation by human spectators. A version of the classic platform game “Super Mario Bros” is used as the testbed game in this study but the methods are applicable to other games that are based on character movement in space. <s> BIB003
In some applications, it is as important for the action to appear natural and believable as it is to achieve the target. It is difficult to quantify the believability of a performance, even by comparing it to recorded human performances. In this case, it is common for the performance to be subjectively judged by human inspection. As with any subjective analysis, it is advisable to use more than one observer and regulate their decisions as comparable scores. BIB002 uses a believability index to assess agents in a multiplayer game. The index is calculated from ratings on a scale of 1 to 5 of how human-like the performance appears. The ratings are performed by a number of independent observers. BIB003 conducted a qualitative study in the form of an online Turing test. The users are presented with a number of pairs of play-through footage of a Mario game. For each pair, the user has to decide which footage belonged to a human player and which belonged to an AI controller. A similar evaluation is used in for the same task. In BIB003 , the AI controllers were distinguishable from human players. However, agents that are designed to imitate human players were more believable than agents designed to perform well in the game. In addition to the Turing test, a qualitative evaluation is performed to test how closely the trained agents resemble human players. One drawback with this approach is that you can only compare an agent's behavior to one human demonstration, as human demonstrations can vary significantly among themselves. It is clear that quantitative and qualitative analyses focus on different aspects of the agent's performance and thus have different requirements. Table II summarizes points that help decide whether quantitative or qualitative evaluation is more appropriate for a given task. In addition to evaluating the quality of an agent at performing a task, it is important for researchers to evaluate the value of the various self-improvement methods available. In transfer learning, three measures are used to quantify the advantage of using prior knowledge from other tasks  . The first is to compare a model that has learned only from previous tasks to an untrained one. This measure demonstrates if the skills learned in the source task are of any use in the target task, without any subsequent learning. The second measure compares the time to completely learn the target task from scratch against a learner initialized using transfer learning. This is useful to find out if transfer learning can speed up the learning process for the target task. And finally the third measure is used to evaluate the performance of the initialized agent after subsequently learning from the target task against one that learned the target task from scratch. Comparison of the final performance represents the ability of transfer learning to improve over a fully trained learner. A graph plotting the performance of an agent learning from scratch against one that uses transfer learning over time can illustrate all three measures  BIB001 .