Hierarchical Reinforcement Learning: A Survey <s> B. HRL in Computer Games <s> Hierarchical finite state machine (HFSM) has proven to be a powerful tool for controlling Non-player characters (NPCs) in computer games due to its flexibility and modularity. For most implementations, however, it is often the case that the control details at all levels are hand-coded. As a result, the development process is often time intensive and error prone. In this paper, we explore the use of a hierarchical reinforcement learning approach, based on Hierarchies of Abstract Machines (HAMs), to help overcome some of these limitations. We analyse in detail both HAMs and its use for designing HFSM, propose two HAMs-related machines, and make a preliminary experiment: applying HAMs to design NPCs' behavior and implementing it in Quake2. The result shows that this method can satisfy the need for controlling NPC and has faster convergence speed than flat reinforcement learning. <s> BIB001 </s> Hierarchical Reinforcement Learning: A Survey <s> B. HRL in Computer Games <s> In this work, we analyze and improve upon reinforcement learning techniques used to build agents that can learn to play Infinite Mario, an action game. We extend the object-oriented representation by introducing the concept of object classes which can be effectively used to constrain state spaces. We then use this representation combined with the hierarchical reinforcement learning model as a learning framework. We also extend the idea of hierarchical RL by designing a hierarchy in action selection using domain specific knowledge. With the help of experimental results, we show that this approach facilitates faster and efficient learning for this domain. <s> BIB002 </s> Hierarchical Reinforcement Learning: A Survey <s> B. HRL in Computer Games <s> Nowadays, video games conforms a huge industry that is always developing new technology. In particular, artificial intelligence techniques have been used broadly in the well-known non-player characters (NPC) given the opportunity to users to feel video games more real. This paper proposes the usage of the MaxQ-Q hierarchical reinforcement learning algorithm in non-player characters in order to increase the experience of the user in terms of naturalness. A case study of an NPC with the proposed artificial intelligence based algorithm in a first personal shooter video game was developed. Experimental results show that this implementation improves naturalness from the user’s point of view. In addition, the proposed MaxQ-Q based algorithm in NPCs allow to programmers a robust way to give artificial intelligence to them. <s> BIB003
Computer games are one of the hot topics for research in AI and machine learning. One of the issues in computer games that attract researchers to work on is the NonPlayer Characters (NPCs) behaviors due to their complexity and difficulty in to be represented by typical finite state machines. The control details of NPCs at all stages are commonly hand-coded; the reason that makes the development task consumes more time and exposed for errors. To overcome these limitations, HRL has been used based on the Hierarchies of Abstract Machines (HAMs). Through applying the proposed solution, system designers can determine points within the program itself where they don't care about how the code will be written while it's determined through the robot's learning process. Experiments have been conducted to test the efficiency of the proposed solution under the Quake2UR (which performs as 3D Game Server) and ALisp system (which performs as a client). Results revealed that the proposed solution was very flexible and satisfies the need for controlling NPCs easily BIB001 . Similarly, BIB003 has proposed the MaxQ-Q HRL algorithm in the NPCs in order to enhance the user's experience and to improve the natural humanness while interacting with computer games. Experiments were performed using "Capture the Flag" strategic game. A comparison was made between the NPCs based on Finite State Machines (FSM) and the NPCs based on MaxQ-Q through the game. Results indicated that NPCs based on MaxQ-Q HRL are 52% much better than NPCs based on FSM. http://journals.uob.edu.bh Moreover, the Infinite Mario game is one of the interesting action games that got popularity in the area of AI and machine learning. The domain of this game is very complex and contains huge state-action spaces. BIB002 has integrated the HRL along with the object-oriented representation in order to reduce the state-action spaces in the game domain. Accordingly, involving HRL has increased the agent-learning performance.