Hierarchical Reinforcement Learning: A Survey <s> B. Hierarchical Reinforcement Learning <s> In this paper, we investigate the use of hierarchical reinforcement learning (HRL) to speed up the acquisition of cooperative multi-agent tasks. We introduce a hierarchical multi-agent reinforcement learning (RL) framework, and propose a hierarchical multi-agent RL algorithm called Cooperative HRL. In this framework, agents are cooperative and homogeneous (use the same task decomposition). Learning is decentralized, with each agent learning three interrelated skills: how to perform each individual subtask, the order in which to carry them out, and how to coordinate with other agents. We define cooperative subtasks to be those subtasks in which coordination among agents significantly improves the performance of the overall task. Those levels of the hierarchy which include cooperative subtasks are called cooperation levels. A fundamental property of the proposed approach is that it allows agents to learn coordination faster by sharing information at the level of cooperative subtasks, rather than attempting to learn coordination at the level of primitive actions. We study the empirical performance of the Cooperative HRL algorithm using two testbeds: a simulated two-robot trash collection task, and a larger four-agent automated guided vehicle (AGV) scheduling problem. We compare the performance and speed of Cooperative HRL with other learning algorithms, as well as several well-known industrial AGV heuristics. We also address the issue of rational communication behavior among autonomous agents in this paper. The goal is for agents to learn both action and communication policies that together optimize the task given a communication cost. We extend the multi-agent HRL framework to include communication decisions and propose a cooperative multi-agent HRL algorithm called COM-Cooperative HRL. In this algorithm, we add a communication level to the hierarchical decomposition of the problem below each cooperation level. Before an agent makes a decision at a cooperative subtask, it decides if it is worthwhile to perform a communication action. A communication action has a certain cost and provides the agent with the actions selected by the other agents at a cooperation level. We demonstrate the efficiency of the COM-Cooperative HRL algorithm as well as the relation between the communication cost and the learned communication policy using a multi-agent taxi problem. <s> BIB001 </s> Hierarchical Reinforcement Learning: A Survey <s> B. Hierarchical Reinforcement Learning <s> This paper presents a new method which uses a connect-based thought to automatically discover subgoals in a dynamic environment. We argue that the states which are not only visited frequently in the whole space but also have a relative high visiting frequency in their neighboring regions performing a critical role in connecting the neighbor states are subgoals, and then propose a novel algorithm for identifying them by considering one state who has a high sum of the relative exceeding frequencies from all its neighbor states as a subgoal. Most earlier methods actually discover a frequent visited region and then filter and select the critical states in this region, but our method directly discovers the critical states of a frequent visited region in the space without generating excessive useless candidates. Combining the global and local perspectives is a key property of our algorithm and the one that differentiates it from most of existing works in this area. Experiments show that this simple and robust approach detects subgoals quickly and correctly. <s> BIB002 </s> Hierarchical Reinforcement Learning: A Survey <s> B. Hierarchical Reinforcement Learning <s> Hierarchical reinforcement learning methods have not been able to simultaneously abstract and reuse subtasks with discounted value functions. The contribution of this paper is to introduce two completion functions that jointly decompose the value function hierarchically to solve this problem. The significance of this result is that the benefits of hierarchical reinforcement learning can be extended to discounted value functions and to continuing (infinite horizon) reinforcement learning problems. This paper demonstrates the method with the an algorithm that discovers subtasks automatically. An example is given where the optimum policy requires a subtask never to terminate. <s> BIB003 </s> Hierarchical Reinforcement Learning: A Survey <s> B. Hierarchical Reinforcement Learning <s> Transfer learning seeks to leverage previously learned tasks to achieve faster learning in a new task. In this paper, we consider transfer learning in the context of related but distinct Reinforcement Learning (RL) problems. In particular, our RL problems are derived from Semi-Markov Decision Processes (SMDPs) that share the same transition dynamics but have different reward functions that are linear in a set of reward features. We formally define the transfer learning problem in the context of RL as learning an efficient algorithm to solve any SMDP drawn from a fixed distribution after experiencing a finite number of them. Furthermore, we introduce an online algorithm to solve this problem, Variable-Reward Reinforcement Learning (VRRL), that compactly stores the optimal value functions for several SMDPs, and uses them to optimally initialize the value function for a new SMDP. We generalize our method to a hierarchical RL setting where the different SMDPs share the same task hierarchy. Our experimental results in a simplified real-time strategy domain show that significant transfer learning occurs in both flat and hierarchical settings. Transfer is especially effective in the hierarchical setting where the overall value functions are decomposed into subtask value functions which are more widely amenable to transfer across different SMDPs. <s> BIB004 </s> Hierarchical Reinforcement Learning: A Survey <s> B. Hierarchical Reinforcement Learning <s> Hierarchical finite state machine (HFSM) has proven to be a powerful tool for controlling Non-player characters (NPCs) in computer games due to its flexibility and modularity. For most implementations, however, it is often the case that the control details at all levels are hand-coded. As a result, the development process is often time intensive and error prone. In this paper, we explore the use of a hierarchical reinforcement learning approach, based on Hierarchies of Abstract Machines (HAMs), to help overcome some of these limitations. We analyse in detail both HAMs and its use for designing HFSM, propose two HAMs-related machines, and make a preliminary experiment: applying HAMs to design NPCs' behavior and implementing it in Quake2. The result shows that this method can satisfy the need for controlling NPC and has faster convergence speed than flat reinforcement learning. <s> BIB005 </s> Hierarchical Reinforcement Learning: A Survey <s> B. Hierarchical Reinforcement Learning <s> Teleoperated rescue robots designed to explore disaster scenes and find victims face serious limitations due to the cluttered nature of the environments as well as the rescue operators becoming stressed and disoriented in these scenes. An alternative to using teleoperated control is to develop fully autonomous controllers for rescue robots. However, these robots are also not capable of traversing these complex unpredictable environments. In order to address the limitations of both teleoperation and fully autonomous robotic control for urban search and rescue (USAR) environments, semi-autonomous controllers can be developed to allow task sharing and cooperation between a human operator and a robot. In this paper, a unique Hierarchical Reinforcement Learning (HRL) based semi-autonomous control architecture is proposed. The architecture provides the robot with the ability to learn and make decisions regarding which rescue tasks, exploration or victim identification, should be carried out at a given time and whether an autonomous robot or a human controlled robot can perform these tasks more quickly and efficiently without compromising the safety of the victims, rescue workers and the rescue robot. Preliminary experiments presented here evaluate the performance of the proposed HRL control approach for a rescue robot in an unknown cluttered USAR environment. <s> BIB006 </s> Hierarchical Reinforcement Learning: A Survey <s> B. Hierarchical Reinforcement Learning <s> A hierarchical reinforcement learning method based on heuristic reward function is proposed to solve the problem of “curse of dimensionality”, that is the states space will grow exponentially in the number of features, and low convergence speed. The method can reduce state spaces greatly and can enhance the speed of the study. Choose actions with favorable purpose and efficiency so as to optimize reward function and enhance convergence speed. Apply this method to the Tetris game; the experiment result shows that the method can partly solve the “curse of dimensionality” and can enhance the convergence speed prominent. <s> BIB007 </s> Hierarchical Reinforcement Learning: A Survey <s> B. Hierarchical Reinforcement Learning <s> Temporal abstraction and task decomposition drastically reduce the search space for planning and control, and are fundamental to making complex tasks amenable to learning. In the context of reinforcement learning, temporal abstractions are studied within the paradigm of hierarchical reinforcement learning. <s> BIB008 </s> Hierarchical Reinforcement Learning: A Survey <s> B. Hierarchical Reinforcement Learning <s> SUMMARY Human behavior displays hierarchical structure: simpleactionscohereinto subtask sequences,which work together to accomplish overall task goals. Although the neural substrates of such hierarchy have been the target of increasing research, they remain poorly understood. We propose that the computations supporting hierarchical behavior may relate to those in hierarchical reinforcement learning (HRL), a machine-learning framework that extends reinforcement-learning mechanisms into hierarchical domains. To test this, we leveraged a distinctive predictionarisingfromHRL.Inordinaryreinforcement learning,rewardpredictionerrorsarecomputedwhen there is an unanticipated change in the prospects for accomplishing overall task goals. HRL entails that prediction errors should also occur in relation to task subgoals. In threeneuroimaging studieswe observed neural responses consistent with such subgoalrelated reward prediction errors, within structures previously implicated in reinforcement learning. The results reported support the relevance of HRL to the neural processes underlying hierarchical behavior. <s> BIB009 </s> Hierarchical Reinforcement Learning: A Survey <s> B. Hierarchical Reinforcement Learning <s> Language generators in situated domains face a number of content selection, utterance planning and surface realisation decisions, which can be strictly interdependent. We therefore propose to optimise these processes in a joint fashion using Hierarchical Reinforcement Learning. To this end, we induce a reward function for content selection and utterance planning from data using the PARADISE framework, and suggest a novel method for inducing a reward function for surface realisation from corpora. It is based on generation spaces represented as Bayesian Networks. Results in terms of task success and human-likeness suggest that our unified approach performs better than a baseline optimised in isolation or a greedy or random baseline. It receives human ratings close to human authors. <s> BIB010 </s> Hierarchical Reinforcement Learning: A Survey <s> B. Hierarchical Reinforcement Learning <s> The hierarchical structure of human and animal behavior has been of critical interest in neuroscience for many years. Yet understanding the neural processes that give rise to such structure remains an open challenge. In recent research, a new perspective on hierarchical behavior has begun to take shape, inspired by ideas from machine learning, and in particular the framework of hierarchical reinforcement learning. Hierarchical reinforcement learning builds on traditional reinforcement learning mechanisms, extending them to accommodate temporally extended behaviors or subroutines. The resulting computational paradigm has begun to influence both theoretical and empirical work in neuroscience, conceptually aligning the study of hierarchical behavior with research on other aspects of learning and decision making, and giving rise to some thought-provoking new findings. <s> BIB011 </s> Hierarchical Reinforcement Learning: A Survey <s> B. Hierarchical Reinforcement Learning <s> In this paper we intend to study the possibility to improve the performance of the Q-Learning algorithm, by automatically finding subgoals and making better use of the acquired knowledge. This research explores a method that allows an agent to gather information about sequences of states that lead to a goal, detect classes of common sequences and introduce the states at the end of these sequences as subgoals. We use the taxiproblem (a standard in Hierarchical Reinforcement Learning literature) and conclude that, even though this problem's scale is relatively small, in most of the cases subgoals do improve the learning speed, achieving relatively good results faster than standard Q-Learning. We propose a specific iteration interval as the most appropriate to insert subgoals in the learning process. We also found that early adoption of subgoals may lead to suboptimal learning. The extension to more challenging problems is an interesting subject for future work. <s> BIB012
Hierarchical Reinforcement Learning (HRL) refers to the notion in which RL problem is decomposed into subproblems (sub-tasks) where solving each of which will be more powerful than solving the entire problem BIB012 , BIB008 , BIB005 and  , BIB004 . BIB011 and BIB009 has defined HRL as the set of http://dx.doi.org/10.12785/ijcds/040207 http://journals.uob.edu.bh computational techniques that enlarge RL procedures in order to involve temporarily abstract actions. The hierarchical decomposition has some advantages such as: reducing the sub-problem's computational complexity, managing the sub-problems individually will maximize its reusability which in turn will speed up the learning process BIB012 . HRL techniques use several forms of abstractions that have the ability to handle the exponentially increasing number of parameters which are required to be learned specifically in big problems for effectively reducing the search space which in turn allows the agent to determine the optimal solution BIB006 . HRL is one of the well-known methods to resolve the problem of "Curse of Dimensionality" BIB002 . BIB007 stated that a welldesigned reward function along with HRL can decrease the number of impractical acts of exploration which in turn allows the agent to interact easily and quickly with the environment. BIB010 presented that utterance planning and content selection in Natural Language Generation (NLG) can be optimized via HRL along with Bayesian Networks. Various HRL models are available like MAXQ, Hierarchical abstract machines (HAMs), ALisp and options BIB003 , BIB001 ; the models that scale RL to large state spaces problems by decomposing them into subproblems.