Hierarchical Reinforcement Learning: A Survey <s> CONCLUSION & FUTURE WORK <s> Teleoperated rescue robots designed to explore disaster scenes and find victims face serious limitations due to the cluttered nature of the environments as well as the rescue operators becoming stressed and disoriented in these scenes. An alternative to using teleoperated control is to develop fully autonomous controllers for rescue robots. However, these robots are also not capable of traversing these complex unpredictable environments. In order to address the limitations of both teleoperation and fully autonomous robotic control for urban search and rescue (USAR) environments, semi-autonomous controllers can be developed to allow task sharing and cooperation between a human operator and a robot. In this paper, a unique Hierarchical Reinforcement Learning (HRL) based semi-autonomous control architecture is proposed. The architecture provides the robot with the ability to learn and make decisions regarding which rescue tasks, exploration or victim identification, should be carried out at a given time and whether an autonomous robot or a human controlled robot can perform these tasks more quickly and efficiently without compromising the safety of the victims, rescue workers and the rescue robot. Preliminary experiments presented here evaluate the performance of the proposed HRL control approach for a rescue robot in an unknown cluttered USAR environment. <s> BIB001 </s> Hierarchical Reinforcement Learning: A Survey <s> CONCLUSION & FUTURE WORK <s> Through web services composition, distributed applications and enterprise business processes can be integrated by individual service components developed independently. In this chapter, we concentrate on the optimization problems of dynamic web service composition, and our goal is to find an optimal composite policy. We introduce a hierarchical reinforcement learning technique, i.e., a continuous-time unified MAXQ algorithm, to solve large-scale web service composition problems in the context of continuous-time semi-Markov decision process (SMDP) model under either average- or discounted-cost criteria. Finally, this proposed algorithm is tested in a simulation, and the experimental results show that it has better optimization performance than the flat Q-learning. <s> BIB002 </s> Hierarchical Reinforcement Learning: A Survey <s> CONCLUSION & FUTURE WORK <s> In this paper we intend to study the possibility to improve the performance of the Q-Learning algorithm, by automatically finding subgoals and making better use of the acquired knowledge. This research explores a method that allows an agent to gather information about sequences of states that lead to a goal, detect classes of common sequences and introduce the states at the end of these sequences as subgoals. We use the taxiproblem (a standard in Hierarchical Reinforcement Learning literature) and conclude that, even though this problem's scale is relatively small, in most of the cases subgoals do improve the learning speed, achieving relatively good results faster than standard Q-Learning. We propose a specific iteration interval as the most appropriate to insert subgoals in the learning process. We also found that early adoption of subgoals may lead to suboptimal learning. The extension to more challenging problems is an interesting subject for future work. <s> BIB003
Reinforcement Learning (RL) plays a significant role in the area of Machine Learning and AI. HRL focuses on decomposing the RL problems into sub-problems where solving each sub-problem independently is much easier and powerful than solving the entire problem. A review of the state-of-the-art of Hierarchical Reinforcement Learning has been investigated. Different research areas with different problems based on HRL have been surveyed within this paper such as: rescue robots in cluttered environments, computer games, course scheduling, motion planning in mobile robotics, web service composition, path clustering, multi-robot cooperation and intelligent robots. While working on this survey, some ideas have been emerged and can be pursued as a future work; the ideas that require more attention from researchers who are interested in the HRL field. Such ideas could be summarized as per the following:  Will multi-robot cooperation be an effective way to search for victims in cluttered USAR environments? As to compare with the proposed solutions by  , BIB001 . Moreover, these solutions have been implemented in small-scale environments so further research needs to focus on large-scale environments to test their efficiency.  How multi-robot cooperation will support the problems of web-service compositions? As to compare with BIB002 .  BIB003 focused on smaller discrete RL problems through the use of path clustering. Further research may focus on larger continues RL problems.  BIB002 compares both MAXQ and Q-learning algorithms in the area of web-service compositions. Further work could compare the two algorithms in robots race and see which one is going to learn and reach the goal state faster?