Hierarchical Reinforcement Learning: A Survey <s> A. Reinforcement Learning <s> Reinforcement learning is bedeviled by the curse of dimensionality: the number of parameters to be learned grows exponentially with the size of any compact encoding of a state. Recent attempts to combat the curse of dimensionality have turned to principled ways of exploiting temporal abstraction, where decisions are not required at each step, but rather invoke the execution of temporally-extended activities which follow their own policies until termination. This leads naturally to hierarchical control architectures and associated learning algorithms. We review several approaches to temporal abstraction and hierarchical organization that machine learning researchers have recently developed. Common to these approaches is a reliance on the theory of semi-Markov decision processes, which we emphasize in our review. We then discuss extensions of these ideas to concurrent activities, multiagent coordination, and hierarchical memory for addressing partial observability. Concluding remarks address open challenges facing the further development of reinforcement learning in a hierarchical setting. <s> BIB001 </s> Hierarchical Reinforcement Learning: A Survey <s> A. Reinforcement Learning <s> We consider the problem of multi-task reinforcement learning, where the agent needs to solve a sequence of Markov Decision Processes (MDPs) chosen randomly from a fixed but unknown distribution. We model the distribution over MDPs using a hierarchical Bayesian infinite mixture model. For each novel MDP, we use the previously learned distribution as an informed prior for modelbased Bayesian reinforcement learning. The hierarchical Bayesian framework provides a strong prior that allows us to rapidly infer the characteristics of new environments based on previous environments, while the use of a nonparametric model allows us to quickly adapt to environments we have not encountered before. In addition, the use of infinite mixtures allows for the model to automatically learn the number of underlying MDP components. We evaluate our approach and show that it leads to significant speedups in convergence to an optimal policy after observing only a small number of tasks. <s> BIB002 </s> Hierarchical Reinforcement Learning: A Survey <s> A. Reinforcement Learning <s> Reinforcement learning provides both qualitative and quantitative frameworks for understanding and modeling adaptive decision-making in the face of rewards and punishments. Here we review the latest dispatches from the forefront of this field, and map out some of the territories where lie monsters. <s> BIB003 </s> Hierarchical Reinforcement Learning: A Survey <s> A. Reinforcement Learning <s> The field of reinforcement learning has greatly influenced the neuroscientific study of conditioning. This article provides an introduction to reinforcement learning followed by an examination of the successes and challenges using reinforcement learning to understand the neural bases of conditioning. Successes reviewed include (1) the mapping of positive and negative prediction errors to the firing of dopamine neurons and neurons in the lateral habenula, respectively; (2) the mapping of model-based and model-free reinforcement learning to associative and sensorimotor cortico-basal ganglia-thalamo-cortical circuits, respectively; and (3) the mapping of actor and critic to the dorsal and ventral striatum, respectively. Challenges reviewed consist of several behavioral and neural findings that are at odds with standard reinforcement-learning models, including, among others, evidence for hyperbolic discounting and adaptive coding. The article suggests ways of reconciling reinforcement-learning models with many of the challenging findings, and highlights the need for further theoretical developments where necessary. Additional information related to this study may be downloaded from http://cabn.psychonomic-journals.org/content/supplemental. <s> BIB004 </s> Hierarchical Reinforcement Learning: A Survey <s> A. Reinforcement Learning <s> This paper proposes applying Option-based hierarchical reinforcement learning algorithm to timetable scheduling strategy learning in colleges and universities so as to enhance the performance of traditional reinforcement learning course-scheduling algorithm. From experimentation and application in course-scheduling system, it can be seen that Option-based hierarchical reinforcement learning course scheduling algorithm has a better effect than traditional reinforcement learning course-scheduling algorithm. <s> BIB005 </s> Hierarchical Reinforcement Learning: A Survey <s> A. Reinforcement Learning <s> This article addresses the problem of scalable optimization for spatially-aware dialogue systems. These kinds of systems must perceive, reason, and act about the spatial environment where they are embedded. We formulate the problem in terms of Semi-Markov Decision Processes and propose a hierarchical reinforcement learning approach to optimize subbehaviors rather than full behaviors. Because of the vast number of policies that are required to control the interaction in a dynamic environment (e.g., a dialogue system assisting a user to navigate in a building from one location to another), our learning approach is based on two stages: (a) the first stage learns low-level behavior, in advance; and (b) the second stage learns high-level behavior, in real time. For such a purpose we extend an existing algorithm in the literature of reinforcement learning in order to support reusable policies and therefore to perform fast learning. We argue that our learning approach makes the problem feasible, and we report on a novel reinforcement learning dialogue system that performs a joint optimization between dialogue and spatial behaviors. Our experiments, using simulated and real environments, are based on a text-based dialogue system for indoor navigation. Experimental results in a realistic environment reported an overall user satisfaction result of 89p, which suggests that our proposed approach is attractive for its application in real interactions as it combines fast learning with adaptive and reasonable behavior. <s> BIB006 </s> Hierarchical Reinforcement Learning: A Survey <s> A. Reinforcement Learning <s> In this paper, we propose a novel hierarchical path planning algorithm for mobile robots based on A* and reinforcement learning (RL) with the structure of two layers. In the first layer, we adopt the A* search algorithm to plan a geometric path and select several points as sub-target points for the planning of the next stage. In the second layer, a local path planning algorithm based on an approximate RL method called Least Square Policy Iteration (LSPI) is used to find a kinematically feasible path with these sub-targets. After learning, the local path planner in the second layer has good generalization performance. The path obtained by the proposed algorithm is smooth and safe for executing. Simulations have been carried out and the results demonstrate the validity of the proposed scheme. <s> BIB007 </s> Hierarchical Reinforcement Learning: A Survey <s> A. Reinforcement Learning <s> The motion planning task for a mobile robot aims to generate a free-collision path from an initial point to a target point. This task may be highly complex because it requires a complete knowledge of the robot's environment. In this paper an option-based hierarchical learning approach is proposed to this problem in which basic behaviors are applied in order to accomplish the robot motion planning task. Each behavior is independently learned by the robot in the learning phase. Afterward, the robot learns to coordinate these basic behaviors to resolve the motion planning task. The application of the learning approach is validated with robot motion planning tasks in simulation as well as in an experimental environment. The results show a solution to the motion planning problem that can be highly successful in new unknown environments. <s> BIB008 </s> Hierarchical Reinforcement Learning: A Survey <s> A. Reinforcement Learning <s> In this paper we intend to study the possibility to improve the performance of the Q-Learning algorithm, by automatically finding subgoals and making better use of the acquired knowledge. This research explores a method that allows an agent to gather information about sequences of states that lead to a goal, detect classes of common sequences and introduce the states at the end of these sequences as subgoals. We use the taxiproblem (a standard in Hierarchical Reinforcement Learning literature) and conclude that, even though this problem's scale is relatively small, in most of the cases subgoals do improve the learning speed, achieving relatively good results faster than standard Q-Learning. We propose a specific iteration interval as the most appropriate to insert subgoals in the learning process. We also found that early adoption of subgoals may lead to suboptimal learning. The extension to more challenging problems is an interesting subject for future work. <s> BIB009
Reinforcement learning (RL) is one of the machine learning areas in which an agent has to interact with its environment in order to achieve a goal (as in Figure 1 ). RL based on the structure of Markov Decision Processes (MDPs); a reliable structure for the agent learning while interacting with its environment in order to receive rewards and drawbacks BIB002 , BIB007 , BIB006 . The essential elements of RL are the states, actions and reinforcements BIB004 . Via the agent's sensors, the agent recognizes the environment and implements actions (according to a policy) in which leads to changes in the environment. According to these changes, the agent obtains rewards based on the taken actions BIB008 , BIB003 . RL improves strategy through the learning via trial and error by interacting with the environment and recognize the best actions at each state in order to reach the goal and gain the best rewards BIB005 , BIB009 . RL tries to find the best policy that increases the total reward. BIB001 indicated that RL algorithms work on how the agent can learn to estimate an optimal strategy while to interact with its environment. 