Hierarchical Reinforcement Learning: A Survey <s> INTRODUCTION <s> Reinforcement learning is bedeviled by the curse of dimensionality: the number of parameters to be learned grows exponentially with the size of any compact encoding of a state. Recent attempts to combat the curse of dimensionality have turned to principled ways of exploiting temporal abstraction, where decisions are not required at each step, but rather invoke the execution of temporally-extended activities which follow their own policies until termination. This leads naturally to hierarchical control architectures and associated learning algorithms. We review several approaches to temporal abstraction and hierarchical organization that machine learning researchers have recently developed. Common to these approaches is a reliance on the theory of semi-Markov decision processes, which we emphasize in our review. We then discuss extensions of these ideas to concurrent activities, multiagent coordination, and hierarchical memory for addressing partial observability. Concluding remarks address open challenges facing the further development of reinforcement learning in a hierarchical setting. <s> BIB001 </s> Hierarchical Reinforcement Learning: A Survey <s> INTRODUCTION <s> This research integrates rigorous methods of reinforcement learning (RL) and control engineering with a behavioral (ethology) approach to the agent technology. The main outcome is a hybrid architecture for intelligent autonomous agents targeted to the Artificial Life like environments. The architecture adopts several biology concepts and shows that they can provide robust solutions to some areas. The resulting agents perform from primitive behaviors, simple goal directed behaviors, to complex planning. The agents are fully autonomous through environment feedback evaluating internal agent state and motivates the agent to perform behaviors that return the agent towards optimal conditions. This principle is typical to animals. Learning and control is realized by multiple RL controllers working in a hierarchy of Semi Markov Decision Processes (SMDP). Used model free Q(lambda) learning works online, the agents gain experiences during interaction with the environment. The decomposition of the root SMDP into hierarchy is automated as opposed to the conventional methods that are manual. The agents assess utility of the behavior and provide rewards to RL controller as opposed to the conventional RL methods where the rewards-situations map is defined by the designer upfront. The resulting learning algorithm converges to a recursively optimal solution with probability 1. Agent behavior is continuously optimized according to the distance from the agentpsilas optimal conditions. <s> BIB002 </s> Hierarchical Reinforcement Learning: A Survey <s> INTRODUCTION <s> This paper proposes applying Option-based hierarchical reinforcement learning algorithm to timetable scheduling strategy learning in colleges and universities so as to enhance the performance of traditional reinforcement learning course-scheduling algorithm. From experimentation and application in course-scheduling system, it can be seen that Option-based hierarchical reinforcement learning course scheduling algorithm has a better effect than traditional reinforcement learning course-scheduling algorithm. <s> BIB003 </s> Hierarchical Reinforcement Learning: A Survey <s> INTRODUCTION <s> A hierarchical reinforcement learning method based on heuristic reward function is proposed to solve the problem of “curse of dimensionality”, that is the states space will grow exponentially in the number of features, and low convergence speed. The method can reduce state spaces greatly and can enhance the speed of the study. Choose actions with favorable purpose and efficiency so as to optimize reward function and enhance convergence speed. Apply this method to the Tetris game; the experiment result shows that the method can partly solve the “curse of dimensionality” and can enhance the convergence speed prominent. <s> BIB004 </s> Hierarchical Reinforcement Learning: A Survey <s> INTRODUCTION <s> Language generators in situated domains face a number of content selection, utterance planning and surface realisation decisions, which can be strictly interdependent. We therefore propose to optimise these processes in a joint fashion using Hierarchical Reinforcement Learning. To this end, we induce a reward function for content selection and utterance planning from data using the PARADISE framework, and suggest a novel method for inducing a reward function for surface realisation from corpora. It is based on generation spaces represented as Bayesian Networks. Results in terms of task success and human-likeness suggest that our unified approach performs better than a baseline optimised in isolation or a greedy or random baseline. It receives human ratings close to human authors. <s> BIB005 </s> Hierarchical Reinforcement Learning: A Survey <s> INTRODUCTION <s> This paper presents a hierarchical representation policy iteration (HRPI) algorithm. It is based on the method of state space decomposition implemented by introducing a binary tree. Combining the RPI algorithm with the state space decomposition method, the HRPI algorithm is proposed. In HRPI, the state space is decomposed into multiple sub-spaces according to an approximate value function, then the local policies are estimated on each sub-space and finally the global near-optimal policy is obtained by combining these local policies. The simulation results indicate that the proposed method has better performance compared to the conventional RPI algorithm. <s> BIB006 </s> Hierarchical Reinforcement Learning: A Survey <s> INTRODUCTION <s> In applying reinforcement learning (RL) to multi-robot control, the size of the learning state space easily explodes because the state space has a high dimension. Hierarchical reinforcement learning (HRL) is one of the most practical approaches to solve the problem; however, automatically decomposing a plain MDP state space into sub-spaces has not been studied thoroughly enough to be applied to practical robotics problems. We propose a method that automatically forms hierarchical sub-tasks for multi-robot delivery missions. The method executes sub-task decomposition and the learning process in a step-by-step manner, by widening the robot's range of movements around the load and gradually decreasing the domain of the load position. The method automatically detects the state in which cooperative motion among the robots is needed for them to accomplish the mission. The performance of the method is demonstrated by simulations. <s> BIB007
Reinforcement Learning (RL) has been an interesting research field in the community of Machine Learning and AI that received many attentions from the fields of operations research due to its self-adaptation and selflearning BIB006 . RL algorithms work on maximizing the agent learning while interacting with its environment directly BIB001 . Hierarchical Reinforcement Learning (HRL) works on decomposing the RL problem into sub-problems where solving each of which will be more powerful than solving the entire problem BIB007 . In the recent years, BIB003 , BIB004 and BIB005 stated that the problem of "Curse of Dimensionality" (which is the exponential growth of memory requirements with the number of state variables) has been solved via HRL. The RL then works on reducing dimensionality through decomposing it into several levels. HRL helps to overcome the agent-learning complexities that are considered as one of the typical issues in the learning environments BIB002 . Different HRL-based domains have been investigated within this work. Different problems in such different domains along with some proposed solutions have been addressed. The paper is organized as follows: section 2 demonstrates a background on RL, HRL and Q-learning. Section 3 addresses the main contribution in the area of HRL. Section 4 presents the conclusion and the ideas that have been emerged while conducting this research.