Hierarchical Reinforcement Learning: A Survey <s> D. HRL Approach for Motion Planning in Mobile Robotics <s> The motion planning task for a mobile robot aims to generate a free-collision path from an initial point to a target point. This task may be highly complex because it requires a complete knowledge of the robot's environment. In this paper an option-based hierarchical learning approach is proposed to this problem in which basic behaviors are applied in order to accomplish the robot motion planning task. Each behavior is independently learned by the robot in the learning phase. Afterward, the robot learns to coordinate these basic behaviors to resolve the motion planning task. The application of the learning approach is validated with robot motion planning tasks in simulation as well as in an experimental environment. The results show a solution to the motion planning problem that can be highly successful in new unknown environments. <s> BIB001 </s> Hierarchical Reinforcement Learning: A Survey <s> D. HRL Approach for Motion Planning in Mobile Robotics <s> In this paper we intend to study the possibility to improve the performance of the Q-Learning algorithm, by automatically finding subgoals and making better use of the acquired knowledge. This research explores a method that allows an agent to gather information about sequences of states that lead to a goal, detect classes of common sequences and introduce the states at the end of these sequences as subgoals. We use the taxiproblem (a standard in Hierarchical Reinforcement Learning literature) and conclude that, even though this problem's scale is relatively small, in most of the cases subgoals do improve the learning speed, achieving relatively good results faster than standard Q-Learning. We propose a specific iteration interval as the most appropriate to insert subgoals in the learning process. We also found that early adoption of subgoals may lead to suboptimal learning. The extension to more challenging problems is an interesting subject for future work. <s> BIB002
Motion planning task is one of the interesting tasks in mobile robotics that looks for generating a free-collision path from the initial point to the goal point for the robot. [10] applied RL in order to avoid all the obstacles in mobile robotics through the use of Neural Networks, However this became an old technique where BIB001 proposed an option-based HRL in which basic behaviors are used. Each behavior is individually learned in the learning process; this allows the robot to organize all the basic behaviors to solve the problem of the motion planning. Semi-Markov Q-learning has been used to estimate the state-option function values Q(s, o) via choosing an option o in the state s based on the policy μ. After implementing o, the final state s' along with the Qvalue will be updated based on the equation: Where  denotes the discount rate, α denotes the learning rate and k refers to number of steps between s and s'. Experimental tests have been implemented in simulation. Results revealed that the proposed algorithm has the ability to work effectively in unknown environments as well as to avoid all the encountered obstacles by the robot in the motion planning task without the use of Neural Networks. BIB002 intended to resolve small and medium scales RL problems through the use of path clustering in order to enable its hierarchical decomposition. Moreover, how to enhance the Q-learning algorithm performance via automatically finding sub-goals and making better usage of the knowledge acquired. HRL path clustering method has been proposed which allows the robot to acquire the knowledge about the states' sequences which lead to the goal and introduce those states at the end of the sequences as sub-goals. Taxi-problem (as one of the standards in RL and is being used for testing the HRL solutions) has been used. In this problem, sub-goals enhance the learning speed by achieving good results faster than the traditional Q-learning due to the fact that the problem scale is very small. It has been proposed to insert the sub-goals into the learning process. Results revealed that the early involvement of sub-goals will achieve a sub-optimal learning.