Hierarchical Reinforcement Learning: A Survey <s> C. Q-Learning <s> The development of mechanisms to understand and model the expected behaviour of multiagent learners is becoming increasingly important as the area rapidly find application in a variety of domains. In this paper we present a framework to model the behaviour of Q-learning agents using the e-greedy exploration mechanism. For this, we analyse a continuous-time version of the Q-learning update rule and study how the presence of other agents and the e-greedy mechanism affect it. We then model the problem as a system of difference equations which is used to theoretically analyse the expected behaviour of the agents. The applicability of the framework is tested through experiments in typical games selected from the literature. <s> BIB001 </s> Hierarchical Reinforcement Learning: A Survey <s> C. Q-Learning <s> The motion planning task for a mobile robot aims to generate a free-collision path from an initial point to a target point. This task may be highly complex because it requires a complete knowledge of the robot's environment. In this paper an option-based hierarchical learning approach is proposed to this problem in which basic behaviors are applied in order to accomplish the robot motion planning task. Each behavior is independently learned by the robot in the learning phase. Afterward, the robot learns to coordinate these basic behaviors to resolve the motion planning task. The application of the learning approach is validated with robot motion planning tasks in simulation as well as in an experimental environment. The results show a solution to the motion planning problem that can be highly successful in new unknown environments. <s> BIB002
Q-learning is one of the RL algorithms that has been successfully used in many domains such as: face recognition, simple toys, web-based education and many others BIB001 . Q-learning tries to find an optimal action policy by estimating the optimal state-action function Q(s, a) where s  state from the set of the possible states S, a  action from the set of the possible actions A. The Q function described the maximum reward achieved when an action a is executed over the state s BIB002 . The Qlearning equation is described as follows: Where  refers to the learning rate,  refers to the discount factor and r refers to the reward of executing the action a over the state s.