Hierarchical Reinforcement Learning: A Survey <s> F. Web Service Composition method using HRL <s> There are many static and dynamic Web services composition strategies, however literatures about automatic composition is very rare. In this article, a new algorithm based on reinforcement learning is proposed to realize web service composition automatically and randomly. On the other hand, the existing composition prototype systems mainly focus on function-oriented composition, but not QoS-oriented composition. After understanding the function-oriented composition by reinforcement learning, this paper then introduces preference logic to seek a QoS optimization solution, which is some kind of qualitative solution. When compared with quantitative solution it has many advantages. The result is a novel algorithm RLPLA, which is an algorithm of Web services composition based on reinforcement learning and preference logic <s> BIB001 </s> Hierarchical Reinforcement Learning: A Survey <s> F. Web Service Composition method using HRL <s> Dynamic web service composition is currently a hot issue in research. It mainly focuses on function-oriented or QoS-oriented, but the research combines with them are scanty. In this paper, we propose an HRLPLA algorithm to implement dynamic composition using hierarchical reinforcement learning and preference reasoning, which can consider both function-oriented and Qos-oriented. Experimental results will show its high effectiveness. <s> BIB002 </s> Hierarchical Reinforcement Learning: A Survey <s> F. Web Service Composition method using HRL <s> Through web services composition, distributed applications and enterprise business processes can be integrated by individual service components developed independently. In this chapter, we concentrate on the optimization problems of dynamic web service composition, and our goal is to find an optimal composite policy. We introduce a hierarchical reinforcement learning technique, i.e., a continuous-time unified MAXQ algorithm, to solve large-scale web service composition problems in the context of continuous-time semi-Markov decision process (SMDP) model under either average- or discounted-cost criteria. Finally, this proposed algorithm is tested in a simulation, and the experimental results show that it has better optimization performance than the flat Q-learning. <s> BIB003
Web services composition facilitates the combination of single web-services into featured/valued services that could satisfy the users' needs while the individual single web-services couldn't so. The dynamic web service composition model is shown in Figure 2 . When the "task acceptor" of the "service agency i" receives the data, the corresponding flow chart will be generated through the "composed service engine". Accordingly, each simple service will be executed by the "business execution engine". "Service agency i" will receive the results and the system continues to invoke "service agency j" that delivers other services BIB002 . One of the main problems of the dynamic web-service composition is the optimization problem (i.e. how to find an optimal policy). Different solutions have been proposed to accomplish an optimal policy for dynamic web service composition. BIB001 proposed an algorithm based on RL; however this algorithm suffers from the "Curse of dimensionality" specifically in large-scale problems of web-service composition. On the other hand, BIB003 introduces the HRL, a continuous time integrated MAXQ algorithm to handle the large-scale problems in the context of Semi-Markov decision process (SMDP). This algorithm has been compared to the Q-learning algorithm. Simulation results revealed that the performance of MAXQ algorithm is much better than the Q-learning algorithm by comparing the both with a discount factor a = 0.01 due to the reason that the MAXQ algorithm has the ability to accelerate the learning speed. Moreover, by comparing both algorithms with different number of tasks, it has been observed that as much as the number of tasks is increasing, the success rate of the Qlearning decreases much faster than MAXQ. BIB003 proved that the proposed algorithm is much better than the Qlearning algorithm to handle the problem of Curse of dimensionality in large-scale problems of web-service composition. Another problem in dynamic web-service composition is that how to combine a collection of simple web-services based on the users' functional needs and how to choose such services based on users' QoS needs among all the available services. BIB002 proposed an algorithm based on HRL and Logic of Preference; the algorithm that efficiently deals with both users' functional and QoS needs and has the ability to work in large-scale problems. The algorithm is decomposed into two parts: MAXQ (for service composition) and Logic of Preference (for choosing the service). An experiment has been conducted using 500 web services and 180 states. Experiments results revealed that the computation cost is significantly decreasing as long as the number of execution times is increasing. Moreover, results showed that utilizing HRL can effectively speed up the composition task.