On the use of hybrid reinforcement learning for autonomic resource allocation
Abstract Reinforcement Learning (RL) provides a promising new approach to systems performance management that differs radically from standard queuing theoretic approaches making use of explicit system performance models. In principle, RL can automatically learn high quality management policies without an explicit performance model or traffic model, and with little or no built in system specific knowledge. In our original work (Das, R. Tesauro, G. Walsh, W.E. IBM Research, Tech. Rep. RC23802 (2005) Tesauro, G. In: Proc. of AAAI 05, pp. 886 891 (2005) Tesauro, G. Das, R. Walsh, W.E. Kephart, J.O. In: Proc. of ICAC 05, pp. 342 343 (2005) we showed the feasibility of using online RL to learn resource valuation estimates (in lookup table form) which can be used to make high quality server allocation decisions in a multi application prototype Data Center scenario. The present work shows how to combine the strengths of both RL and queuing models in a hybrid approach, in which RL trains offline on data collected while a queuing model policy controls the system. By training offline we avoid suffering potentially poor performance in live online training. We also now use RL to train nonlinear function approximators (e.g. multi layer perceptrons) instead of lookup tables; this enables scaling to substantially larger state spaces. Our results now show that, in both open loop and closed loop traffic, hybrid RL training can achieve significant performance improvements over a variety of initial model based policies. We also find that, as expected, RL can deal effectively with both transients and switching delays, which lie outside the scope of traditional steady state queuing theory.
Cluster Computing