Hierarchical Linearly Solvable Markov Decision Problems Extended Version with Supplementary Material
We present a hierarchical reinforcement learning framework that formulates each task in the hierarchy as a special type of Markov decision process for which the Bellman equation is linear and has analytical solution. Problems of this type, called linearly solvable MDPs (LMDPs) have interesting properties that can be exploited in a hierarchical setting, such as efficient learning of the optimal value function or task compositionality. The proposed hierarchical approach can also be seen as a novel alternative to solving LMDPs with large state spaces. We derive a hierarchical version of the so called Z learning algorithm that learns different tasks simultaneously and show empirically that it significantly outperforms the state of the art learning methods in two classical HRL domains: the taxi domain and an autonomous guided vehicle task.