Hierarchical reinforcement learning (HRL) decomposes problems into low-level subtasks that are learned by relatively simple agents for the purpose of either improved performance or explainability [1]}, [2]}. Larger problems are solved by choreographing these subtasks through an orchestration agent that learns the high-level dynamics of its environment [3]}. HRL has, for instance, been used to control a robotic arm: while low-level agents learned simple tasks such as moving forward / backward or picking up / placing down, an orchestration agent learned to retrieve objects on a surface by choreographing these tasks [4]}, [5]}. The agents were not only efficient at learning, but their policies were more easily interpreted by human experts. In [6]}, the authors used HRL to train a hierarchical set of agents to play a game. Their low-level agents learned to solve simple tasks such as “pick up a key” or “open a door” while receiving extrinsic rewards from the environment. A high-level agent then orchestrated these sub-tasks and received intrinsic rewards generated by a critic based on whether or not larger objectives were met.
