Deep Hierarchical Planning from Pixels
Intelligent agents need to select long sequences of actions to solve complex
tasks. While humans easily break down tasks into subgoals and reach them
through millions of muscle commands, current artificial intelligence is limited
to tasks with horizons of a few hundred decisions, despite large compute
budgets. Research on hierarchical reinforcement learning aims to overcome this
limitation but has proven to be challenging, current methods rely on manually
specified goal spaces or subtasks, and no general solution exists. We introduce
Director, a practical method for learning hierarchical behaviors directly from
pixels by planning inside the latent space of a learned world model. The
high-level policy maximizes task and exploration rewards by selecting latent
goals and the low-level policy learns to achieve the goals. Despite operating
in latent space, the decisions are interpretable because the world model can
decode goals into images for visualization. Director outperforms exploration
methods on tasks with sparse rewards, including 3D maze traversal with a
quadruped robot from an egocentric camera and proprioception, without access to
the global position or top-down view that was used by prior work. Director also
learns successful behaviors across a wide range of environments, including
visual control, Atari games, and DMLab levels.