Matching options to tasks using Option-Indexed Hierarchical Reinforcement Learning
The options framework in Hierarchical Reinforcement Learning breaks down
overall goals into a combination of options or simpler tasks and associated
policies, allowing for abstraction in the action space. Ideally, these options
can be reused across different higher-level goals; indeed, such reuse is
necessary to realize the vision of a continual learning agent that can
effectively leverage its prior experience. Previous approaches have only
proposed limited forms of transfer of prelearned options to new task settings.
We propose a novel option indexing approach to hierarchical learning (OI-HRL),
where we learn an affinity function between options and the items present in
the environment. This allows us to effectively reuse a large library of
pretrained options, in zero-shot generalization at test time, by restricting
goal-directed learning to only those options relevant to the task at hand. We
develop a meta-training loop that learns the representations of options and
environments over a series of HRL problems, by incorporating feedback about the
relevance of retrieved options to the higher-level goal. We evaluate OI-HRL in
two simulated settings - the CraftWorld and AI2THOR environments - and show
that we achieve performance competitive with oracular baselines, and
substantial gains over a baseline that has the entire option pool available for
learning the hierarchical policy.