Stein Variational Goal Generation For Reinforcement Learning in Hard Exploration Problems
Multi-goal Reinforcement Learning has recently attracted a large amount of
research interest. By allowing experience to be shared between related training
tasks, this setting favors generalization for new tasks at test time, whenever
some smoothness exists in the considered representation space of goals.
However, in settings with discontinuities in state or goal spaces (e.g. walls
in a maze), a majority of goals are difficult to reach, due to the sparsity of
rewards in the absence of expert knowledge. This implies hard exploration, for
which some curriculum of goals must be discovered, to help agents learn by
adapting training tasks to their current capabilities. Building on recent
automatic curriculum learning techniques for goal-conditioned policies, we
propose a novel approach: Stein Variational Goal Generation (SVGG), which seeks
at preferably sampling new goals in the zone of proximal development of the
agent, by leveraging a learned model of its abilities, and a goal distribution
modeled as particles in the exploration space. Our approach relies on Stein
Variational Gradient Descent to dynamically attract the goal sampling
distribution in areas of appropriate difficulty. We demonstrate the
performances of the approach, in terms of success coverage in the goal space,
compared to recent state-of-the-art RL methods for hard exploration problems.