Discrete State-Action Abstraction via the Successor Representation
When reinforcement learning is applied with sparse rewards, agents must spend
a prohibitively long time exploring the unknown environment without any
learning signal. Abstraction is one approach that provides the agent with an
intrinsic reward for transitioning in a latent space. Prior work focuses on
dense continuous latent spaces, or requires the user to manually provide the
representation. Our approach is the first for automatically learning a discrete
abstraction of the underlying environment. Moreover, our method works on
arbitrary input spaces, using an end-to-end trainable regularized successor
representation model. For transitions between abstract states, we train a set
of temporally extended actions in the form of options, i.e., an action
abstraction. Our proposed algorithm, Discrete State-Action Abstraction (DSAA),
iteratively swaps between training these options and using them to efficiently
explore more of the environment to improve the state abstraction. As a result,
our model is not only useful for transfer learning but also in the online
learning setting. We empirically show that our agent is able to explore the
environment and solve provided tasks more efficiently than baseline
reinforcement learning algorithms. Our code is publicly available at
\url{https://github.com/amnonattali/dsaa}.