CAMEO: Curiosity Augmented Metropolis for Exploratory Optimal Policies
Reinforcement Learning has drawn huge interest as a tool for solving optimal
control problems. Solving a given problem (task or environment) involves
converging towards an optimal policy. However, there might exist multiple
optimal policies that can dramatically differ in their behaviour; for example,
some may be faster than the others but at the expense of greater risk. We
consider and study a distribution of optimal policies. We design a
curiosity-augmented Metropolis algorithm (CAMEO), such that we can sample
optimal policies, and such that these policies effectively adopt diverse
behaviours, since this implies greater coverage of the different possible
optimal policies. In experimental simulations we show that CAMEO indeed obtains
policies that all solve classic control problems, and even in the challenging
case of environments that provide sparse rewards. We further show that the
different policies we sample present different risk profiles, corresponding to
interesting practical applications in interpretability, and represents a first
step towards learning the distribution of optimal policies itself.