Knowing how to efficiently balance between exploring unfamiliar parts of an environment and exploiting currently available knowledge is an essential ingredient of any intelligent agent. In theory, it is possible to obtain a Bayes-optimal solution to this exploration-exploitation dilemma by solving an augmented problem referred to as Bayes-adaptive Markov decision process (BAMDP, [1]}). However, BAMDPs are in general intractable to solve as analytical solutions are only available for a few special cases [2]}. The intractability of this problem prompted researchers to develop a body of heuristic solutions [3]}, [4]}, [5]}, [6]}. These heuristics can be divided into two broad categories: directed and random exploration strategies [7]}, [8]}. Directed exploration strategies provide bonus rewards that encourage the agent to visit parts of the environment that ought to be explored, whereas random exploration strategies inject some form of stochasticity into the policy.
