This paper studies directed exploration for reinforcement learning agents by tracking uncertainty about the value of each available action. We identify two sources of uncertainty that are relevant for exploration. The first originates from limited data (parametric uncertainty), while the second originates from the distribution of the returns (return uncertainty). We identify methods to learn these distributions with deep neural networks, where we estimate parametric uncertainty with Bayesian drop-out, while return uncertainty is propagated through the Bellman equation as a Gaussian distribution. Then, we identify that both can be jointly estimated in one network, which we call the Double Uncertain Value Network. The policy is directly derived from the learned distributions based on Thompson sampling. Experimental results show that both types of uncertainty may vastly improve learning in domains with a strong exploration challenge.
The idea of exploration based on parametric uncertainty also connects to the difference between action space and parameter space episode-based exploration . Most (undirected) exploration methods, like @math -greedy and Boltzmann, inject exploration noise at the action space level. However, it can be beneficial to inject the noise at parameter level instead, usually because it allows you to retain a particular noise setting over multiple steps (e.g. an entire episode). The risk of action-space exploration noise is that the agent has to redecide at every timestep and therefore cannot stick with an exploration decision. The effect might be jittering behaviour between exploration and exploitation steps. This has also been identified as the challenge of ensuring deep' exploration @cite_7 . We have not considered this problem in this paper, but it could for example be implemented by fixing the dropout mask over an entire episode.
@cite_7
2280163991
Efficient exploration in complex environments remains a major challenge for reinforcement learning. We propose bootstrapped DQN, a simple algorithm that explores in a computationally and statistically efficient manner through use of randomized value functions. Unlike dithering strategies such as epsilon-greedy exploration, bootstrapped DQN carries out temporally-extended (or deep) exploration; this can lead to exponentially faster learning. We demonstrate these benefits in complex stochastic MDPs and in the large-scale Arcade Learning Environment. Bootstrapped DQN substantially improves learning times and performance across most Atari games.