We show how an ensemble of $Q^*$-functions can be leveraged for more effective exploration in deep reinforcement learning. We build on well established algorithms from the bandit setting, and adapt them to the $Q$-learning setting. We propose an exploration strategy based on upper-confidence bounds (UCB). Our experiments show significant gains on the Atari benchmark. Deep reinforcement learning seeks to learn mappings from high-dimensional observations to actions. Deep Q-learning BID15 ) is a leading technique that has been used successfully, especially for video game benchmarks. However, fundamental challenges remain, for example, improving sample efficiency and ensuring convergence to high quality solutions. Provably optimal solutions exist in the bandit setting and for small MDPs, and at the core of these solutions are exploration schemes. However these provably optimal exploration techniques do not extend to deep RL in a straightforward way.Bootstrapped DQN BID16 ) is a previous attempt at adapting a theoretically verified approach to deep RL. In particular, it draws inspiration from posterior sampling for reinforcement learning (PSRL, BID17 ; BID16 ), which has near-optimal regret bounds. PSRL samples an MDP from its posterior each episode and exactly solves Q * , its optimal Q-function. However, in high-dimensional settings, both approximating the posterior over MDPs and solving the sampled MDP are intractable. Bootstrapped DQN avoids having to establish and sample from the posterior over MDPs by instead approximating the posterior over Q * . In addition, bootstrapped DQN uses a multi-headed neural network to represent the Q-ensemble. While the authors proposed bootstrapping to estimate the posterior distribution, their empirical findings show best performance is attained by simply relying on different initializations for the different heads, not requiring the sampling-with-replacement process that is prescribed by bootstrapping.In this paper, we design new algorithms that build on the Q-ensemble approach from BID16 . However, instead of using posterior sampling for exploration, we construct uncertainty estimates from the Q-ensemble. Specifically, we first propose the Ensemble Voting algorithm where the agent takes action by a majority vote from the Q-ensemble. Next, we propose the UCB exploration strategy. This strategy is inspired by established UCB algorithms in the bandit setting and constructs uncertainty estimates of the Q-values. In this strategy, agents are optimistic and take actions with the highest UCB. We demonstrate that our algorithms significantly improve performance on the Atari benchmark.
Adapting UCB exploration to ensemble Q-learning improves over prior methods such as Double DQN, A3C+ on Atari benchmark
{"src": "scitldr-AIC_train", "paper_id": "H1cKvl-Rb"}