Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters
Motivated by the success of ensembles for uncertainty estimation in
supervised learning, we take a renewed look at how ensembles of $Q$-functions
can be leveraged as the primary source of pessimism for offline reinforcement
learning (RL). We begin by identifying a critical flaw in a popular algorithmic
choice used by many ensemble-based RL algorithms, namely the use of shared
pessimistic target values when computing each ensemble member's Bellman error.
Through theoretical analyses and construction of examples in toy MDPs, we
demonstrate that shared pessimistic targets can paradoxically lead to value
estimates that are effectively optimistic. Given this result, we propose MSG, a
practical offline RL algorithm that trains an ensemble of $Q$-functions with
independently computed targets based on completely separate networks, and
optimizes a policy with respect to the lower confidence bound of predicted
action values. Our experiments on the popular D4RL and RL Unplugged offline RL
benchmarks demonstrate that on challenging domains such as antmazes, MSG with
deep ensembles surpasses highly well-tuned state-of-the-art methods by a wide
margin. Additionally, through ablations on benchmarks domains, we verify the
critical significance of using independently trained $Q$-functions, and study
the role of ensemble size. Finally, as using separate networks per ensemble
member can become computationally costly with larger neural network
architectures, we investigate whether efficient ensemble approximations
developed for supervised learning can be similarly effective, and demonstrate
that they do not match the performance and robustness of MSG with separate
networks, highlighting the need for new efforts into efficient uncertainty
estimation directed at RL.