This paper studies the potential of the return distribution for exploration in deterministic reinforcement learning (RL) environments. We study network losses and propagation mechanisms for Gaussian, Categorical and Gaussian mixture distributions. Combined with exploration policies that leverage this return distribution, we solve, for example, a randomized Chain task of length 100, which has not been reported before when learning with neural networks.
While the distributional Bellman equation (Eq. ) is certainly not new @cite_1 @cite_4 , nearly all RL research has focussed on the mean action-value. Most papers that do study the underlying return distribution study the 'variance of the return'. learned the distribution of the return with Gaussian Processes, but did not use it for exploration. studied the variance of the return with linear function approximation. theoretically studies policies that bound the variance of the return.
@cite_1 @cite_4
2313791856 2952647718
Formulae are presented for the variance and higher moments of the present value of single-stage rewards in a finite Markov decision process. Similar formulae are exhibited for a semi-Markov decision process. There is a short discussion of the obstacles to using the variance formula in algorithms to maximize the mean minus a multiple of the standard deviation. We consider finite horizon Markov decision processes under performance measures that involve both the mean and the variance of the cumulative reward. We show that either randomized or history-based policies can improve performance. We prove that the complexity of computing a policy that maximizes the mean reward under a variance constraint is NP-hard for some cases, and strongly NP-hard for others. We finally offer pseudopolynomial exact and approximation algorithms.