Action Noise in Off-Policy Deep Reinforcement Learning: Impact on Exploration and Performance
Many deep reinforcement learning algorithms rely on simple forms of
exploration, such as the additive action-noise often used in continuous control
domains. Typically, the scaling factor of this action noise is chosen as a
hyper-parameter and kept constant during training. In this paper, we analyze
how the learned policy is impacted by the noise type, scale, and reducing of
the scaling factor over time. We consider the two most prominent types of
action-noise: Gaussian and Ornstein-Uhlenbeck noise, and perform a vast
experimental campaign by systematically varying the noise type and scale
parameter, and by measuring variables of interest like the expected return of
the policy and the state space coverage during exploration. For the latter, we
propose a novel state-space coverage measure
$\operatorname{X}_{\mathcal{U}\text{rel}}$ that is more robust to boundary
artifacts than previously proposed measures. Larger noise scales generally
increase state space coverage. However, we found that increasing the space
coverage using a larger noise scale is often not beneficial. On the contrary,
reducing the noise-scale over the training process reduces the variance and
generally improves the learning performance. We conclude that the best
noise-type and scale are environment dependent, and based on our observations,
derive heuristic rules for guiding the choice of the action noise as a starting
point for further optimization.