Stabilizing Off-Policy Deep Reinforcement Learning from Pixels
Off-policy reinforcement learning (RL) from pixel observations is notoriously
unstable. As a result, many successful algorithms must combine different
domain-specific practices and auxiliary losses to learn meaningful behaviors in
complex environments. In this work, we provide novel analysis demonstrating
that these instabilities arise from performing temporal-difference learning
with a convolutional encoder and low-magnitude rewards. We show that this new
visual deadly triad causes unstable training and premature convergence to
degenerate solutions, a phenomenon we name catastrophic self-overfitting. Based
on our analysis, we propose A-LIX, a method providing adaptive regularization
to the encoder's gradients that explicitly prevents the occurrence of
catastrophic self-overfitting using a dual objective. By applying A-LIX, we
significantly outperform the prior state-of-the-art on the DeepMind Control and
Atari 100k benchmarks without any data augmentation or auxiliary losses.