Beyond Tabula Rasa: Reincarnating Reinforcement Learning
Learning tabula rasa, that is without any prior knowledge, is the prevalent
workflow in reinforcement learning (RL) research. However, RL systems, when
applied to large-scale settings, rarely operate tabula rasa. Such large-scale
systems undergo multiple design or algorithmic changes during their development
cycle and use ad hoc approaches for incorporating these changes without
re-training from scratch, which would have been prohibitively expensive.
Additionally, the inefficiency of deep RL typically excludes researchers
without access to industrial-scale resources from tackling
computationally-demanding problems. To address these issues, we present
reincarnating RL as an alternative workflow, where prior computational work
(e.g., learned policies) is reused or transferred between design iterations of
an RL agent, or from one RL agent to another. As a step towards enabling
reincarnating RL from any agent to any other agent, we focus on the specific
setting of efficiently transferring an existing sub-optimal policy to a
standalone value-based RL agent. We find that existing approaches fail in this
setting and propose a simple algorithm to address their limitations. Equipped
with this algorithm, we demonstrate reincarnating RL's gains over tabula rasa
RL on Atari 2600 games, a challenging locomotion task, and the real-world
problem of navigating stratospheric balloons. Overall, this work argues for an
alternative approach to RL research, which we believe could significantly
improve real-world RL adoption and help democratize it further.