Task-Agnostic Continual Reinforcement Learning: In Praise of a Simple Baseline
We study task-agnostic continual reinforcement learning (TACRL) in which
standard RL challenges are compounded with partial observability stemming from
task agnosticism, as well as additional difficulties of continual learning
(CL), i.e., learning on a non-stationary sequence of tasks. Here we compare
TACRL methods with their soft upper bounds prescribed by previous literature:
multi-task learning (MTL) methods which do not have to deal with non-stationary
data distributions, as well as task-aware methods, which are allowed to operate
under full observability. We consider a previously unexplored and
straightforward baseline for TACRL, replay-based recurrent RL (3RL), in which
we augment an RL algorithm with recurrent mechanisms to address partial
observability and experience replay mechanisms to address catastrophic
forgetting in CL.
  Studying empirical performance in a sequence of RL tasks, we find surprising
occurrences of 3RL matching and overcoming the MTL and task-aware soft upper
bounds. We lay out hypotheses that could explain this inflection point of
continual and task-agnostic learning research. Our hypotheses are empirically
tested in continuous control tasks via a large-scale study of the popular
multi-task and continual learning benchmark Meta-World. By analyzing different
training statistics including gradient conflict, we find evidence that 3RL's
outperformance stems from its ability to quickly infer how new tasks relate
with the previous ones, enabling forward transfer.