Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations
Offline reinforcement learning has shown great promise in leveraging large
pre-collected datasets for policy learning, allowing agents to forgo
often-expensive online data collection. However, to date, offline reinforcement
learning from has been relatively under-explored, and there is a lack of
understanding of where the remaining challenges lie. In this paper, we seek to
establish simple baselines for continuous control in the visual domain. We show
that simple modifications to two state-of-the-art vision-based online
reinforcement learning algorithms, DreamerV2 and DrQ-v2, suffice to outperform
prior work and establish a competitive baseline. We rigorously evaluate these
algorithms on both existing offline datasets and a new testbed for offline
reinforcement learning from visual observations that better represents the data
distributions present in real-world offline reinforcement learning problems,
and open-source our code and data to facilitate progress in this important
domain. Finally, we present and analyze several key desiderata unique to
offline RL from visual observations, including visual distractions and visually
identifiable changes in dynamics.