Bootstrapped Transformer for Offline Reinforcement Learning
Offline reinforcement learning (RL) aims at learning policies from previously
collected static trajectory data without interacting with the real environment.
Recent works provide a novel perspective by viewing offline RL as a generic
sequence generation problem, adopting sequence models such as Transformer
architecture to model distributions over trajectories, and repurposing beam
search as a planning algorithm. However, the training datasets utilized in
general offline RL tasks are quite limited and often suffer from insufficient
distribution coverage, which could be harmful to training sequence generation
models yet has not drawn enough attention in the previous works. In this paper,
we propose a novel algorithm named Bootstrapped Transformer, which incorporates
the idea of bootstrapping and leverages the learned model to self-generate more
offline data to further boost the sequence model training. We conduct extensive
experiments on two offline RL benchmarks and demonstrate that our model can
largely remedy the existing offline RL training limitations and beat other
strong baseline methods. We also analyze the generated pseudo data and the
revealed characteristics may shed some light on offline RL training. The codes
are available at https://seqml.github.io/bootorl.