You Can't Count on Luck: Why Decision Transformers Fail in Stochastic Environments
Recently, methods such as Decision Transformer that reduce reinforcement
learning to a prediction task and solve it via supervised learning (RvS) have
become popular due to their simplicity, robustness to hyperparameters, and
strong overall performance on offline RL tasks. However, simply conditioning a
probabilistic model on a desired return and taking the predicted action can
fail dramatically in stochastic environments since trajectories that result in
a return may have only achieved that return due to luck. In this work, we
describe the limitations of RvS approaches in stochastic environments and
propose a solution. Rather than simply conditioning on the return of a single
trajectory as is standard practice, our proposed method, ESPER, learns to
cluster trajectories and conditions on average cluster returns, which are
independent from environment stochasticity. Doing so allows ESPER to achieve
strong alignment between target return and expected performance in real
environments. We demonstrate this in several challenging stochastic offline-RL
tasks including the challenging puzzle game 2048, and Connect Four playing
against a stochastic opponent. In all tested domains, ESPER achieves
significantly better alignment between the target return and achieved return
than simply conditioning on returns. ESPER also achieves higher maximum
performance than even the value-based baselines.