Sequential decision-making as sequence modeling. We are not the first to consider sequential decision problems as a sequence modeling problem. [1]} and [2]} focus on RL and show how one can use GPT-style (causally-masked) Transformer models to directly generate high-reward trajectories in an offline RL setting.
Unlike this line of work, we focus on many tasks that a sequence modeling perspective enables one to do, rather than just offline-RL. While some previous work has cast doubt on the necessity of using transformers to achieve good results in offline RL [3]}, we note that offline RL [4]} is just one of the various tasks we consider.
Concurrent work to ours generalizes the left-to-right masking in the Transformer to condition on future trajectory information for tasks such as state marginal matching [5]} and multi-agent motion forecasting [6]}. In contrast to these works, we systematically investigate how a single bidirectional Transformer model can be trained to perform arbitrary downstream tasks (in more complex settings than motion forecasting—i.e., we also consider agent actions and rewards in addition to states). The main thing that sets us apart from these works is a systematic view of all tasks that can be represented by this sequence-modeling perspective, and a detailed investigation of how different multi-task training regimes compare.
