@inproceedings{
miao2023scaling,
title={Scaling Offline Q-Learning with Vision Transformers},
author={Yingjie Miao and Jordi Orbay and Rishabh Agarwal and Aviral Kumar and George Tucker and Aleksandra Faust},
booktitle={NeurIPS 2023 Foundation Models for Decision Making Workshop},
year={2023},
url={https://openreview.net/forum?id=FtgaSZKFti}
}
It has been shown that offline RL methods, such as conservative Q-learning~(CQL), scale favorably for training generalist agents with a ResNet backbone. Recent vision and natural language processing research shows that transformer-based models scale more favorably compared to domain specific models with strong inductive biases (such as convolutional neural networks and recurrent neural networks). In this paper, we investigate how well visual transformers (ViTs) serve as backbones for CQL for training single-game agents. In this work, we enhance the Vision Transformer (ViT) for image-based RL by introducing spatio-temporal attention layers. We further investigate the impact of various embedding sequence aggregation methods on ViT performance. Overall, our modified ViT outperforms the standard ViTs in the single-game Atari setting.
Scaling Offline Q-Learning with Vision Transformers
Yingjie Miao Jordi Orbay Rishabh Agarwal Aviral Kumar George Tucker Aleksandra Faust