@inproceedings{
wu2024variational,
title={Variational Delayed Policy Optimization},
author={Qingyuan Wu and Simon Sinong Zhan and Yixuan Wang and Yuhui Wang and Chung-Wei Lin and Chen Lv and Qi Zhu and Chao Huang},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=DAtNDZHbqj}
}
In environments with delayed observation, state augmentation by including actions within the delay window is adopted to retrieve Markovian property to enable reinforcement learning (RL). Whereas, state-of-the-art (SOTA) RL techniques with Temporal-Difference (TD) learning frameworks commonly suffer from learning inefficiency, due to the significant expansion of the augmented state space with the delay. To improve the learning efficiency without sacrificing performance, this work novelly introduces Variational Delayed Policy Optimization (VDPO), reforming delayed RL as a variational inference problem. This problem is further modelled as a two-step iterative optimization problem, where the first step is TD learning in the delay-free environment with a small state space, and the second step is behaviour cloning which can be addressed much more efficiently than TD learning. We not only provide a theoretical analysis of VDPO in terms of sample complexity and performance, but also empirically demonstrate that VDPO can achieve consistent performance with SOTA methods, with a significant enhancement of sample efficiency (approximately 50\% less amount of samples) in the MuJoCo benchmark.
Variational Delayed Policy Optimization
Qingyuan Wu Simon Sinong Zhan Yixuan Wang Yuhui Wang Chung-Wei Lin Chen Lv Qi Zhu Chao Huang