Mildly Conservative Q-Learning for Offline Reinforcement Learning
Offline reinforcement learning (RL) defines the task of learning from a
static logged dataset without continually interacting with the environment. The
distribution shift between the learned policy and the behavior policy makes it
necessary for the value function to stay conservative such that
out-of-distribution (OOD) actions will not be severely overestimated. However,
existing approaches, penalizing the unseen actions or regularizing with the
behavior policy, are too pessimistic, which suppresses the generalization of
the value function and hinders the performance improvement. This paper explores
mild but enough conservatism for offline learning while not harming
generalization. We propose Mildly Conservative Q-learning (MCQ), where OOD
actions are actively trained by assigning them proper pseudo Q values. We
theoretically show that MCQ induces a policy that behaves at least as well as
the behavior policy and no erroneous overestimation will occur for OOD actions.
Experimental results on the D4RL benchmarks demonstrate that MCQ achieves
remarkable performance compared with prior work. Furthermore, MCQ shows
superior generalization ability when transferring from offline to online, and
significantly outperforms baselines.