Hybrid Value Estimation for Off-policy Evaluation and Offline Reinforcement Learning
Value function estimation is an indispensable subroutine in reinforcement
learning, which becomes more challenging in the offline setting. In this paper,
we propose Hybrid Value Estimation (HVE) to reduce value estimation error,
which trades off bias and variance by balancing between the value estimation
from offline data and the learned model. Theoretical analysis discloses that
HVE enjoys a better error bound than the direct methods. HVE can be leveraged
in both off-policy evaluation and offline reinforcement learning settings. We,
therefore, provide two concrete algorithms Off-policy HVE (OPHVE) and
Model-based Offline HVE (MOHVE), respectively. Empirical evaluations on MuJoCo
tasks corroborate the theoretical claim. OPHVE outperforms other off-policy
evaluation methods in all three metrics measuring the estimation effectiveness,
while MOHVE achieves better or comparable performance with state-of-the-art
offline reinforcement learning algorithms. We hope that HVE could shed some
light on further research on reinforcement learning from fixed data.