On the Role of Discount Factor in Offline Reinforcement Learning
Offline reinforcement learning (RL) enables effective learning from
previously collected data without exploration, which shows great promise in
real-world applications when exploration is expensive or even infeasible. The
discount factor, $\gamma$, plays a vital role in improving online RL sample
efficiency and estimation accuracy, but the role of the discount factor in
offline RL is not well explored. This paper examines two distinct effects of
$\gamma$ in offline RL with theoretical analysis, namely the regularization
effect and the pessimism effect. On the one hand, $\gamma$ is a regulator to
trade-off optimality with sample efficiency upon existing offline techniques.
On the other hand, lower guidance $\gamma$ can also be seen as a way of
pessimism where we optimize the policy's performance in the worst possible
models. We empirically verify the above theoretical observation with tabular
MDPs and standard D4RL tasks. The results show that the discount factor plays
an essential role in the performance of offline RL algorithms, both under small
data regimes upon existing offline methods and in large data regimes without
other conservative methods.