Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RL
We introduce an offline reinforcement learning (RL) algorithm that explicitly
clones a behavior policy to constrain value learning. In offline RL, it is
often important to prevent a policy from selecting unobserved actions, since
the consequence of these actions cannot be presumed without additional
information about the environment. One straightforward way to implement such a
constraint is to explicitly model a given data distribution via behavior
cloning and directly force a policy not to select uncertain actions. However,
many offline RL methods instantiate the constraint indirectly -- for example,
pessimistic value estimation -- due to a concern about errors when modeling a
potentially complex behavior policy. In this work, we argue that it is not only
viable but beneficial to explicitly model the behavior policy for offline RL
because the constraint can be realized in a stable way with the trained model.
We first suggest a theoretical framework that allows us to incorporate
behavior-cloned models into value-based offline RL methods, enjoying the
strength of both explicit behavior cloning and value learning. Then, we propose
a practical method utilizing a score-based generative model for behavior
cloning. With the proposed method, we show state-of-the-art performance on
several datasets within the D4RL and Robomimic benchmarks and achieve
competitive performance across all datasets tested.