Imitating Past Successes can be Very Suboptimal
Prior work has proposed a simple strategy for reinforcement learning (RL):
label experience with the outcomes achieved in that experience, and then
imitate the relabeled experience. These outcome-conditioned imitation learning
methods are appealing because of their simplicity, strong performance, and
close ties with supervised learning. However, it remains unclear how these
methods relate to the standard RL objective, reward maximization. In this
paper, we prove that existing outcome-conditioned imitation learning methods do
not necessarily improve the policy; rather, in some settings they can decrease
the expected reward. Nonetheless, we show that a simple modification results in
a method that does guarantee policy improvement, under some assumptions. Our
aim is not to develop an entirely new method, but rather to explain how a
variant of outcome-conditioned imitation learning can be used to maximize
rewards.