We also compare against SAC [1]} and QT-OPT [2]}, two commonly used off-policy RL algorithms. Despite significant effort, we have so far not succeeded in training a successful policy using either method on this goal-reaching task (as shown by scores of 0 in Figure REF  for both algorithms) or on the simpler task of returning a ball to the opponent's side of the table. We do not claim that off-policy methods do not work on this problem, however we do observe that it appears significantly more difficult to train policies using this approach compared with either on-policy RL or GoalsEye.
<FIGURE>