ES and PPO [1]} are two widely used on-policy RL algorithms. In the low data setting limited to 120k trajectories (Figure REF ) ES fails to learn anything, whilst PPO just reaches 4% success rate on the 30cm metric. However, given enough environment trajectories both algorithms achieve comparable or better performance compared with GoalsEye when trained from scratch without any demonstration data (see Figure REF ). However they are significantly less sample efficient, requiring \(\approx 2\times \)  (PPO) and \(\approx 150\times \)  (ES) more trajectories in the environment to do so.
