Trust region optimisation for reinforcement learning was popularised by TRPO [1]} which implements iterative guaranteed monotonic improvements. PPO [2]} preserves many of TRPO's empirical benefits while trading in theoretical guarantees for computational speed. PPO's policy update regularisation using clipped probability ratios has been subject to much scrutiny in single-agent settings: Truly PPO [3]} suggests modifications in order to ensure guaranteed monotonic improvements with little computational overhead. Recent work confirms that PPO's performance may crucially depend on the choice of policy surrogate objective [4]}. In addition, code-level design choices have been shown to significantly impact PPO performance in practice [5]}.
