A range of algorithms can be used to determine an optimal policy through Deep RL training, including off-policy algorithms such as Soft Actor Critic (SAC) [1]}, Deep Deterministic Policy Gradient (DDPG) [2]}, and Twin Delayed Deep Deterministic Policy (TD3) Gradients [3]}, and on-policy algorithms such as Trust Region Policy Optimization (TRPO) [4]}, Asynchronous Advantage Actor Critic (A3C) [5]} and Proximal Policy Optimization (PPO) [6]}. While off-policy methods tend to be more sample efficient (since they can reuse old data through the use of a replay buffer), on-policy methods directly optimize the objective and favor stability over sample efficiency [7]}. TRPO is a second-order algorithm which prevents large, sometimes destructive updates to the policy parameters by restricting the amount by which an update is allowed to change the policy [8]}. PPO is a first order approximation to TRPO and restricts the size of policy update via a KL divergence penalty or clipping mechanism [8]}. PPO is one of the most popular Deep RL methods, achieving state-of-the-art performance across a wide range of challenging tasks [10]}, [6]}. In this work we use PPO for all Deep RL training.
