PPO is an advantage based actor-critic algorithm that tries to be conservative with policy updates. Using KL divergence and clipped surrogate function it can update multiple steps of update in one trajectory without worrying about policy fluctuating wildly[1]}. TRPO another similar algorithm does this by enforcing a hard trust-region constraint which solved with second-order methods. PPO does this by with first-order optimization its clipped objective. [2]}[3]}
