By contrast, RC-PPO has no explicit regularization in its optimization:
it merely clips the ratios between the policy to be optimized and the policy used to collect samples in each epoch.
The objective with these clipped probability ratios forms a pessimistic estimate of the surrogate objective proposed in Conservative Policy Iteration (CPI) [1]}.
Compared to KL-PPO and TRPO, ratio clipping avoids expensive computation of the KL divergence
and thus scales to large problem settings [2]}, [3]}.
It also yields stronger empirical performance than its KL regularized counterpart [4]}.
