Unlike the explicit regularization used in regularized policy optimization methods,
Proximal Policy Optimization (PPO) [1]} exploits a simple scheme:
clipping probability ratios between policies at each optimization epoch.
Although this scheme produces strong empirical performance,
it remains largely unknown how ratio clipping relates to trust region enforcement.
Existing studies, including the original PPO paper [1]},
related this clipping scheme to the bounding of ratios [3]}, [4]}.
Many other studies, however, suggest the ratios could become unbounded via this clipping [5]}, [6]}, [7]}.
[5]} show that, even if the ratios are bounded, the trust region constraint, defined with the KL as in TRPO,
could still be violated.
Ratio clipping has also been interpreted
as a way to equivalently enforce a TV divergence [3]}, [4]}
or approximately apply a hinge loss [11]}.
