PPO performs multiple mini-batch updates (i.e., epochs) of policy updates on the same set of samples.
With proper regularization in each optimization epoch, e.g., a trust region constraint, monotonic improvement can be guaranteed [1]}, [2]}.
We thus first investigate whether ratio clipping can enforce a trust region constraint in those optimization epochs.
We also point out discrepancies between the existing theory and practice.
