Before introducing our new policy optimization algorithm,
we first revisit the underlying idea in PPO.
As pointed out by [1]},
PPO reuses samples for multiple epochs of policy optimization.
This reuse of samples, however, does not come for free.
Theorem REF  suggests that,
whenever the policy is updated, it deviates from the policy used to collect samples, i.e., behaviour policy, and thus
reusing those samples for the next optimization epoch yields a weaker lower bound for policy improvement.
If one proceeds with multiple epochs anyway,
the lower bound in Theorem REF  would not guarantee any improvement of the policy [2]}, [3]}.
This inspires us to develop an algorithm that finds the proper time to stop sample reuse,
before the lower bound becomes trivial.
However, the performance lower bound presented in Theorem REF  relies on the maximum TV divergence,
which is intractable to estimate in practice.
In the following, we present a new monotonic improvement guarantee with a more practical lower bound.
