We also propose a novel policy optimization method,
Early Stopping Policy Optimization (ESPO),
which simply computes a ratio deviation to effectively determine
when to stop the optimization epoch.
The proposed ratio deviation is theoretically derived from a novel monotonic improvement guarantee and
can be effectively estimated without resorting to any complex computations, e.g., KL in TRPO.
We evaluate the performance of ESPO in many high-dimensional control benchmarks
and show that ESPO consistently outperforms PPO.
Moreover, we show that ESPO scales up easily in distributed training
to many workers [1]} and its performance is similar to or better than that of Distributed PPO [2]}.
