RC-PPO is also known for its ability to scale up in distributed settings [1]}, [2]}.
In this section, we show that ESPO can also be easily scaled up.
Similar to Distributed PPO in [3]},
we distribute the data collection and gradient computation over workers.
During training, gradients are first averaged across all workers
and then synchronised to each worker.
We adopt this distributed training paradigm as it is reported to yield strong empirical results in practice [3]}.
Likewise, we compute the ratio deviations for each worker
and use the minimum value of ratio deviations across all workers, i.e., reduce-min,
to decide whether to early stop from the optimization epochs.
Such early stopping is synchronised over all workers.
We have experimented with other ways of reducing ratio deviations across all workers
and have found that synchronously reducing to the minimum leads to better results in practice.
Refer to Algorithm REF  in Appendix REF .
