For our experiments we first compare ESPO with RC-PPO and other baselines across a wide range of tasks.
Specifically, we first consider continuous control Mujoco benchmarks [1]},
and then scale up to higher dimension continuous control in the DeepMind Control Suite [2]}.
Table REF  in Appendix REF  gives an overview of the control complexity in terms of state-action dimensions.
In all tasks we use a Gaussian distribution for the policy whose mean and covariance are parameterized by a neural network
(see Appendix REF  Table REF  for more details).
Second, we scale up the training to many workers
and compare the performance of Distributed ESPO with Distributed RC-PPO.
We set the stopping threshold in ESPO to \(0.25\) 
as we found it generalizes well to all control tasks.
Following [3]}, [4]},
we also normalize observations, rewards and advantages for ESPO and Distributed ESPO.
See Figure REF  in Appendix REF  for the ablations.
The implementation details can also be found in Appendix REF .
All the empirical results are reported by the mean and standard deviation,
with 5 random repeated runs (as indicated in the figure legend).
