PPO is also known for its ability to scale up in distributed settings [1]}, [2]}, [3]}, [4]}.
We thus also compare ESPO to Distributed PPO (DPPO) [2]}.
Specifically, we train ESPO in a single process for 50 million timesteps
and compare this single-process ESPO, i.e., ESPO-1-worker, with PPO trained with multiple processes.
Similar to [2]}, we run 10 workers in parallel, i.e., PPO-10-workers,
each of which is assigned one actor to collect many timesteps of data
and one learner to compute the gradient and update policy parameters [7]}, [4]}.
The gradients of the PPO learners are averaged through the Message Passing Interface (MPI) all-reduce algorithm [9]}.
We use the more complicated Mujoco tasks,
including Ant, Humanoid, HumanoidStandup and Walker2d.
We also scale up the training of ESPO in the same manner according to Algorithm REF ,
and denote its performance as ESPO-10-workers.
The training performance of ESPO and DPPO is presented in Figure REF .
We also report the single process PPO, PP0-1-worker, for reference.
Clearly, scaling up the training via multiple workers
significantly improves PPO's performance:
PPO-10-workers significantly outperforms its single process version,
on Ant, Humanoid, and HumanoidStandup.
Hence, large-scale training may explain the success of PPO in many game applications [3]}, [4]},
Also, with the same number of samples,
ESPO still outperforms DPPO on Humanoid and HumanoidStandup,
even performs better than its scaled-up variant, ESPO-10-workers.
On Ant domain, the performance of ESPO is worse than that of DPPO.
But the distributed ESPO, i.e., ESPO-10-worker, performs rather similarly as DPPO
It could be that, on this specific domain,
averaging the gradient across many workers reduces the variance
and makes policy optimization more effective in each iteration.
