For continuous-control experiments, we use a learning rate of \(3\cdot 10^{-4}\) , 10 updates per episode, and 32 minibatches. From one environment, we collected 2048 steps per epoch. For the arcade games experiments, following the setup of [1]}, we use a learning rate of \(1\cdot 10^{-4}\) , 3 updates per episode, and 8 minibatches. From 128 parallel environments, we collected 128 steps from each, per epoch. The policy value loss coefficient is fixed to \(0.5\)  and the entropy coefficient to \(0.001\) .
