The PPO algorithm alternates between data collection and policy optimization. During data collection the policy \(\pi _\theta \)  is executed on multiple instances of the environment in parallel [1]}, [2]}, performing a fixed number of action-steps on each parallel environment. During optimization stochastic gradient descent is performed by cycling through minibatches of the collected data to obtain sampled estimates of \(\nabla _\theta L^{PPO}(\theta )\) . The final output of DeepRL training is a fully trained policy network \(\pi _\theta (a|s)\) . In this work we use the Stable Baselines3 codebase [3]} for implementation of the PPO algorithm.
