EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Engine
There has been significant progress in developing reinforcement learning (RL)
training systems. Past works such as IMPALA, Apex, Seed RL, Sample Factory, and
others aim to improve the system's overall throughput. In this paper, we try to
address a common bottleneck in the RL training system, i.e., parallel
environment execution, which is often the slowest part of the whole system but
receives little attention. With a curated design for paralleling RL
environments, we have improved the RL environment simulation speed across
different hardware setups, ranging from a laptop, and a modest workstation, to
a high-end machine like NVIDIA DGX-A100. On a high-end machine, EnvPool
achieves 1 million frames per second for the environment execution on Atari
environments and 3 million frames per second on MuJoCo environments. When
running on a laptop, the speed of EnvPool is 2.8 times of the Python
subprocess. Moreover, great compatibility with existing RL training libraries
has been demonstrated in the open-sourced community, including CleanRL,
rl_games, DeepMind Acme, etc. Finally, EnvPool allows researchers to iterate
their ideas at a much faster pace and has the great potential to become the de
facto RL environment execution engine. Example runs show that it takes only 5
minutes to train Atari Pong and MuJoCo Ant, both on a laptop. EnvPool has
already been open-sourced at https://github.com/sail-sg/envpool.