Motivated by progress in distributed deep learning, [1]} proposed IMPALA (Importance Weighted Actor-Learner Architecture) to scale-up DRL training to achieve high throughput. Similar to A3C, multiple actors interact with their environments in parallel. On the other hand, IMPALA separates acting from learning, and actors do not calculate gradients. Instead, they only produce trajectories of experience and communicate those data to a central learner, and the learner computes gradients and optimizes both policy and value functions. Each actor renews its parameters with the last version of the policy from the learner periodically (Figure REF ).
<FIGURE>