A2C is a synchronous variant of A3C (the first A corresponding asynchronous is removed). In A3C, each worker agent talks to the global parameters independently, so sometimes the workers would perform with varying policies; hence, the accumulated update is not necessarily optimal. In contrast to A3C, a coordinator in A2C waits for all workers to complete their work before optimizing the global parameters, and then in the next iteration, each worker starts from the same policy. So the synchronized gradient update resolves mentioned inconsistency and works better with large batch sizes yet lacks the independence of workers. An
overview of the architecture is shown in Figure REF . Batched A2C [1]} is a variant of A2C, which at each step, it generates a batch of actions and applies them to a batch of environments. Since game logic and rendering are computationally cheap, batched A2C has high performance in Atari games.
<FIGURE>