We consider three different state of the art agents: A3C [1]}, Rainbow [2]} and the IMPALA reinforcement learning agent [3]}. A3C comes from a family of actor-critic methods which combine value learning and policy gradient approaches in order to reduce the variance of the gradients. Rainbow is an extension of the standard DQN [4]} agent, which combines double Q-learning [5]},
dueling networks [6]}, distributional RL [7]} and noisy nets [8]}.
Moreover, it is equipped with a replay buffer that stores the previous million transitions of the form \((o_t, a_t, r_{t+1}, o_{t+1})\) , which is then sampled using a prioritized weighting scheme based on temporal difference errors [9]}.
Finally, IMPALA is an extension of A3C, which uses the standard actor-critic architecture with off-policy corrections in order to scale effectively to a large scale distributed setup. We train IMPALA with population based training [10]}.
