Similar to IMPALA, the trajectories are off-policy, so V-trace is applied to compensate for the trajectories. Additionally, R2D2 has many improvements to DQN, especially recurrent distributed replay buffer, enabling RNNs to predict future values based on all past information. However, [1]} implement the replay buffer locally in the learner rather than keeping it distributed and diminish complexity by excluding a load of work. In this setup, the replay buffer is restricted by the learner's memory, but they show it is not a problem for many ranges of tasks.
