The approach introduced in [1]}, entitled Ape-X, which was used and improved in the method of Recurrent Experience Replay Distributed Reinforcement Learning, R2D2 [2]}, is a value-based framework similar to GORILA from the aspect of the distributed architecture. Ape-X aims to reduce variance and accelerate convergence and benefits from importance sampling and prioritized experience replay. That is, the learners sample the experiences with weights proportional to the difference that they would make. Since the older experiences with more priority are less disregarded, the gradient is less likely to overfit, regarding the latest experiences. So in a sense, prioritized replay aims the same purpose as PPO. Nevertheless, prioritized experience replay alone introduces a bias to the gradient updates; Therefore, importance sampling is responsible for canceling this bias.
The Ape-X framework uses variants of DQN, such as the double Q-learning algorithm [3]}, and Deep Deterministic Policy Gradient (DDPG) [4]} multi-step bootstrap targets (Sutton, 1988), as the deep network for evaluation, and uses a dueling network architecture [5]} as the function approximator. In Ape-X, parallelism happens more extensively in acting, and the variant with a single learner on a GPU has gained the best empirical performance.
The model network in R2D2 is RNN-based, and like in Ape-X, double Q-learning with multi-step returns is used. R2D2 applies modifications on the model network, and the experience replay stores overlapping sequences of \((s, a, r)\)  tuples of experience instead of the default \((s, a, r, s^{\prime })\)  tuples.
