In GORILA, the agent collects data in the replay buffer so that the data can be batched or sampled, non-stationarity is decreased, and updates are decorrelated. However, storing experiences in the replay buffer restricts the methods to off-policy algorithms, and requires more computation and memory per interaction. That is why in Asynchronous Advantage Actor-Critic [1]}, short for A3C, multiple agents asynchronously run in parallel to generate data. This approach provides a more practical alternative to experience replay since parallelization also diversifies and decorrelates the data.
<FIGURE>