The General Reinforcement Learning Architecture, GORILA [1]}, is an asynchronous distributed reinforcement learning architecture. Gorila has similarities with DQN [ref] in the algorithm, yet has multiple workers and learners, each running within a single machine, and stores a Q-network inside each. It utilizes the DistBelief method [2]} to compute the stochastic gradient descent.
The Gorila framework architecture consists of 4 main parts. The parameter server, the workers, the learners, and the replay buffer.
The parameter server is the source of truth for the Q-network parameters \(\theta \) . Workers and learners each store a replica of \(\theta \) . Therefore, the parameter server tries to keep \(\theta \)  up-to-date in workers and learners.
Each worker stores the parameters \(\theta \) , according to which it acts. It is just used for acting and does not partake in the stochastic gradient descent procedure. So it gets synced with the updates to the parameter server. Workers send their experience \((s, a, s^{\prime }, a^{\prime })\) , to the replay memory, placed either on an independent distributed database or inside the workers.
Learners sample experiences from the replay buffer and perform the stochastic gradient descent update. Such in DQN, each learner keeps an "improving" Q-network, with parameters \(\theta \)  and an assumably "constant" target Q-network, with parameters \(\theta ^-\) . The target network gets updated once in N iterations, such in DQN. The learner's responsibility is to determine the loss from experiences sampled from the replay buffer. Still, the learner does not update the improving Q-network, as opposed to DQN, it sends the resulting gradient to the parameter server. The improving Q-network only gets synced whenever the parameter server updates \(\theta \) , because there are other learners issuing updates on \(\theta \) , and the outgoing update is not necessarily valid.
The parameter server contains a DistBelief network. It is responsible for applying the incoming gradients from learners and sending network parameters to the Q-networks in learners and actors, as described above. The parameter server drops "outdated" gradient updates, making the parameters from this source reliable even on process and communication delays and failures.
Several variants of this architecture are categorized as called GORILA. The number of learners and actors could be anything, one or many, and the learner and actors could be "bundled," meaning that there is the same number of learners and actors, each couple placed in one machine. This variant reduces communication overhead, so it has better performance and was used to evaluate the framework.
