Learning frameworks might seldom take a significant time to conclude, and computation power could not always increase to suit the needs. Parallelization usually comes to the rescue.
Deep learning can get parallelized easily, and many of the distributed learning frameworks exploit this property. The DistBelief framework [1]} introduces a distributed deep learning framework and shows that such approaches can drastically overperform deep learning run on a single machine, even in low parallelization levels. DistBelief benefits from model and data parallelism. Having model parallelism, DistBelief partitions and trains the model among multiple machines, and having data parallelism, it replicates the model, and each replica learns based on a different partition of data. DistBelief method does not assume any high-level synchronicity. [2]} introduces another way to parallelize training convolutional neural networks with stochastic gradient descent, with a similar structure to the DistBelief method, but instead applying the resulting gradient updates synchronously. Recent RL frameworks mostly take advantage of neural networks for stochastic gradient descent.
Regarding the bottlenecks, [3]} does an in-depth overview of scalability in some supervised learning methods. Due to the requirement of sharing the parameter updates between the nodes, communication overheads emerge. Nevertheless, using larger batches and step sizes is a workaround to the mentioned problem, though it sacrifices the model's accuracy.
Finally, [4]} extensively analyzes concurrency concerns in multiple learning frameworks, an in-depth review of this subject.
A reinforcement learning system consists of the environment, the actor, and the learner, communicating one another. Distributed RL architectures often take advantage of parallelism in either the learner or the actor independently, so it affects the computation cost and efficiency. Nevertheless, the components themselves might have independently parallelized components internally.
