HogWild+ A New Mechanism for Decentralized Asynchronous Stochastic Gradient Descent
Stochastic Gradient Descent (SGD) is a popular technique for solving large scale machine learning problems. In order to parallelize SGD on multi core machines, asynchronous SGD (Hogwild! has been proposed, where each core updates a global model vector stored in a shared memory simultaneously, without using explicit locks. We show that the scalability of Hogwild! on modern multi socket CPUs is severely limited, especially on NUMA (Non Uniform Memory Access) system, due to the excessive cache invalidation requests and false sharing. In this paper we propose a novel decentralized asynchronous SGD algorithm called HogWild+ that overcomes these drawbacks and shows almost linear speedup on multi socket NUMA systems. The main idea in HogWild+ is to replace the global model vector with a set of local model vectors that are shared by a cluster (a set of cores) keep them synchronized through a decentralized token based protocol that minimizes remote memory access conflicts and ensures convergence. We present the design and experimental evaluation of HogWild+ on a variety of datasets and show that it outperforms state of the art parallel SGD implementations in terms of efficiency and scalability.
2016 IEEE 16th International Conference on Data Mining (ICDM)