This paper concerns the distributed training of nonlinear kernel machines on Map-Reduce. We show that a re-formulation of Nystr "om approximation based solution which is solved using gradient based techniques is well suited for this, especially when it is necessary to work with a large number of basis points. The main advantages of this approach are: avoidance of computing the pseudo-inverse of the kernel sub-matrix corresponding to the basis points; simplicity and efficiency of the distributed part of the computations; and, friendliness to stage-wise addition of basis points. We implement the method using an AllReduce tree on Hadoop and demonstrate its value on a few large benchmark datasets.
P-packsvm @cite_20 uses primal stochastic gradient descent (SGD) applied in the kernel feature space. Training examples are partitioned into various nodes. The most expensive computation of an SGD iteration is the calculation of the output associated with a given example. This is done by passing the example to all nodes, locally computing the sub-outputs associated with the nodes and then using MPI AllReduce operation to get the summed output to all nodes. A packing strategy is used to do @math iterations together; since @math effort is involved, @math is kept small, say 100. P-packsvm is shown to work well on an MPI environment. On other distributed settings with communication overheads it can become slow due to the large number ( @math ) of iterations.
@cite_20
1574269637
Stochastic gradient descent (SGD) on a low-rank factorization is commonly employed to speed up matrix problems including matrix completion, subspace tracking, and SDP relaxation. In this paper, we exhibit a step size scheme for SGD on a low-rank least-squares problem, and we prove that, under broad sampling conditions, our method converges globally from a random starting point within @math steps with constant probability for constant-rank problems. Our modification of SGD relates it to stochastic power iteration. We also show experiments to illustrate the runtime and convergence of the algorithm.