A distributed randomized block coordinate descent method for minimizing a convex function of a huge number of variables is proposed. The complexity of the method is analyzed under the assumption that the smooth part of the objective function is partially block separable. The number of iterations required is bounded by a function of the error and the degree of separability, which extends the results in Richtarik and Takac (Parallel Coordinate Descent Methods for Big Data Optimization, Mathematical Programming, DOI:10.1007 s10107-015-0901-6) to a distributed environment. Several approaches to the distribution and synchronization of the computation across a cluster of multi-core computer are described and promising computational results are provided.
Distributed block-coordinate descent. Distributed coordinate descent was first proposed by Bertsekas and Tsitsiklis @cite_13 . The literature on this topic was rather sparse, c.f. @cite_29 , until the research presented in this paper raised the interest, which lead to the analyses of Richt ' a rik and Tak ' a c @cite_26 and @cite_35 . These papers do not consider blocks, and specialise our results to convex functions admitting a quadratic upper bound. In the machine-learning community, distributed algorithms have been studied for particular problems, e.g., training of support vector machines @cite_0 . Google @cite_21 developed a library called PSVM, where parallel row-based incomplete Cholesky factorization is employed in an interior-point method. A MapReduce-based distributed algorithm for SVM was found to be effective in automatic image annotation @cite_6 . Nevertheless, none of these papers use coordinate descent.
@cite_35 @cite_26 @cite_29 @cite_21 @cite_6 @cite_0 @cite_13
2964252705 2949085567 2165966284 2099262739 1963611305 2540030413 1603765807
We propose an efficient distributed randomized coordinate descent method for minimizing regularized non-strongly convex loss functions. The method attains the optimal O(1 k) convergence rate, where k is the iteration counter. The core of the work is the theoretical study of stepsize parameters. We have implemented the method on Archer—the largest supercomputer in the UK—and show that the method is capable of solving a (synthetic) LASSO optimization problem with 50 billion variables. In this paper we develop and analyze Hydra: HYbriD cooRdinAte descent method for solving loss minimization problems with big data. We initially partition the coordinates (features) and assign each partition to a different node of a cluster. At every iteration, each node picks a random subset of the coordinates from those it owns, independently from the other computers, and in parallel computes and applies updates to the selected coordinates based on a simple closed-form formula. We give bounds on the number of iterations sufficient to approximately solve the problem with high probability, and show how it depends on the data and on the partitioning. We perform numerical experiments with a LASSO instance described by a 3TB matrix. In many applications, data appear with a huge number of instances as well as features. Linear Support Vector Machines (SVM) is one of the most popular tools to deal with such large-scale sparse data. This paper presents a novel dual coordinate descent method for linear SVM with L1-and L2-loss functions. The proposed method is simple and reaches an e-accurate solution in O(log(1 e)) iterations. Experiments indicate that our method is much faster than state of the art solvers such as Pegasos, TRON, SVMperf, and a recent primal coordinate descent implementation. Support Vector Machines (SVMs) suffer from a widely recognized scalability problem in both memory use and computational time. To improve scalability, we have developed a parallel SVM algorithm (PSVM), which reduces memory use through performing a row-based, approximate matrix factorization, and which loads only essential data to each machine to perform parallel computation. Let n denote the number of training instances, p the reduced matrix dimension after factorization (p is significantly smaller than n), and m the number of machines. PSVM reduces the memory requirement from O(n2) to O(np m), and improves computation time to O(np2 m). Empirical study shows PSVM to be effective. PSVM Open Source is available for download at http: code.google.com p psvm . Machine learning techniques have facilitated image retrieval by automatically classifying and annotating images with keywords. Among them Support Vector Machines (SVMs) have been used extensively due to their generalization properties. However, SVM training is notably a computationally intensive process especially when the training dataset is large. This paper presents MRSMO, a MapReduce based distributed SVM algorithm for automatic image annotation. The performance of the MRSMO algorithm is evaluated in an experimental environment. By partitioning the training dataset into smaller subsets and optimizing the partitioned subsets across a cluster of computers, the MRSMO algorithm reduces the training time significantly while maintaining a high level of accuracy in both binary and multiclass classifications. Support Vector Machine (SVM) is an efficient data mining approach for data classification. However, SVM algorithm requires very large memory requirement and computational time to deal with very large dataset. To reduce the computational time during the process of training the SVM, a combination of distributed and parallel computing method, CoDLib have been proposed. Instead of using a single machine for parallel computing, multiple machines in a cluster are used. Message Passing Interface (MPI) is used in the communication between machines in the cluster. The original dataset is split and distributed to the respective machines. Experiments results shows a great speed up on the training of the MNIST dataset where training time has been significantly reduced compared with standard LIBSVM without affecting the quality of the SVM.