Training of one-vs.-rest SVMs can be parallelized over the number of classes in a straight forward way. Given enough computational resources, one-vs.-rest SVMs can thus be trained on data involving a large number of classes. The same cannot be stated, however, for the so-called all-in-one SVMs, which require solving a quadratic program of size quadratically in the number of classes. We develop distributed algorithms for two all-in-one SVM formulations ( and Weston and Watkins) that parallelize the computation evenly over the number of classes. This allows us to compare these models to one-vs.-rest SVMs on unprecedented scale. The results indicate superior accuracy on text classification data.
Note that beyond SVMs there is a large body of work on distributed multi-class [e.g.,][] vw,gopal2013distributed and multi-label learning algorithms @cite_21 , which is outside of the scope of the present paper.
@cite_21
2119991813 2007573800 1999428432 2897372286
In this paper, we study the problem of learning from weakly labeled data, where labels of the training examples are incomplete. This includes, for example, (i) semi-supervised learning where labels are partially known; (ii) multi-instance learning where labels are implicitly known; and (iii) clustering where labels are completely unknown. Unlike supervised learning, learning with weak labels involves a difficult Mixed-Integer Programming (MIP) problem. Therefore, it can suffer from poor scalability and may also get stuck in local minimum. In this paper, we focus on SVMs and propose the WELLSVM via a novel label generation strategy. This leads to a convex relaxation of the original MIP, which is at least as tight as existing convex Semi-Definite Programming (SDP) relaxations. Moreover, the WELLSVM can be solved via a sequence of SVM subproblems that are much more scalable than previous convex SDP relaxations. Experiments on three weakly labeled learning tasks, namely, (i) semi-supervised learning; (ii) multi-instance learning for locating regions of interest in content-based information retrieval; and (iii) clustering, clearly demonstrate improved performance, and WELLSVM is also readily applicable on large data sets. With data sizes constantly expanding, and with classical machine learning algorithms that analyze such data requiring larger and larger amounts of computation time and storage space, the need to distribute computation and memory requirements among several computers has become apparent. Although substantial work has been done in developing distributed binary SVM algorithms and multi-class SVM algorithms individually, the field of multi-class distributed SVMs remains largely unexplored. This research proposes a novel algorithm that implements the Support Vector Machine over a multi-class dataset and is efficient in a distributed environment (here, Hadoop). The idea is to divide the dataset into half recursively and thus compute the optimal Support Vector Machine for this half during the training phase, much like a divide and conquer approach. While testing, this structure has been effectively exploited to significantly reduce the prediction time. Our algorithm has shown better computation time during the prediction phase than the traditional sequential SVM methods (One vs. One, One vs. Rest) and out-performs them as the size of the dataset grows. This approach also classifies the data with higher accuracy than the traditional multi-class algorithms. We present an algorithm and implementation for distributed parallel training of single-machine multiclass SVMs. While there is ongoing and healthy debate about the best strategy for multiclass classification, there are some features of the single-machine approach that are not available when training alternatives such as one-vs-all, and that are quite complex for tree based methods. One obstacle to exploring single-machine approaches on large datasets is that they are usually limited to running on a single machine! We build on a framework borrowed from parallel convex optimization — the alternating direction method of multipliers (ADMM) — to develop a new consensus based algorithm for distributed training of single-machine approaches. This is demonstrated with an implementation of our novel sequential dual algorithm (DCMSVM) which allows distributed parallel training with small communication requirements. Benchmark results show significant reduction in wall clock time compared to current state of the art multiclass SVM implementation (Liblinear) on a single node. Experiments are performed on large scale image classification including results with modern high-dimensional features. This paper presents a novel multi-label active learning (MLAL) technique in the framework of multi-label remote sensing (RS) image scene classification problems. The proposed MLAL technique is developed in the framework of the multi-label SVM classifier (ML-SVM). Unlike the standard AL methods, the proposed MLAL technique redefines active learning by evaluating the informativeness of each image based on its multiple land-cover classes. Accordingly, the proposed MLAL technique is based on the joint evaluation of two criteria for the selection of the most informative images: i) multi-label uncertainty and ii) multi-label diversity. The multi-label uncertainty criterion is associated to the confidence of the multi-label classification algorithm in correctly assigning multi-labels to each image, whereas multi-label diversity criterion aims at selecting a set of un-annotated images that are as more diverse as possible to reduce the redundancy among them. In order to evaluate the multi-label uncertainty of each image, we propose a novel multi-label margin sampling strategy that: 1) considers the functional distances of each image to all ML-SVM hyperplanes; and then 2) estimates the occurrence on how many times each image falls inside the margins of ML-SVMs. If the occurrence is small, the classifiers are confident to correctly classify the considered image, and vice versa. In order to evaluate the multi-label diversity of each image, we propose a novel clustering-based strategy that clusters all the images inside the margins of the ML-SVMs and avoids selecting the uncertain images from the same clusters. The joint use of the two criteria allows one to enrich the training set of images with multi-labels. Experimental results obtained on a benchmark archive with 2100 images with their multi-labels show the effectiveness of the proposed MLAL method compared to the standard AL methods that neglect the evaluation of the uncertainty and diversity on multi-labels.