This paper addresses the problem of distributed learning under communication constraints, motivated by distributed signal processing in wireless sensor networks and data mining with distributed databases. After formalizing a general model for distributed learning, an algorithm for collaboratively training regularized kernel least-squares regression estimators is derived. Noting that the algorithm can be viewed as an application of successive orthogonal projection algorithms, its convergence properties are investigated and the statistical behavior of the estimator is discussed in a simplified theoretical setting.
Ongoing research in the machine learning community seeks to design statistically sound learning algorithms that scale to large data sets (e.g., @cite_10 and references therein). One approach is to decompose the database into smaller chunks", and subsequently parallelize the learning process by assigning distinct processors agents to each of the chunks. In principle, algorithms for parallelizing learning may be useful for distributed learning, and vice-versa. To our knowledge, there has not been an attempt to parallelize reproducing kernel methods using the approach outlined below.
@cite_10
2135106139
Very high dimensional learning systems become theoretically possible when training examples are abundant. The computing cost then becomes the limiting factor. Any efficient learning algorithm should at least take a brief look at each example. But should all examples be given equal attention?This contribution proposes an empirical answer. We first present an online SVM algorithm based on this premise. LASVM yields competitive misclassification rates after a single pass over the training examples, outspeeding state-of-the-art SVM solvers. Then we show how active example selection can yield faster training, higher accuracies, and simpler models, using only a fraction of the training example labels.