This paper addresses the problem of distributed learning under communication constraints, motivated by distributed signal processing in wireless sensor networks and data mining with distributed databases. After formalizing a general model for distributed learning, an algorithm for collaboratively training regularized kernel least-squares regression estimators is derived. Noting that the algorithm can be viewed as an application of successive orthogonal projection algorithms, its convergence properties are investigated and the statistical behavior of the estimator is discussed in a simplified theoretical setting.
We consider the problem of decentralized detection under constraints on the number of bits that can be transmitted by each sensor. In contrast to most previous work, in which the joint distribution of sensor observations is assumed to be known, we address the problem when only a set of empirical samples is available. We propose a novel algorithm using the framework of empirical risk minimization and marginalized kernels, and analyze its computational and statistical properties both theoretically and empirically. We provide an efficient implementation of the algorithm, and demonstrate its performance on both simulated and real data sets.  We present distributed regression, an efficient and general framework for in-network modeling of sensor data. In this framework, the nodes of the sensor network collaborate to optimally fit a global function to each of their local measurements. The algorithm is based upon kernel linear regression, where the model takes the form of a weighted sum of local basis functions; this provides an expressive yet tractable class of models for sensor network data. Rather than transmitting data to one another or outside the network, nodes communicate constraints on the model parameters, drastically reducing the communication required. After the algorithm is run, each node can answer queries for its local region, or the nodes can efficiently transmit the parameters of the model to a user outside the network. We present an evaluation of the algorithm based upon data from a 48-node sensor network deployment at the Intel Research - Berkeley Lab, demonstrating that our distributed algorithm converges to the optimal solution at a fast rate and is very robust to packet losses. We introduce a new formal model in which a learning algorithm must combine a collection of potentially poor but statistically independent hypothesis functions in order to approximate an unknown target function arbitrarily well. Our motivation includes the question of how to make optimal use of multiple independent runs of a mediocre learning algorithm, as well as settings in which the many hypotheses are obtained by a distributed population of identical learning agents. Motivated by sensor networks and other distributed settings, several models for distributed learning are presented. The models differ from classical works in statistical pattern recognition by allocating observations of an independent and identically distributed (i.i.d.) sampling process among members of a network of simple learning agents. The agents are limited in their ability to communicate to a central fusion center and thus, the amount of information available for use in classification or regression is constrained. For several basic communication models in both the binary classification and regression frameworks, we question the existence of agent decision rules and fusion rules that result in a universally consistent ensemble; the answers to this question present new issues to consider with regard to universal consistency. This paper addresses the issue of whether or not the guarantees provided by Stone's theorem in centralized environments hold in distributed settings. In this paper, we propose a general framework for distributed boosting intended for efficient integrating specialized classifiers learned over very large and distributed homogeneous databases that cannot be merged at a single location. Our distributed boosting algorithm can also be used as a parallel classification technique, where a massive database that cannot fit into main computer memory is partitioned into disjoint subsets for a more efficient analysis. In the proposed method, at each boosting round the classifiers are first learned from disjoint datasets and then exchanged amongst the sites. Finally the classifiers are combined into a weighted voting ensemble on each disjoint data set. The ensemble that is applied to an unseen test set represents an ensemble of ensembles built on all distributed sites. In experiments performed on four large data sets the proposed distributed boosting method achieved classification accuracy comparable or even slightly better than the standard boosting algorithm while requiring less memory and less computational time. In addition, the communication overhead of the distributed boosting algorithm is very small making it a viable alternative to the standard boosting for large-scale databases. 1 Introduction.- 1.1 Distributed Detection Systems.- 1.2 Outline of the Book.- 2 Elements of Detection Theory.- 2.1 Introduction.- 2.2 Bayesian Detection Theory.- 2.3 Minimax Detection.- 2.4 Neyman-Pearson Test.- 2.5 Sequential Detection.- 2.6 Constant False Alarm Rate (CFAR) Detection.- 2.7 Locally Optimum Detection.- 3 Distributed Bayesian Detection: Parallel Fusion Network.- 3.1 Introduction.- 3.2 Distributed Detection Without Fusion.- 3.3 Design of Fusion Rules.- 3.4 Detection with Parallel Fusion Network.- 4 Distributed Bayesian Detection: Other Network Topologies.- 4.1 Introduction.- 4.2 The Serial Network.- 4.3 Tree Networks.- 4.4 Detection Networks with Feedback.- 4.5 Generalized Formulation for Detection Networks.- 5 Distributed Detection with False Alarm Rate Constraints.- 5.1 Introduction.- 5.2 Distributed Neyman-Pearson Detection.- 5.3 Distributed CFAR Detection.- 5.4 Distributed Detection of Weak Signals.- 6 Distributed Sequential Detection.- 6.1 Introduction.- 6.2 Sequential Test Performed at the Sensors.- 6.3 Sequential Test Performed at the Fusion Center.- 7 Information Theory and Distributed Hypothesis Testing.- 7.1 Introduction.- 7.2 Distributed Detection Based on Information Theoretic Criterion.- 7.3 Multiterminal Detection with Data Compression.- Selected Bibliography.