The problem of statistical learning is to construct a predictor of a random variable Y as a function of a related random variable X on the basis of an i.i.d. training sample from the joint distribution of (X, Y). Allowable predictors are drawn from some specified class, and the goal is to approach asymptotically the performance (expected loss) of the best predictor in the class. We consider the setting in which one has perfect observation of the X-part of the sample, while the Y-part has to be communicated at some finite bit rate. The encoding of the Y-values is allowed to depend on the X-values. Under suitable regularity conditions on the admissible predictors, the underlying family of probability distributions and the loss function, we give an information-theoretic characterization of achievable predictor performance in terms of conditional distortion-rate functions. The ideas are illustrated on the example of nonparametric regression in Gaussian noise.
Previously, the problem of statistical estimation from compressed data was considered by Zhang and Berger @cite_5 , Ahlswede and Burnashev @cite_3 and Han and Amari @cite_2 from the viewpoint of multiterminal information theory. In these papers, the underlying family of distributions of @math is parametric, i.e., of the form @math , where @math is a subset of @math for some finite @math , and one wishes to estimate the true" parameter @math . The i.i.d. observations @math are drawn from @math , and the input part @math is communicated to the statistician at some rate @math , while the output part @math is communicated at some rate @math . The present work generalizes to the nonparametric setting the case considered by Ahlswede and Burnashev @cite_3 , namely when @math . To the best of the author's knowledge, this paper is the first to consider the problem of nonparametric learning from compressed observations with side information.
@cite_5 @cite_3 @cite_2
2095374884 2071730188 2004945083 2494296723
We introduce and investigate a new model of learning probability distributions from independent draws. Our model is inspired by the popular Probably Approximately Correct (PAC) model for learning boolean functions from labeled examples [24], in the sense that we emphasize efficient and approximate learning, and we study the learnability of restricted classes of target distributions. The dist ribut ion classes we examine are often defined by some simple computational mechanism for transforming a truly random string of input bits (which is not visible to the learning algorithm) into the stochastic observation (output) seen by the learning algorithm. In this paper, we concentrate on discrete distributions over O, I n. The problem of inferring an approximation to an unknown probability distribution on the basis of independent draws has a long and complex history in the pattern recognition and statistics literature. For instance, the problem of estimating the parameters of a Gaussian density in highdimensional space is one of the most studied statistical problems. Distribution learning problems have often been investigated in the context of unsupervised learning, in which a linear mixture of two or more distributions is generating the observations, and the final goal is not to model the distributions themselves, but to predict from which distribution each observation was drawn. Data clustering methods are a common tool here. There is also a large literature on nonpararnetric density estimation, in which no assumptions are made on the unknown target density. Nearest-neighbor approaches to the unsupervised learning problem often arise in the nonparametric setting. While we obviously cannot do justice to these areas here, the books of Duda and Hart [9] and Vapnik [25] provide excellent overviews and introductions to the pattern recognition work, as well as many pointers for further reading. See also Izenman’s recent survey article [16]. Roughly speaking, our work departs from the traditional statistical and pattern recognition approaches in two ways. First, we place explicit emphasis on the comput ationrd complexity of distribution learning. It seems fair to say that while previous research has provided an excellent understanding of the information-theoretic issues involved in dis- In this paper, we propose an efficient algorithm, called vector field consensus, for establishing robust point correspondences between two sets of points. Our algorithm starts by creating a set of putative correspondences which can contain a very large number of false correspondences, or outliers, in addition to a limited number of true correspondences (inliers). Next, we solve for correspondence by interpolating a vector field between the two point sets, which involves estimating a consensus of inlier points whose matching follows a nonparametric geometrical constraint. We formulate this a maximum a posteriori (MAP) estimation of a Bayesian model with hidden latent variables indicating whether matches in the putative set are outliers or inliers. We impose nonparametric geometrical constraints on the correspondence, as a prior distribution, using Tikhonov regularizers in a reproducing kernel Hilbert space. MAP estimation is performed by the EM algorithm which by also estimating the variance of the prior model (initialized to a large value) is able to obtain good estimates very quickly (e.g., avoiding many of the local minima inherent in this formulation). We illustrate this method on data sets in 2D and 3D and demonstrate that it is robust to a very large number of outliers (even up to 90 ). We also show that in the special case where there is an underlying parametric geometrical model (e.g., the epipolar line constraint) that we obtain better results than standard alternatives like RANSAC if a large number of outliers are present. This suggests a two-stage strategy, where we use our nonparametric model to reduce the size of the putative set and then apply a parametric variant of our approach to estimate the geometric parameters. Our algorithm is computationally efficient and we provide code for others to use it. In addition, our approach is general and can be applied to other problems, such as learning with a badly corrupted training data set. This paper develops a nonparametric density estimator with parametric overtones. Suppose f(x, θ) is some family of densities, indexed by a vector of parameters θ. We define a local kernel-smoothed likelihood function which, for each x, can be used to estimate the best local parametric approximant to the true density. This leads to a new density estimator of the form f(x, o(x)), thus inserting the best local parameter estimate for each new value of x. When the bandwidth used is large, this amounts to ordinary full likelihood parametric density estimation, while for moderate and small bandwidths the method is essentially nonparametric, using only local properties of data and the model. Alternative ways more general than via the local likelihood are also described. The methods can be seen as ways of nonparametrically smoothing the parameter within a parametric class. Properties of this new semiparametric estimator are investigated. Our preferred version has approximately the same variance as the ordinary kernel method but potentially a smaller bias. The new method is seen to perform better than the traditional kernel method in a broad nonparametric vicinity of the parametric model employed, while at the same time being capable of not losing much in precision to full likelihood methods when the model is correct. Other versions of the method are approximately equivalent to using particular higher order kernels in a semiparametric framework. The methodology we develop can be seen as the density estimation parallel to local likelihood and local weighted least squares theory in nonparametric regression. In some statistical non-parametric models the mean of the random variable y has to satisfy specific constraints. We consider the case where the set defined by the constraints is a closed polyhedral cone K in Rk. For example, when the mean is required to be concave in x, the set of acceptable means is a closed convex cone defined by k-2 linear inequalities in Rk. The least squares estimate of the mean is then the projection of the data point y on the cone K. In this paper, we present an algorithm to find the least square estimate of the mean in a finite number of steps. Other algorithms to solve this problem have been given before. The successive approximations in such algorithms are usually points on the faces of K. The solution here is reached by following a fixed line joining an abitrary but suitably chosen initial point y0 to the data point y. The 1- dimensional subspace spanned by the generators of the cone K is divided into 2' regions which can be described as the set of points with non-negative coordinates in mixed primal-dual bases relative to the cone K. Any point y belongs to one and only one of these regions SJ with corresponding basis 3j. The projection of y on K is immediately obtained from the expression of y in Oj by dropping the dual component of y.