Kernel regression is an essential and ubiquitous tool for non-parametric data analysis, particularly popular among time series and spatial data. However, the central operation which is performed many times, evaluating a kernel on the data set, takes linear time. This is impractical for modern large data sets. In this paper we describe coresets for kernel regression: compressed data sets which can be used as proxy for the original data and have provably bounded worst case error. The size of the coresets are independent of the raw number of data points, rather they only depend on the error guarantee, and in some cases the size of domain and amount of smoothing. We evaluate our methods on very large time series and spatial data, and demonstrate that they incur negligible error, can be constructed extremely efficiently, and allow for great computational gains.
This is the first work to address sample complexity and coreset size for Nadaraya-Watson kernel regression. There is an enormous body of work on other types of coresets, see the recent survey on coresets @cite_23 , including many for parametric regression variants like least-square regression @cite_19 and @math regression @cite_7 .
@cite_19 @cite_7 @cite_23
1992065791 2010371250 2230030897
We study the (constrained) least-squares regression as well as multiple response least-squares regression and ask the question of whether a subset of the data, a coreset, suffices to compute a good approximate solution to the regression. We give deterministic, low-order polynomial-time algorithms to construct such coresets with approximation guarantees, together with lower bounds indicating that there is not much room for improvement upon our results. The @math regression problem takes as input a matrix @math , a vector @math , and a number @math , and it returns as output a number @math and a vector @math such that @math . In this paper, we construct coresets and obtain an efficient two-stage sampling-based approximation algorithm for the very overconstrained ( @math ) version of this classical problem, for all @math . The first stage of our algorithm nonuniformly samples @math rows of @math and the corresponding elements of @math , and then it solves the @math regression problem on the sample; we prove this is an 8-approximation. The second stage of our algorithm uses the output of the first stage to resample @math constraints, and then it solves the @math regression problem on the new sample; we prove this is a @math -approximation. Our algorithm unifies, improves upon, and extends the existing algorithms for special cases of @math regression, namely, @math [K. L. Clarkson, in Proceedings of the 16th Annual ACM-SIAM Symposium on Discrete Algorithms, ACM, New York, SIAM, Philadelphia, 2005, pp. 257-266; P. Drineas, M. W. Mahoney, and S. Muthukrishnan, in Proceedings of the 17th Annual ACM-SIAM Symposium on Discrete Algorithms, ACM, New York, SIAM, Philadelphia, 2006, pp. 1127-1136]. In the course of proving our result, we develop two concepts—well-conditioned bases and subspace-preserving sampling—that are of independent interest. Geometric data summarization has become an essential tool in both geometric approximation algorithms and where geometry intersects with big data problems. In linear or near-linear time large data sets can be compressed into a summary, and then more intricate algorithms can be run on the summaries whose results approximate those of the full data set. Coresets and sketches are the two most important classes of these summaries. We survey five types of coresets and sketches: shape-fitting, density estimation, high-dimensional vectors, high-dimensional point sets matrices, and clustering.