In this paper we consider the problem of approximating frequency moments in the streaming model. Given a stream @math of numbers from @math , a frequency of @math is defined as @math . The @math -th of @math is defined as @math . In this paper we give an upper bound on the space required to find a @math -th frequency moment of @math bits that matches, up to a constant factor, the lower bound of Woodruff and Zhang (STOC 12) for constant @math and constant @math . Our algorithm makes a single pass over the stream and works for any constant @math .
In @cite_14 , the authors observed that it is possible to approximate @math in optimal polylogarithmic space. Kane, Nelson and Woodruff @cite_49 gave a space-optimal solution for @math . Kane, Nelson, and Woodruff @cite_52 gave optimal-space results for @math . In addition to the original model of @cite_14 , a variety of different models of streams have been introduced. These models include the turnstile model (that allows insertion and deletion) @cite_16 , the sliding window model @cite_13 , and the distributed model @cite_48 @cite_19 @cite_17 . In the turnstile model, where the updates can be integers in the range @math , the latest bound by Ganguly @cite_6 is @math where @math . This bound is roughly @math for constant @math . Recently, Li and Woodruff provided a matching lower bound for @math @cite_23 . Thus, for the turnstile model, the problem has been solved optimally for @math @cite_6 @cite_23 . These results combined with our result demonstrate that the turnstile model is fundamentally different from the model of Alon, Matias, and Szegedy.
@cite_14 @cite_48 @cite_52 @cite_6 @cite_19 @cite_49 @cite_23 @cite_16 @cite_13 @cite_17
1990465412 1998272044 1522285917 2950318219 2103126020 8957553 2045533739 2128846062 2017757263
This paper presents algorithms for estimating aggregate functions over a "sliding window" of the N most recent data items in one or more streams. Our results include: For a single stream, we present the first e-approximation scheme for the number of 1's in a sliding window that is optimal in both worst case time and space. We also present the first e for the sum of integers in [0..R] in a sliding window that is optimal in both worst case time and space (assuming R is at most polynomial in N). Both algorithms are deterministic and use only logarithmic memory words. In contrast, we show that an deterministic algorithm that estimates, to within a small constant relative error, the number of 1's (or the sum of integers) in a sliding window over the union of distributed streams requires O(N) space. We present the first randomized (e,s)-approximation scheme for the number of 1's in a sliding window over the union of distributed streams that uses only logarithmic memory words. We also present the first (e,s)-approximation scheme for the number of distinct values in a sliding window over distributed streams that uses only logarithmic memory words. < olOur results are obtained using a novel family of synopsis data structures. We settle the 1-pass space complexity of (1 ± e)-approximating the Lp norm, for real p with 1 ≤ p ≤ 2, of a length-n vector updated in a length-m stream with updates to its coordinates. We assume the updates are integers in the range [--M, M]. In particular, we show the space required is Θ(e-2 log(mM) + log log(ns)) bits. Our result also holds for 0 We present an algorithm for computing @math , the @math th moment of an @math -dimensional frequency vector of a data stream, for @math , to within @math factors, @math with high constant probability. Let @math be the number of stream records and @math be the largest magnitude of a stream update. The algorithm uses space in bits @math where, @math . Here @math is @math for @math and @math for @math . This improves upon the space required by current algorithms iw:stoc05,bgks:soda06,ako:arxiv10,bo:arxiv10 by a factor of at least @math . The update time is @math . We use a new technique for designing estimators for functions of the form @math , where, @math is a random variable and @math is a smooth function, based on a low-degree Taylor polynomial expansion of @math around an estimate of @math . We resolve several fundamental questions in the area of distributed functional monitoring, initiated by Cormode, Muthukrishnan, and Yi (SODA, 2008). In this model there are @math sites each tracking their input and communicating with a central coordinator that continuously maintain an approximate output to a function @math computed over the union of the inputs. The goal is to minimize the communication. We show the randomized communication complexity of estimating the number of distinct elements up to a @math factor is @math , improving the previous @math bound and matching known upper bounds up to a logarithmic factor. For the @math -th frequency moment @math , @math , we improve the previous @math communication bound to @math . We obtain similar improvements for heavy hitters, empirical entropy, and other problems. We also show that we can estimate @math , for any @math , using @math communication. This greatly improves upon the previous @math bound of Cormode, Muthukrishnan, and Yi for general @math , and their @math bound for @math . For @math , our bound resolves their main open question. Our lower bounds are based on new direct sum theorems for approximate majority, and yield significant improvements to problems in the data stream model, improving the bound for estimating @math in @math passes from @math to @math , giving the first bound for estimating @math in @math passes of @math bits of space that does not use the gap-hamming problem. We give the first optimal algorithm for estimating the number of distinct elements in a data stream, closing a long line of theoretical research on this problem begun by Flajolet and Martin in their seminal paper in FOCS 1983. This problem has applications to query optimization, Internet routing, network topology, and data mining. For a stream of indices in 1,...,n , our algorithm computes a (1 ± e)-approximation using an optimal O(1 e-2 + log(n)) bits of space with 2 3 success probability, where 0 We also give an algorithm to estimate the Hamming norm of a stream, a generalization of the number of distinct elements, which is useful in data cleaning, packet tracing, and database auditing. Our algorithm uses nearly optimal space, and has optimal O(1) update and reporting times. We show an Ω((n 1 − 2 p logM) e 2) bits of space lower bound for (1 + e)-approximating the p-th frequency moment (F_p = |x |_p^p = i=1 ^n |x_i|^p ) of a vector x ∈ − M, − M + 1, …, M n with constant probability in the turnstile model for data streams, for any p > 2 and e ≥ 1 n 1 p (we require e ≥ 1 n 1 p since there is a trivial O(n logM) upper bound). This lower bound matches the space complexity of an upper bound of Ganguly for any e 2 and e ≥ 1 n 1 p . This is again optimal for e < 1 log O(1) n. In this article, we show several results obtained by combining the use of stable distributions with pseudorandom generators for bounded space. In particular:---We show that, for any p ∈ (0, 2], one can maintain (using only O(log n e2) words of storage) a sketch C(q) of a point q ∈ lnp under dynamic updates of its coordinates. The sketch has the property that, given C(q) and C(s), one can estimate Vq − sVp up to a factor of (1 p e) with large probability. This solves the main open problem of [1999].---We show that the aforementioned sketching approach directly translates into an approximate algorithm that, for a fixed linear mapping A, and given x ∈ ℜn and y ∈ ℜm, estimates VAx − yVp in O(n p m) time, for any p ∈ (0, 2]. This generalizes an earlier algorithm of Wasserman and Blum [1997] which worked for the case p e 2.---We obtain another sketch function C′ which probabilistically embeds ln1 into a normed space lm1. The embedding guarantees that, if we set m e log(1 Δ)O(1 e), then for any pair of points q, s ∈ ln1, the distance between q and s does not increase by more than (1 p e) with constant probability, and it does not decrease by more than (1 − e) with probability 1 − Δ. This is the only known dimensionality reduction theorem for the l1 norm. In fact, stronger theorems of this type (i.e., that guarantee very low probability of expansion as well as of contraction) cannot exist [Brinkman and Charikar 2003].---We give an explicit embedding of ln2 into lnO(log n)1 with distortion (1 p 1 nΘ(1)). In the streaming model elements arrive sequentially and can be observed only once. Maintaining statistics and aggregates is an important and non-trivial task in the model. This becomes even more challenging in the sliding windows model, where statistics must be maintained only over the most recent n elements. In their pioneering paper, Datar, Gionis, Indyk and Motwani [15] presented exponential histograms, an effective method for estimating statistics on sliding windows. In this paper we present a new smooth histograms method that improves the approximation error rate obtained via exponential histograms. Furthermore, our smooth histograms method not only captures and improves multiple previous results on sliding windows bur also extends the class functions that can be approximated on sliding windows. In particular, we provide the first approximation algorithms for the following functions: Lp norms for p notin [1,2], frequency moments, length of increasing subsequence and geometric mean. In the model of continuous distributed monitoring, a number of observers each see a stream of observations. Their goal is to work together to compute a function of the union of their observations. This can be as simple as counting the total number of observations, or more complex non-linear functions such as tracking the entropy of the induced distribution. Assuming that it is too costly to simply centralize all the observations, it becomes quite challenging to design solutions which provide a good approximation to the current answer, while bounding the communication cost of the observers, and their other resources such as their space usage. This survey introduces this model, and describe a selection results in this setting, from the simple counting problem to a variety of other functions that have been studied.