The Shannon entropy is a widely used summary statistic, for example, network traffic measurement, anomaly detection, neural computations, spike trains, etc. This study focuses on estimating Shannon entropy of data streams. It is known that Shannon entropy can be approximated by Reenyi entropy or Tsallis entropy, which are both functions of the p-th frequency moments and approach Shannon entropy as p->1. Compressed Counting (CC) is a new method for approximating the p-th frequency moments of data streams. Our contributions include: 1) We prove that Renyi entropy is (much) better than Tsallis entropy for approximating Shannon entropy. 2) We propose the optimal quantile estimator for CC, which considerably improves the previous estimators. 3) Our experiments demonstrate that CC is indeed highly effective approximating the moments and entropies. We also demonstrate the crucial importance of utilizing the variance-bias trade-off.
CC improves symmetric stable random projections @cite_4 @cite_9 uniformly for all @math as shown in Figure in Section . However, one can still considerably improve CC around @math , by developing better estimators, as in this study. In addition, no empirical studies on CC were reported.
@cite_9 @cite_4
2014689751 2045533739
!# @math ) 4 ' " + @math 4 ) @ & A B C D E . * . ! F ' G H ' IJ K L * M & + *) N z'cC g) ' &  p : q :  r Htd|#t9 ,s tvuw x yFz |;w]tS 9 x |;   C n+ U <dU' 4 2 In this paper we introduce the Cognitive Compressive Sensing (CCS) problem, modeling a Cognitive Receiver (CR) that optimizes the @math projections of a @math dimensional vector dynamically, by optimizing the objective of correctly detecting the maximum number of idle entries, while updating each time its Bayesian beliefs on the future vector realizations. We formulate and study the CCS as a Restless Multi-Armed Bandit problem, generalizing the popular Cognitive Spectrum Sensing model, in which the CR can sense @math out of the @math sub-channels and propose a novel adaptive Finite Rate of Innovation (FRI) sampling method based on the CCS approach. While in general the optimum policy remains elusive, we provide sufficient conditions in which in the limit for large @math and @math the greedy policy is optimum. Numerical results corroborate our theoretical findings.