We consider a basic problem in unsupervised learning: learning an unknown . A Poisson Binomial Distribution (PBD) over @math is the distribution of a sum of @math independent Bernoulli random variables which may have arbitrary, potentially non-equal, expectations. These distributions were first studied by S. Poisson in 1837 Poisson:37 and are a natural @math -parameter generalization of the familiar Binomial Distribution. Surprisingly, prior to our work this basic learning problem was poorly understood, and known results for it were far from optimal. We essentially settle the complexity of the learning problem for this basic class of distributions. As our first main result we give a highly efficient algorithm which learns to @math -accuracy (with respect to the total variation distance) using @math samples . The running time of the algorithm is in the size of its input data, i.e., @math bit-operations. (Observe that each draw from the distribution is a @math -bit string.) Our second main result is a proper learning algorithm that learns to @math -accuracy using @math samples, and runs in time @math . This is nearly optimal, since any algorithm for this problem must use @math samples. We also give positive and negative results for some extensions of this learning problem to weighted sums of independent Bernoulli random variables.
Many results in probability theory study approximations to the Poisson Binomial distribution via simpler distributions. In a well-known result, Le Cam @cite_24 shows that for any PBD @math with @math [ (X, (p_1 + + p_n)) 2 i=1 ^n p_i^2, ] where @math denotes the Poisson distribution with parameter @math . Subsequently many other proofs of this result and similar ones were given using a range of different techniques; @cite_28 @cite_4 @cite_14 @cite_10 is a sampling of work along these lines, and Steele @cite_12 gives an extensive list of relevant references. Significant work has also been done on approximating PBDs by normal distributions (see e.g. @cite_37 @cite_21 @cite_22 @cite_40 ) and by Binomial distributions (see e.g. @cite_15 @cite_9 @cite_38 ). These results provide structural information about PBDs that can be well-approximated via simpler distributions, but fall short of our goal of obtaining approximations of a general, unknown PBD up to an arbitrary accuracy . Indeed, the approximations obtained in the probability literature (such as, the Poisson, Normal and Binomial approximations) typically depend on the first few moments of the target PBD, while higher moments are crucial for arbitrary approximation @cite_38 .
@cite_38 @cite_37 @cite_14 @cite_4 @cite_22 @cite_28 @cite_9 @cite_21 @cite_24 @cite_40 @cite_15 @cite_10 @cite_12
2026924428 2337787166 2419099043 2048587146
where (an)n ENo is some sequence of nonnegative numbers, (Sn),nENo is the sequence of partial sums, S0 = 0, Sn = XflXk, of another sequence (Xk)kEN of i.i.d. random variables, and A c R is a fixed Borel set such as [0,1] or [0, oo). Examples of such convolution series are subordinated distributions (f=0Oan = 1) which arise as distributions of random sums, and harmonic and ordinary renewal measures (a0 = 0, an = 1 n for all n C N in the first, an = 1 for all n C NO in the second case). These examples are in turn essential for the analysis of the large time behaviour of diverse applied models such as branching and queueing processes, they are also of interest in connection with representation theorems such as the Levy representation of infinitely divisible distributions. A traditional approach to such problems is via regular variation: If the underlying random variables are nonnegative we can use Laplace transforms and the related Abelian and Tauberian theorems [see, e.g., Stam (1973) in the context of subordination and Feller (1971, XIV.3) in connection with renewal theory; Embrechts, Maejima, and Omey (1984) is a recent treatment of generalized renewal measures along these lines]. The approach of the present paper is based on the Wiener-Levy-Gel'fand theorem and has occasionally been called the Banach algebra method. In Gruibel (1983) we gave a new variant of this method for the special case of lattice distributions, showing that by using the appropriate Banach algebras of sequences, arbitrarily fine expansions are possible under certain assumptions on the higher-order differences of (P(X1 = n))fnEN. Here we give a corresponding treatment of nonlattice distributions. We restrict ourselves to an analogue of first-order differences and obtain a number of theorems which perhaps are described best as next-term results. To explain this let us consider a special case in more detail. A binomial approximation theorem for dependent indicators using Stein's method and coupling is proved. The approximating binomial distribution B(n � ,p � ) is chosen in such a way that its first moment is equal to that of W and its variance is asymptotically equal to that of W as ntends to infinity where W is the sum of independent indicators and pis bounded away from 1. Three examples, one of which concerns two different approximations for the hypergeometric distribution, are given to illustrate applications of the theorem obtained. We consider the following basic learning task: given independent draws from an unknown distribution over a discrete support, output an approximation of the distribution that is as accurate as possible in L1 distance (equivalently, total variation distance, or "statistical distance"). Perhaps surprisingly, it is often possible to "de-noise" the empirical distribution of the samples to return an approximation of the true distribution that is significantly more accurate than the empirical distribution, without relying on any prior assumptions on the distribution. We present an instance optimal learning algorithm which optimally performs this de-noising for every distribution for which such a de-noising is possible. More formally, given n independent draws from a distribution p, our algorithm returns a labelled vector whose expected distance from p is equal to the minimum possible expected error that could be obtained by any algorithm, even one that is given the true unlabeled vector of probabilities of distribution p and simply needs to assign labels---up to an additive subconstant term that is independent of p and goes to zero as n gets large. This somewhat surprising result has several conceptual implications, including the fact that, for any large sample from a distribution over discrete support, prior knowledge of the rates of decay of the tails of the distribution (e.g. power-law type assumptions) is not significantly helpful for the task of learning the distribution. As a consequence of our techniques, we also show that given a set of n samples from an arbitrary distribution, one can accurately estimate the expected number of distinct elements that will be observed in a sample of any size up to n log n. This sort of extrapolation is practically relevant, particularly to domains such as genomics where it is important to understand how much more might be discovered given larger sample sizes, and we are optimistic that our approach is practically viable. Let X i (i=1,2, …) be a sequence of subexponential positive independent and identically distributed random variables. In this paper, we offer two alternative approaches to obtain higher-order expansions of the tail of and subsequently for ruin probabilities in renewal risk models with claim sizes X i . In particular, these emphasize the importance of the term for the accuracy of the resulting asymptotic expansion of . Furthermore, we present a more rigorous approach to the often suggested technique of using approximations with shifted arguments. The cases of a Pareto type, Weibull and Lognormal distribution for X i are discussed in more detail and numerical investigations of the increase in accuracy by including higher-order terms in the approximation of ruin probabilities for finite realistic ranges of s are given.