Let @math be a class of probability distributions over the discrete domain @math We show that if @math satisfies a rather general condition -- essentially, that each distribution in @math can be well-approximated by a variable-width histogram with few bins -- then there is a highly efficient (both in terms of running time and sample complexity) algorithm that can learn any mixture of @math unknown distributions from @math We analyze several natural types of distributions over @math , including log-concave, monotone hazard rate and unimodal distributions, and show that they have the required structural property of being well-approximated by a histogram with few bins. Applying our general algorithm, we obtain near-optimally efficient algorithms for all these mixture learning problems.
Unimodal distributions: The problem of learning a single unimodal distribution is well-understood: Birg ' e @cite_10 gave an efficient algorithm for learning continuous unimodal distributions (whose density is absolutely bounded); his algorithm, when translated to the discrete domain @math , requires @math samples. This sample size is also known to be optimal (up to constant factors) @cite_43 . In recent work, @cite_16 gave an efficient algorithm to learn @math -modal distributions over @math . We remark that their result does not imply ours, as even a mixture of two unimodal distributions over @math may have @math modes. We are not aware of prior work on efficiently learning mixtures of unimodal distributions.
@cite_43 @cite_16 @cite_10
2014255551 2952993179 2126204693
On considere la classe de toutes les densites unimodales definies sur un intervalle de longueur L et borne par H; on etudie le risque minimax sur cette classe, lorsqu'on estime en utilisant n observations i.i.d., la perte etant mesuree par la distance L 1 entre l'estimateur et la densite reelle A @math -modal probability distribution over the discrete domain @math is one whose histogram has at most @math "peaks" and "valleys." Such distributions are natural generalizations of monotone ( @math ) and unimodal ( @math ) probability distributions, which have been intensively studied in probability theory and statistics. In this paper we consider the problem of (i.e., performing density estimation of) an unknown @math -modal distribution with respect to the @math distance. The learning algorithm is given access to independent samples drawn from an unknown @math -modal distribution @math , and it must output a hypothesis distribution @math such that with high probability the total variation distance between @math and @math is at most @math Our main goal is to obtain algorithms for this problem that use (close to) an information-theoretically optimal number of samples. We give an efficient algorithm for this problem that runs in time @math . For @math , the number of samples used by our algorithm is very close (within an @math factor) to being information-theoretically optimal. Prior to this work computationally efficient algorithms were known only for the cases @math Birge:87b,Birge:97 . A novel feature of our approach is that our learning algorithm crucially uses a new algorithm for as a key subroutine. The learning algorithm uses the property tester to efficiently decompose the @math -modal distribution into @math (near-)monotone distributions, which are easier to learn. The Grenander estimator of a decreasing density, which is defined as the derivative of the concave envelope of the empirical c.d.f., is known to be a very good estimator of an unknown decreasing density on the half-line R + when this density is not assumed to be smooth. It is indeed the maximum likelihood estimator and one can get precise upper bounds for its risk when the loss is measured by the L 1 -distance between densities. Moreover, if one restricts oneself to the compact subsets of decreasing densities bounded by H with support on [0, L] the risk of this estimator is within a fixed factor of the minimax risk. The same is true if one deals with the maximum likelihood estimator for unimodal densities with known mode. When the mode is unknown, the maximum likelihood estimator does not exist any more. We shall provide a general purpose estimator (together with a computational algorithm) for estimating nonsmooth unimodal densities. Its risk is the same as the risk of the Grenander estimator based on the knowledge of the true mode plus some lower order term. It can also cope with small departures from unimodality.