Information divergence functions play a critical role in statistics and information theory. In this paper we show that a non-parametric f-divergence measure can be used to provide improved bounds on the minimum binary classification probability of error for the case when the training and test data are drawn from the same distribution and for the case where there exists some mismatch between training and test distributions. We confirm the theoretical results by designing feature selection algorithms using the criteria from these bounds and by evaluating the algorithms on a series of pathological speech classification tasks.
Beyond the bounds on the BER based on the divergence measures, a number of other bounds exist based on different functionals of the distributions. In @cite_25 , the authors derive a new functional based on a Gaussian-Weighted sinusoid that yields tighter bounds on the BER than other popular approaches. Avi-Itzhak proposes arbitrarily tight bounds on the BER in @cite_19 . Both of these sets of bounds are tighter than the bounds we derive here; however, these bounds cannot be estimated without at least partial knowledge of the underlying distribution. A strength of the bounds proposed in this paper is that they are empirically estimable without knowing a parametric model for the underlying distribution.
@cite_19 @cite_25
1673728053 2146950091 2133027741 2009274429
We consider nonparametric estimation of L2, Renyi-α and Tsallis-α divergences between continuous distributions. Our approach is to construct estimators for particular integral functionals of two densities and translate them into divergence estimators. For the integral functionals, our estimators are based on corrections of a preliminary plug-in estimator. We show that these estimators achieve the parametric convergence rate of n-1 2 when the densities' smoothness, s, are both at least d 4 where d is the dimension. We also derive minimax lower bounds for this problem which confirm that s > d 4 is necessary to achieve the n-1 2 rate of convergence. We validate our theoretical guarantees with a number of simulations. A novel class of information-theoretic divergence measures based on the Shannon entropy is introduced. Unlike the well-known Kullback divergences, the new measures do not require the condition of absolute continuity to be satisfied by the probability distributions involved. More importantly, their close relationship with the variational distance and the probability of misclassification error are established in terms of bounds. These bounds are crucial in many applications of divergence measures. The measures are also well characterized by the properties of nonnegativity, finiteness, semiboundedness, and boundedness. > These lecture notes highlight the mathematical and computational structure relating to the formulation of, and development of algorithms for, the Bayesian approach to inverse problems in differential equations. This approach is fundamental in the quantification of uncertainty within applications in volving the blending of mathematical models with data. The finite dimensional situation is described first, along with some motivational examples. Then the development of probability measures on separable Banach space is undertaken, using a random series over an infinite set of functions to construct draws; these probability measures are used as priors in the Bayesian approach to inverse problems. Regularity of draws from the priors is studied in the natural Sobolev or Besov spaces implied by the choice of functions in the random series construction, and the Kolmogorov continuity theorem is used to extend regularity considerations to the space of Holder continuous functions. Bayes’ theorem is de rived in this prior setting, and here interpreted as finding conditions under which the posterior is absolutely continuous with respect to the prior, and determining a formula for the Radon-Nikodym derivative in terms of the likelihood of the data. Having established the form of the posterior, we then describe various properties common to it in the infinite dimensional setting. These properties include well-posedness, approximation theory, and the existence of maximum a posteriori estimators. We then describe measure-preserving dynamics, again on the infinite dimensional space, including Markov chain-Monte C arlo and sequential Monte Carlo methods, and measure-preserving reversible stochastic differential equations. By formulating the theory and algorithms on the underlying infinite dimensional space, we obtain a framework suitable for rigorous analysis of the accuracy of reconstructions, of computational complexity, as well as naturally constructing algorithms which perform well under mesh refinement, since they are inherently well-defined in infinite dimensions. We introduce a family of new transforms based on imitating the proximal mapping of Moreau and the associated Moreau-Yosida proximal approximation of a function. The transforms are constructed in terms of the AÂ†-divergence functional a generalization of the relative entropy and of Bregman's measure of distance. An analogue of Moreau's theorem associated with these entropy-like distances is proved. We show that the resulting Entropic Proximal Maps share properties similar to the proximal mapping and provide a fairly general framework for constructing approximation and smoothing schemes for optimization problems. Applications of the results to the construction of generalized augmented Lagrangians for nonlinear programs and the minimax problem are presented.