f-divergence estimation is an important problem in the fields of information theory, machine learning, and statistics. While several divergence estimators exist, relatively few of their convergence rates are known. We derive the MSE convergence rate for a density plug-in estimator of f-divergence. Then by applying the theory of optimally weighted ensemble estimation, we derive a divergence estimator with a convergence rate of O(1 T) that is simple to implement and performs well in high dimensions. We validate our theoretical results with experiments.
Singh and P ' o czos @cite_2 provided an estimator for R ' e nyi- @math divergences that uses a mirror image'' kernel density estimator. They prove a convergence rate of @math when @math for each of the densities. However this method requires several computations at each boundary of the support of the densities which becomes difficult to implement as @math gets large. Also, this method requires knowledge of the support of the densities which may not be possible for some problems.
@cite_2
2963881189
Estimating divergences in a consistent way is of great importance in many machine learning tasks. Although this is a fundamental problem in nonparametric statistics, to the best of our knowledge there has been no finite sample exponential inequality convergence bound derived for any divergence estimators. The main contribution of our work is to provide such a bound for an estimator of Renyi-α divergence for a smooth Holder class of densities on the d-dimensional unit cube [0, 1]d. We also illustrate our theoretical results with a numerical experiment.