Estimating mutual information from i.i.d. samples drawn from an unknown joint density function is a basic statistical problem of broad interest with multitudinous applications. The most popular estimator is one proposed by Kraskov and St "ogbauer and Grassberger (KSG) in 2004, and is nonparametric and based on the distances of each sample to its @math nearest neighboring sample, where @math is a fixed small integer. Despite its widespread use (part of scientific software packages), theoretical properties of this estimator have been largely unexplored. In this paper we demonstrate that the estimator is consistent and also identify an upper bound on the rate of convergence of the bias as a function of number of samples. We argue that the superior performance benefits of the KSG estimator stems from a curious "correlation boosting" effect and build on this intuition to modify the KSG estimator in novel ways to construct a superior estimator. As a byproduct of our investigations, we obtain nearly tight rates of convergence of the @math error of the well known fixed @math nearest neighbor estimator of differential entropy by Kozachenko and Leonenko.
In this paper we have considered the smoothness of the class of pdfs studied via bounded Hessians. In nonparametric estimation, a standard feature is to consider whole families of smooth pdfs as defined by how the differences of derivatives relate to the differences of the samples @cite_27 . Of specific interest is the H " o lder family: @math , i.e., for any tuple @math , define @math . Then for any @math such that @math , where @math is the largest integer smaller than @math , we have: for any @math . The rate of convergence of various nonparametric estimators depends on the parameter @math of the H " o lder family under consideration, cf. @cite_63 @cite_0 for recent work on convergence rate characterization of information theoretic quantities via KDE and resubstitution estimators as a function of the smoothness parameter @math . It is natural to ask if such smoothness considerations could lead to a refined understanding of the rates of convergence of the fixed @math -NN KL and KSG estimators studied here.
@cite_0 @cite_27 @cite_63
2728499573 2034177275 1997019093 2129732816
Given a nonconvex function that is an average of @math smooth functions, we design stochastic first-order methods to find its approximate stationary points. The convergence of our new methods depends on the smallest (negative) eigenvalue @math of the Hessian, a parameter that describes how nonconvex the function is. Our methods outperform known results for a range of parameter @math , and can be used to find approximate local minima. Our result implies an interesting dichotomy: there exists a threshold @math so that the currently fastest methods for @math and for @math have different behaviors: the former scales with @math and the latter scales with @math . Let φ be a smooth function of k + 2 variables. We shall investigate in this paper the rates of convergence of estimators of T(f)= fφ(f(x), f'(x),..., f (k) (x), x) dx when f belongs to some class of densities of smoothness s. We prove that, when δ ≥ 2k + 1 4, one can define an estimator T n of T(f), based on n i.i.d. observations of density f on the real line, which converges at the semiparametric rate 1 √n. On the other hand, when s < 2k + 1 4, T(f) cannot be estimated at a rate faster than n -γ with γ = 4(s-k) [4s + 1]. We shall also provide some extensions to the multidimensional case. Those results extend previous works of Levit, of Bickel and Ritov and of Donoho and Nussbaum on estimation of quadratic functionals. In a companion paper (see Self-Similarity: Part I-Splines and Operators), we characterized the class of scale-invariant convolution operators: the generalized fractional derivatives of order gamma. We used these operators to specify regularization functionals for a series of Tikhonov-like least-squares data fitting problems and proved that the general solution is a fractional spline of twice the order. We investigated the deterministic properties of these smoothing splines and proposed a fast Fourier transform (FFT)-based implementation. Here, we present an alternative stochastic formulation to further justify these fractional spline estimators. As suggested by the title, the relevant processes are those that are statistically self-similar; that is, fractional Brownian motion (fBm) and its higher order extensions. To overcome the technical difficulties due to the nonstationary character of fBm, we adopt a distributional formulation due to Gel'fand. This allows us to rigorously specify an innovation model for these fractal processes, which rests on the property that they can be whitened by suitable fractional differentiation. Using the characteristic form of the fBm, we then derive the conditional probability density function (PDF) p(BH(t)|Y), where Y= BH(k)+n[k] kisinZ are the noisy samples of the fBm BH(t) with Hurst exponent H. We find that the conditional mean is a fractional spline of degree 2H, which proves that this class of functions is indeed optimal for the estimation of fractal-like processes. The result also yields the optimal [minimum mean-square error (MMSE)] parameters for the smoothing spline estimator, as well as the connection with kriging and Wiener filtering We study the convergence properties of an alternating proximal minimization algorithm for nonconvex structured functions of the type: L(x,y)=f(x)+Q(x,y)+g(y), where f and g are proper lower semicontinuous functions, defined on Euclidean spaces, and Q is a smooth function that couples the variables x and y. The algorithm can be viewed as a proximal regularization of the usual Gauss-Seidel method to minimize L. We work in a nonconvex setting, just assuming that the function L satisfies the Kurdyka-Łojasiewicz inequality. An entire section illustrates the relevancy of such an assumption by giving examples ranging from semialgebraic geometry to “metrically regular” problems. Our main result can be stated as follows: If L has the Kurdyka-Łojasiewicz property, then each bounded sequence generated by the algorithm converges to a critical point of L. This result is completed by the study of the convergence rate of the algorithm, which depends on the geometrical properties of the function L around its critical points. When specialized to @math and to f, g indicator functions, the algorithm is an alternating projection mehod (a variant of von Neumann's) that converges for a wide class of sets including semialgebraic and tame sets, transverse smooth manifolds or sets with “regular” intersection. To illustrate our results with concrete problems, we provide a convergent proximal reweighted l1 algorithm for compressive sensing and an application to rank reduction problems.