In statistical inference problems, we wish to obtain lower bounds on the minimax risk, that is to bound the performance of any possible estimator. A standard technique to obtain risk lower bounds involves the use of Fano's inequality. In an information-theoretic setting, it is known that Fano's inequality typically does not give a sharp converse result (error lower bound) for channel coding problems. Moreover, recent work has shown that an argument based on binary hypothesis testing gives tighter results. We adapt this technique to the statistical setting, and argue that Fano's inequality can always be replaced by this approach to obtain tighter lower bounds that can be easily computed and are asymptotically sharp. We illustrate our technique in three applications: density estimation, active learning of a binary classifier, and compressed sensing, obtaining tighter risk lower bounds in each case.
Using Fano's inequality, Yang and Barron @cite_2 obtained order-optimal minimax risk lower bounds that depend only on global metric entropy features of the underlying function class, without explicitly constructing a packing set. The required metric entropy features (bounds on the packing number and covering number) are available from results in approximation theory for many function classes of interest. Guntuboyina @cite_1 obtained a lower bound on the average error probability in terms of general @math -divergences, and also generalized the metric entropy results of Yang and Barron @cite_2 to certain @math -divergences such as the @math -divergence. An interesting direction for future work would be to obtain a non-asymptotic result analogous to Theorem for the case where only the global metric entropy features are available.
@cite_1 @cite_2
2101210333 1524622012
Lower bounds involving f-divergences between the underlying probability measures are proved for the minimax risk in estimation problems. Our proofs just use simple convexity facts. Special cases and straightforward corollaries of our bounds include well known inequalities for establishing minimax lower bounds such as Fano's inequality, Pinsker's inequality and inequalities based on global entropy conditions. Two applications are provided: a new minimax lower bound for the reconstruction of convex bodies from noisy support function measurements and a different proof of a recent minimax lower bound for the estimation of a covariance matrix. Deep neural nets have caused a revolution in many classification tasks. A related ongoing revolution---also theoretically not understood---concerns their ability to serve as generative models for complicated types of data such as images and texts. These models are trained using ideas like variational autoencoders and Generative Adversarial Networks. We take a first cut at explaining the expressivity of multilayer nets by giving a sufficient criterion for a function to be approximable by a neural network with @math hidden layers. A key ingredient is Barron's Theorem Barron1993 , which gives a Fourier criterion for approximability of a function by a neural network with 1 hidden layer. We show that a composition of @math functions which satisfy certain Fourier conditions ("Barron functions") can be approximated by a @math -layer neural network. For probability distributions, this translates into a criterion for a probability distribution to be approximable in Wasserstein distance---a natural metric on probability distributions---by a neural network applied to a fixed base distribution (e.g., multivariate gaussian). Building up recent lower bound work, we also give an example function that shows that composition of Barron functions is more expressive than Barron functions alone.