Performance of machine learning approaches depends strongly on the choice of misfit penalty, and correct choice of penalty parameters, such as the threshold of the Huber function. These parameters are typically chosen using expert knowledge, cross-validation, or black-box optimization, which are time consuming for large-scale applications. We present a principled, data-driven approach to simultaneously learn the model pa- rameters and the misfit penalty parameters. We discuss theoretical properties of these joint inference problems, and develop algorithms for their solution. We show synthetic examples of automatic parameter tuning for piecewise linear-quadratic (PLQ) penalties, and use the approach to develop a self-tuning robust PCA formulation for background separation.
The most relevant works related to this paper focus on the relation between the quantile penalty and the asymmetric Laplace distribution(ALD) . @cite_2 jointly estimate the model and the shape parameters for quantile penalty, and @cite_1 infer the joint posterior distribution of these parameters.
@cite_1 @cite_2
2096315701 2078129874 2303223340 2414843940
In this paper we study sparsity-inducing nonconvex penalty functions using Levy processes. We define such a penalty as the Laplace exponent of a subordinator. Accordingly, we propose a novel approach for the construction of sparsity-inducing nonconvex penalties. Particularly, we show that the nonconvex logarithmic (LOG) and exponential (EXP) penalty functions are the Laplace exponents of Gamma and compound Poisson subordinators, respectively. Additionally, we explore the concave conjugate of nonconvex penalties. We find that the LOG and EXP penalties are the concave conjugates of negative Kullback-Leiber (KL) distance functions. Furthermore, the relationship between these two penalties is due to asymmetricity of the KL distance. SUMMARY It is shown how, in regular parametric problems, the first-order term is removed from the asymptotic bias of maximum likelihood estimates by a suitable modification of the score function. In exponential families with canonical parameterization the effect is to penalize the likelihood by the Jeffreys invariant prior. In binomial logistic models, Poisson log linear models and certain other generalized linear models, the Jeffreys prior penalty function can be imposed in standard regression software using a scheme of iterative adjustments to the data. We propose a new method for pointwise estimation of monotone, uni- modal and U-shaped failure rates, under a right-censoring mechanism, using non- parametric likelihood ratios. The asymptotic distribution of the likelihood ratio is pivotal, though non-standard, and can therefore be used to construct asymptotic confidence intervals for the failure rate at a point of interest, via inversion. Major advantages of the new method lie in the facts that it completely avoids estima- tion of nuisance parameters, or the choice of a bandwidth tuning parameter, and is extremely easy to implement. The new method is shown to perform competi- tively in simulations, and is illustrated on a data set involving time to diagnosis of schizophrenia in the Jerusalem Perinatal Cohort Schizophrenia Study. In this paper, I review the main results on the asymptotic properties of the posterior distribution in nonparametric or high-dimensional models. In particular, I explain how posterior concentration rates can be derived and what we learn from such analysis in terms of the impact of the prior distribution on high-dimensional models. These results concern fully Bayes and empirical Bayes procedures. I also describe some of the results that have been obtained recently in semiparametric models, focusing mainly on the Bernstein–von Mises property. Although these results are theoretical in nature, they shed light on some subtle behaviors of the prior models and sharpen our understanding of the family of functionals that can be well estimated for a given prior model.