Recently, penalties promoting signals that are sparse within and across groups have been proposed. In this letter, we propose a generalization that allows to encode more intricate dependencies within groups. However, this complicates the realization of the threshold function associated with the penalty, which hinders the use of the penalty in energy minimization. We discuss how to sidestep this problem, and demonstrate the use of the modified penalty in an energy minimization formulation for an inverse problem.
The proposed modification to the SWAG penalty aims to introduce further flexibility in forming the groups. First, groups are allowed to overlap. Second, while the original SWAG penalty in @cite_24 uses constant weights within each group, the modified penalty allows the weights within a group to vary. These in turn allow to achieve a more localized and translation-invariant behavior, which is of interest for processing time-domain signals. However, these modifications come at an expense. While it is possible to realize the SWAG threshold function with a finite terminating procedure @cite_24 , such a procedure is not available for the proposed penalty. Therefore, forward-backward splitting type algorithms that might utilize @math @cite_18 @cite_15 @cite_9 @cite_21 are not readily applicable for the proposed penalty. We propose instead a descent algorithm for a generic formulation that employs the proposed penalty. This algorithm is specific to the proposed penalty, and makes use of the quadratic nature of the penalty. Therefore, it has not appeared elsewhere in the literature as far as we are aware.
@cite_18 @cite_9 @cite_21 @cite_24 @cite_15
1994309289 1989060270 2529812697 2460144244
Extracting useful information from high-dimensional data is an important focus of today's statistical research and practice. Penalized loss function minimization has been shown to be effective for this task both theoretically and empirically. With the virtues of both regularization and sparsity, the L 1 -penalized squared error minimization method Lasso has been popular in regression models and beyond. In this paper, we combine different norms including L 1 to form an intelligent penalty in order to add side information to the fitting of a regression or classification model to obtain reasonable estimates. Specifically, we introduce the Composite Absolute Penalties (CAP) family, which allows given grouping and hierarchical relationships between the predictors to be expressed. CAP penalties are built by defining groups and combining the properties of norm penalties at the across-group and within-group levels. Grouped selection occurs for nonoverlapping groups. Hierarchical variable selection is reached by defining groups with particular overlapping patterns. We propose using the BLASSO and cross-validation to compute CAP estimates in general. For a subfamily of CAP estimates involving only the L 1 and L ∞ norms, we introduce the iCAP algorithm to trace the entire regularization path for the grouped selection problem. Within this subfamily, unbiased estimates of the degrees of freedom (df) are derived so that the regularization parameter is selected without cross-validation. CAP is shown to improve on the predictive performance of the LASSO in a series of simulated experiments, including cases with p » n and possibly mis-specified groupings. When the complexity of a model is properly calculated, iCAP is seen to be parsimonious in the experiments. We study the problem of estimating high-dimensional regression models regularized by a structured sparsity-inducing penalty that encodes prior structural information on either the input or output variables. We consider two widely adopted types of penalties of this kind as motivating examples: (1) the general overlapping-group-lasso penalty, generalized from the group-lasso penalty; and (2) the graph-guided-fused-lasso penalty, generalized from the fused-lasso penalty. For both types of penalties, due to their nonseparability and nonsmoothness, developing an efficient optimization method remains a challenging problem. In this paper we propose a general optimization approach, the smoothing proximal gradient (SPG) method, which can solve structured sparse regression problems with any smooth convex loss under a wide spectrum of structured sparsity-inducing penalties. Our approach combines a smoothing technique with an effective proximal gradient method. It achieves a convergence rate significantly faster than the standard first-order methods, subgradient methods, and is much more scalable than the most widely used interior-point methods. The efficiency and scalability of our method are demonstrated on both simulation experiments and real genetic data sets. We introduce a new penalty function that promotes signals composed of a small number of active groups, where within each group, only a few high magnitude coefficients are nonzero. We derive the threshold function associated with the proposed penalty and study its properties. We discuss how the proposed penalty threshold function can be useful for signals with isolated nonzeros, such as audio with isolated harmonics along the frequency axis, or reflection functions in exploration seismology where the nonzeros occur on the boundaries of subsoil layers. We demonstrate the use of the proposed penalty threshold functions in a convex denoising and a nonconvex deconvolution formulation. We provide convergent algorithms for both formulations and compare the performance with state-of-the-art methods. In this paper, we address the challenging task of simultaneously optimizing (i) the weights of a neural network, (ii) the number of neurons for each hidden layer, and (iii) the subset of active input features (i.e., feature selection). While these problems are traditionally dealt with separately, we propose an efficient regularized formulation enabling their simultaneous parallel execution, using standard optimization routines. Specifically, we extend the group Lasso penalty, originally proposed in the linear regression literature, to impose group-level sparsity on the networks connections, where each group is defined as the set of outgoing weights from a unit. Depending on the specific case, the weights can be related to an input variable, to a hidden neuron, or to a bias unit, thus performing simultaneously all the aforementioned tasks in order to obtain a compact network. We carry out an extensive experimental evaluation, in comparison with classical weight decay and Lasso penalties, both on a toy dataset for handwritten digit recognition, and multiple realistic mid-scale classification benchmarks. Comparative results demonstrate the potential of our proposed sparse group Lasso penalty in producing extremely compact networks, with a significantly lower number of input features, with a classification accuracy which is equal or only slightly inferior to standard regularization terms.