Measuring the relationship between any pair of variables is a rich and active area of research that is central to scientific practice. In contrast, characterizing the common information among any group of variables is typically a theoretical exercise with few practical methods for high-dimensional data. A promising solution would be a multivariate generalization of the famous Wyner common information, but this approach relies on solving an apparently intractable optimization problem. We leverage the recently introduced information sieve decomposition to formulate an incremental version of the common information problem that admits a simple fixed point solution, fast convergence, and complexity that is linear in the number of variables. This scalable approach allows us to demonstrate the usefulness of common information in high-dimensional learning problems. The sieve outperforms standard methods on dimensionality reduction tasks, solves a blind source separation problem that cannot be solved with ICA, and accurately recovers structure in brain imaging data.
The sieve could be viewed as a new objective for projection pursuit @cite_12 based on common information. The sieve stands out from standard pursuit algorithms in two ways. First, an information based orthogonality'' criteria for subsequent projections naturally emerges and, second, new factors may depend on factors learned at previous layers (note that in Fig. each learned latent factor is included in the remainder information that is optimized over in the next step). More broadly, the sieve can be viewed as a new approach to unsupervised deep representation learning @cite_36 @cite_9 . In particular, our setup can be directly viewed as an auto-encoder with a novel objective @cite_21 . From that point of view, it is clear that the sieve can also be directly leveraged for unsupervised density estimation @cite_28 .
@cite_36 @cite_28 @cite_9 @cite_21 @cite_12
2040399008 2177136334 2414767909 1964724001
This paper considers a recently emerged hyperspectral unmixing formulation based on sparse regression of a self-dictionary multiple measurement vector (SD-MMV) model, wherein the measured hyperspectral pixels are used as the dictionary. Operating under the pure pixel assumption, this SD-MMV formalism is special in that it allows simultaneous identification of the endmember spectral signatures and the number of endmembers. Previous SD-MMV studies mainly focus on convex relaxations. In this study, we explore the alternative of greedy pursuit, which generally provides efficient and simple algorithms. In particular, we design a greedy SD-MMV algorithm using simultaneous orthogonal matching pursuit. Intriguingly, the proposed greedy algorithm is shown to be closely related to some existing pure pixel search algorithms, especially, the successive projection algorithm (SPA). Thus, a link between SD-MMV and pure pixel search is revealed. We then perform exact recovery analyses, and prove that the proposed greedy algorithm is robust to noise-including its identification of the (unknown) number of endmembers-under a sufficiently low noise level. The identification performance of the proposed greedy algorithm is demonstrated through both synthetic and real-data experiments. Methods for Projection Pursuit aim to facilitate the visual exploration of high-dimensional data by identifying interesting low-dimensional projections. A major challenge is the design of a suitable quality metric of projections, commonly referred to as the projection index, to be maximized by the Projection Pursuit algorithm. In this paper, we introduce a new information-theoretic strategy for tackling this problem, based on quantifying the amount of information the projection conveys to a user given their prior beliefs about the data. The resulting projection index is a subjective quantity, explicitly dependent on the intended user. As a useful illustration, we developed this idea for two particular kinds of prior beliefs. The first kind leads to PCA (Principal Component Analysis), shining new light on when PCA is (not) appropriate. The second kind leads to a novel projection index, the maximization of which can be regarded as a robust variant of PCA. We show how this projection index, though non-convex, can be effectively maximized using a modified power method as well as using a semidefinite programming relaxation. The usefulness of this new projection index is demonstrated in comparative empirical experiments against PCA and a popular Projection Pursuit method. Person re-identification is to seek a correct match for a person of interest across different camera views among a large number of impostors. It typically involves two procedures of non-linear feature extractions against dramatic appearance changes, and subsequent discriminative analysis in order to reduce intra-personal variations while enlarging inter-personal differences. In this paper, we introduce a hybrid deep architecture which combines Fisher vectors and deep neural networks to learn non-linear transformations of pedestrian images to a deep space where data can be linearly separable. The proposed method starts from Fisher vector encoding which computes a sequence of local feature extraction, aggregation, and encoding. The resulting Fisher vector output are fed into stacked supervised layer to seek non-linear transformation into a deep space. On top of the deep neural network, Linear Discriminant Analysis (LDA) is reinforced such that linearly separable latent representations can be learned in an end-to-end fashion. By optimizing an objective function modified from LDA, the network is enforced to produce feature distributions which have a low variance within the same class and high variance between classes. The objective is essentially derived from the general LDA eigenvalue problem and allows to train the network with Stochastic Gradient Descent and back-propagate LDA gradients to compute Gaussian Mixture Model (GMM) gradients in Fisher vector encoding. For empirical evaluations, we test our approach on four benchmark data sets in person re-identification (VIPeR 1, CUHK03 2, CUHK01 3, and Market 1501 4). Extensive experiments on these benchmarks show that our method can achieve state-of-the-art results. HighlightsA hybrid architecture that combines Fisher vectors and deep neural networks.An end-to-end training with linear discriminant analysis as objective.Deep features are linearly separable and class separability is maximally preserved. Abstract A new projection pursuit algorithm for exploring multivariate data is presented that has both statistical and computational advantages over previous methods. A number of practical issues concerning its application are addressed. A connection to multivariate density estimation is established, and its properties are investigated through simulation studies and application to real data. The goal of exploratory projection pursuit is to use the data to find low- (one-, two-, or three-) dimensional projections that provide the most revealing views of the full-dimensional data. With these views the human gift for pattern recognition can be applied to help discover effects that may not have been anticipated in advance. Since linear effects are directly captured by the covariance structure of the variable pairs (which are straightforward to estimate) the emphasis here is on the discovery of nonlinear effects such as clustering or other general nonlinear associations among the variables. Although arbitrary ...