Deep latent variable models are powerful tools for representation learning. In this paper, we adopt the deep information bottleneck model, identify its shortcomings and propose a model that circumvents them. To this end, we apply a copula transformation which, by restoring the invariance properties of the information bottleneck method, leads to disentanglement of the features in the latent space. Building on that, we show how this transformation translates to sparsity of the latent space in the new model.   We evaluate our method on artificial and real data. In recent years, deep latent variable models BID13 BID22 BID7 have become a popular toolbox in the machine learning community for a wide range of applications BID14 BID19 BID11 . At the same time, the compact representation, sparsity and interpretability of the latent feature space have been identified as crucial elements of such models. In this context, multiple contributions have been made in the field of relevant feature extraction BID3 BID0 and learning of disentangled representations of the latent space BID5 BID2 BID9 .In this paper, we consider latent space representation learning. We focus on disentangling features with the copula transformation and, building on that, on forcing a compact low-dimensional representation with a sparsity-inducing model formulation. To this end, we adopt the deep information bottleneck (DIB) model BID0 which combines the information bottleneck and variational autoencoder methods. The information bottleneck (IB) principle BID26 identifies relevant features with respect to a target variable. It takes two random vectors x and y and searches for a third random vector t which, while compressing x, preserves information contained in y. A variational autoencoder (VAE) BID13 BID22 ) is a generative model which learns a latent representation t of x by using the variational approach.Although DIB produces good results in terms of image classification and adversarial attacks, it suffers from two major shortcomings. First , the IB solution only depends on the copula of x and y and is thus invariant to strictly monotone transformations of the marginal distributions. DIB does not preserve this invariance, which means that it is unnecessarily complex by also implicitly modelling the marginal distributions. We elaborate on the fundamental issues arising from this lack of invariance in Section 3. Second , the latent space of the IB is not sparse which results in the fact that a compact feature representation is not feasible.Our contribution is two-fold: In the first step, we restore the invariance properties of the information bottleneck solution in the DIB. We achieve this by applying a transformation of x and y which makes the latent space only depend on the copula. This is a way to fully represent all the desirable features inherent to the IB formulation. The model is also simplified by ensuring robust and fully non-parametric treatment of the marginal distributions. In addition , the problems arising from the lack of invariance to monotone transformations of the marginals are solved. In the second step, once the invariance properties are restored, we exploit the sparse structure of the latent space of DIB. This is possible thanks to the copula transformation in conjunction with using the sparse parametrisation of the information bottleneck, proposed by BID21 . It translates to a more compact latent space that results in a better interpretability of the model. The remainder of this paper is structured as follows: In Section 2, we review publications on related models. Subsequently, in Section 3, we describe the proposed copula transformation and show how it fixes the shortcomings of DIB, as well as elaborate on the sparsity induced in the latent space. In Section 4, we present results of both synthetic and real data experiments. We conclude our paper in Section 5. We have presented a novel approach to compact representation learning of deep latent variable models. To this end, we showed that restoring invariance properties of the Deep Information Bottleneck with a copula transformation leads to disentanglement of the features in the latent space. Subsequently, we analysed how the copula transformation translates to sparsity in the latent space of the considered model. The proposed model allows for a simplified and fully non-parametric treatment of marginal distributions which has the advantage that it can be applied to distributions with arbitrary marginals. We evaluated our method on both artificial and real data. We showed that in practice the copula transformation leads to latent spaces that are disentangled, have an increased prediction capability and are resilient to adversarial attacks. All these properties are not sensitive to the only hyperparameter of the model, λ.In Section 3.2, we motivated the copula transformation for the Deep Information Bottleneck with the lack of invariance properties present in the original Information Bottleneck model, making the copula augmentation particularly suited for the DIB. The relevance of the copula transformation, however, reaches beyond the variational autoencoder, as evidenced by e.g. resilience to adversarial attacks or the positive influence on convergence rates presented in Section 4. These advantages of our model that do not simply follow from restoring the Information Bottleneck properties to the DIB, but are additional benefits of the copula. The copula transformation thus promises to be a simple but powerful addition to the general deep learning toolbox.
We apply the copula transformation to the Deep Information Bottleneck which leads to restored invariance properties and a disentangled latent space with superior predictive capabilities.
{"src": "scitldr-AIC_train", "paper_id": "Hk0wHx-RW"}