The information bottleneck method provides an information-theoretic method for representation learning, by training an encoder to retain all information which is relevant for predicting the label, while minimizing the amount of other, superfluous information in the representation. The original formulation, however, requires labeled data in order to identify which information is superfluous.   In this work, we extend this ability to the multi-view unsupervised setting, in which two views of the same underlying entity are provided but the label is unknown. This enables us to identify superfluous information as that which is not shared by both views. A theoretical analysis leads to the definition of a new multi-view model that produces state-of-the-art results on the Sketchy dataset and on label-limited versions of the MIR-Flickr dataset.   We also extend our theory to the single-view setting by taking advantage of standard data augmentation techniques, empirically showing better generalization capabilities when compared to traditional unsupervised approaches for representation learning. The goal of deep representation learning (LeCun et al., 2015) is to transform a raw observational input, x, into a representation, z, to extract useful information. Significant progress has been made in deep learning via supervised representation learning, where the labels, y, for the downstream task are known while p(y|x) is learned directly (Sutskever et al., 2012; Hinton et al., 2012) . Due to the cost of acquiring large labeled datasets, a recently renewed focus on unsupervised representation learning seeks to generate representations, z, that allow learning of (a priori unknown) target supervised tasks more efficiently, i.e. with fewer labels (Devlin et al., 2018; Radford et al., 2019) . Our work is based on the information bottleneck principle (Tishby et al., 2000) stating that whenever a data representation discards information from the input which is not useful for a given task, it becomes less affected by nuisances, resulting in increased robustness for downstream tasks. In the supervised setting, one can directly apply the information bottleneck method by minimizing the mutual information between z and x while simultaneously maximizing the mutual information between z and y (Alemi et al., 2017) . In the unsupervised setting, discarding only superfluous information is more challenging as without labels one cannot directly identify the relevant information. Recent literature (Devon Hjelm et al., 2019; van den Oord et al., 2018) has instead focused on the InfoMax objective maximizing the mutual information between x and z, I(x, z), instead of minimizing it. In this paper, we extend the information bottleneck method to the unsupervised multi-view setting. To do this, we rely on a basic assumption of the multi-view literature -that each view provides the same task relevant information (Zhao et al., 2017) . Hence, one can improve generalization by discarding from the representation all information which is not shared by both views. We do this through an objective which maximizes the mutual information between the representations of the two views (Multi-View InfoMax) while at the same time reducing the mutual information between each view and its corresponding representation (as with the information bottleneck). The resulting representation contains only the information shared by both views, eliminating the effect of independent factors of variations. Our contributions are three-fold: (1) We extend the information bottleneck principle to the unsupervised multi-view setting and provide a rigorous theoretical analysis of its application. (2) We define a new model that empirically leads to state-of-the-art results in the low-data setting on two standard multi-view datasets, Sketchy and MIR-Flickr. (3) By exploiting standard data augmentation techniques, we empirically show that the representations obtained with our model in single-view settings are more robust than other popular unsupervised approaches for representation learning, connecting our theory to the choice of augmentation function class. In this work, we introduce Multi-View Information Bottleneck, a novel method that relies on multiple data-views to produce robust representation for downstream tasks. Most of the multi-view literature operates under the assumption that each view is individually sufficient for determining the label (Zhao et al., 2017) , while our method only requires the weaker mutual redundancy condition outlined in Section 3, enabling it to be applied to any traditional multi-view task. In our experiments, we compared MIB empirically against other approaches in the literature on three such tasks: sketch-based image retrieval, multi-view and unsupervised representation learning. The strong performance obtained in the different areas show that Multi-View Information Bottleneck can be practically applied to various tasks for which the paired observations are either available or are artificially produced. Furthermore, the positive results on the MIR-Flickr dataset show that our model can work well in practice even when mutual redundancy holds only approximately. There are multiple extensions that we would like to explore in future work. One interesting direction would be considering more than two views. In Appendix D we discuss why the mutual redundancy condition cannot be trivially extended to more than two views, but we still believe such an extension is possible. Secondly, we believe that exploring the role played by different choices of data augmentation could bridge the gap between the Information Bottleneck principle and with the literature on invariant neural networks (Bloem-Reddy & Whye Teh, 2019), which are able to exploit known symmetries and structure of the data to remove superfluous information.
We extend the information bottleneck method to the unsupervised multiview setting and show state of the art results on standard datasets
{"src": "scitldr-AIC_train", "paper_id": "B1xwcyHFDr"}