@inproceedings{
schaeffer2023an,
title={An Information-Theoretic Understanding of Maximum Manifold Capacity Representations},
author={Rylan Schaeffer and Berivan Isik and Victor Lecomte and Mikail Khona and Yann LeCun and Andrey Gromov and Ravid Shwartz-Ziv and Sanmi Koyejo},
booktitle={NeurIPS 2023 workshop: Information-Theoretic Principles in Cognitive Systems},
year={2023},
url={https://openreview.net/forum?id=KeWNER68iP}
}
Maximum Manifold Capacity Representations (MMCR) is a recent multi-view self-supervised learning (MVSSL) method that matches or surpasses other leading MVSSL methods. MMCR is interesting for at least two reasons. Firstly, MMCR is an oddity in the zoo of MVSSL methods: it is not (explicitly) contrastive, applies no masking, performs no clustering, leverages no distillation, and does not (explicitly) reduce redundancy. Secondly, while many self-supervised learning (SSL) methods originate in information theory, MMCR distinguishes itself by claiming a different origin: a statistical mechanical characterization of the geometry of linear separability of data manifolds. However, given the rich connections between statistical mechanics and information theory, and given recent work showing how many SSL methods can be understood from an information-theoretic perspective, we conjecture that MMCR can be similarly understood from an information-theoretic perspective. In this paper, we leverage tools from high dimensional probability and information theory to demonstrate that an optimal solution to MMCR's nuclear norm-based objective function is the same optimal solution that maximizes a well-known lower bound on mutual information between views.
An Information-Theoretic Understanding of Maximum Manifold Capacity Representations
Rylan Schaeffer Berivan Isik Victor Lecomte Mikail Khona Yann LeCun Andrey Gromov Ravid Shwartz-Ziv Sanmi Koyejo