@inproceedings{
isik2023an,
title={An Information-Theoretic Understanding of Maximum Manifold Capacity Representations},
author={Berivan Isik and Victor Lecomte and Rylan Schaeffer and Yann LeCun and Mikail Khona and Ravid Shwartz-Ziv and Sanmi Koyejo and Andrey Gromov},
booktitle={UniReps:  the First Workshop on Unifying Representations in Neural Models},
year={2023},
url={https://openreview.net/forum?id=9I2jTPhCku}
}
Maximum Manifold Capacity Representations (MMCR) is a recent multi-view self-supervised learning (MVSSL) method that matches or surpasses other leading MVSSL methods. MMCR is interesting for at least two reasons. Firstly, MMCR is an oddity in the zoo of MVSSL methods: it is not (explicitly) contrastive, applies no masking, performs no clustering, leverages no distillation, and does not (explicitly) reduce redundancy. Secondly, while many self-supervised learning (SSL) methods originate in information theory, MMCR distinguishes itself by claiming a different origin: a statistical mechanical characterization of the geometry of linear separability of data manifolds. However, given the rich connections between statistical mechanics and information theory, and given recent work showing how many SSL methods can be understood from an information-theoretic perspective, we conjecture that MMCR can be similarly understood from an information-theoretic perspective. In this paper, we leverage tools from high dimensional probability and information theory to demonstrate that an optimal solution to MMCR's nuclear norm-based objective function is the same optimal solution that maximizes a well-known lower bound on mutual information.
An Information-Theoretic Understanding of Maximum Manifold Capacity Representations
Berivan Isik Victor Lecomte Rylan Schaeffer Yann LeCun Mikail Khona Ravid Shwartz-Ziv Sanmi Koyejo Andrey Gromov