Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods
Self-Supervised Learning (SSL) surmises that inputs and pairwise positive
relationships are enough to learn meaningful representations. Although SSL has
recently reached a milestone: outperforming supervised methods in many
modalities\dots the theoretical foundations are limited, method-specific, and
fail to provide principled design guidelines to practitioners. In this paper,
we propose a unifying framework under the helm of spectral manifold learning to
address those limitations. Through the course of this study, we will rigorously
demonstrate that VICReg, SimCLR, BarlowTwins et al. correspond to eponymous
spectral methods such as Laplacian Eigenmaps, Multidimensional Scaling et al.
  This unification will then allow us to obtain (i) the closed-form optimal
representation for each method, (ii) the closed-form optimal network parameters
in the linear regime for each method, (iii) the impact of the pairwise
relations used during training on each of those quantities and on downstream
task performances, and most importantly, (iv) the first theoretical bridge
between contrastive and non-contrastive methods towards global and local
spectral embedding methods respectively, hinting at the benefits and
limitations of each. For example, (i) if the pairwise relation is aligned with
the downstream task, any SSL method can be employed successfully and will
recover the supervised method, but in the low data regime, VICReg's invariance
hyper-parameter should be high; (ii) if the pairwise relation is misaligned
with the downstream task, VICReg with small invariance hyper-parameter should
be preferred over SimCLR or BarlowTwins.