Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embedding
Contrastive learning, especially Self-Supervised Contrastive Learning (SSCL),
has achieved great success in extracting powerful features from unlabeled data,
enabling comparable performance to the supervised counterpart. In this work, we
contribute to the theoretical understanding of SSCL and uncover its connection
to the classic data visualization method, Stochastic Neighbor Embedding (SNE).
In the perspective of SNE, whose goal is matching pairwise distance, SSCL can
be viewed as a special case with the input space pairwise distance specified by
constructed "positive" pairs from data augmentation. The established
correspondence facilitates deeper theoretical understandings of learned
features of SSCL, as well as methodological guidelines for practical
improvement. Specifically, through the lens of SNE, not only can we re-derive
the alignment and uniformity principle, but also provide novel analysis on
domain-agnostic augmentations and implicit bias. To illustrate the practical
advantage, we demonstrate that the modifications from SNE to $t$-SNE can also
be adopted in the SSCL setting, achieving significant improvement in both
in-distribution and out-of-distribution generalization.