Expressiveness and Learnability: A Unifying View for Evaluating Self-Supervised Learning
We propose a unifying view to analyze the representation quality of
self-supervised learning (SSL) models without access to supervised labels,
while being agnostic to the architecture, learning algorithm or data
manipulation used during training. We argue that representations can be
evaluated through the lens of expressiveness and learnability. We propose to
use the Intrinsic Dimension (ID) to assess expressiveness and introduce Cluster
Learnability (CL) to assess learnability. CL is measured as the learning speed
of a KNN classifier trained to predict labels obtained by clustering the
representations with K-means. We thus combine CL and ID into a single
predictor: CLID. Through a large-scale empirical study with a diverse family of
SSL algorithms, we find that CLID better correlates with in-distribution model
performance than other competing recent evaluation schemes. We also benchmark
CLID on out-of-domain generalization, where CLID serves as a predictor of the
transfer performance of SSL models on several classification tasks, yielding
improvements with respect to the competing baselines.