Suppose the extracted features' quality well correlates with a checkpoint's final accuracy on a downstream task. Besides the linear separability above, we can rank the checkpoints by their mutual information between the high-dimensional features and discrete labels of the downstream task. We employ the state-of-the-art \(I_\alpha \)  mutual information estimator [1]}, where \(\alpha \)  controls the trade-off between variance and bias. It is a variational lower bound parameterized by a neural network. Belghazi et al. [2]} report that the neural estimators generally outperform prior mutual information estimations, especially when the variables are high-dimensional. We use the code released by the authors to calculate \(I_\alpha \)  [1]}.
