We may also appreciate the challenge in ranking neural checkpoints by comparing it with another related line of work: predicting DNNs' generalization gaps [1]}, [2]}, [3]}. Jiang et al. [4]} use a linear regressor to predict a DNN's generalization gap, i.e., the discrepancy between its training and test accuracies, by exploring the training data's margin distributions. Other signals studied in the literature include network complexity and noise stability. Ranking neural checkpoints is more challenging than predicting a DNN's generalization gap. Unlike the training and test sets that share the same underlying distribution, the downstream task may be arbitrarily distant from the source task over which a checkpoint is pre-trained. Moreover, we do not have access to the pre-training data at all. Finally, instead of keeping the networks static, fine-tuning dramatically changes all weights of the checkpoints.
