There is an increasing number of pre-trained deep neural networks (DNNs), which we call checkpoints. We may produce hundreds of intermediate checkpoints when we sweep through various learning rates, optimizers, and losses to train a DNN. Furthermore, semi-supervised [1]}, [2]}, [3]}, [4]}, [5]}, [6]}, [7]}, [8]} and self-supervised [9]}, [10]}, [11]}, [12]}, [13]} learning make it feasible to harvest DNN checkpoints with scarce or no labels. Fine-tuning [14]}, [15]} has become a de facto standard to adapt the pre-trained checkpoints to target tasks. It leads to faster convergence [16]}, [17]}, [18]} and better performance [19]} on the downstream tasks.
