It is nontrivial to rank the checkpoints for a downstream task. We explain this point by drawing insights from the related, yet arguably easier, task transferability problem [1]}, [2]}, [3]}, [4]}, which aims to provide high-level guidance about how well a neural network pre-trained in one task might transfer to another. However, not all checkpoints pre-trained in the same source task transfer equally well to the target task [5]}, [6]}. The pre-training strategy also matters. Zhai et al. [7]} find that combining supervision with self-supervision improves a network's transfer results on downstream tasks. He et al. [8]} also show that self-supervised pre-training benefits object detection more than its supervised counterpart under the same fine-tuning setup.
