To simulate a sufficiently large pool of checkpoints in the real applications, we finally combine the checkpoints in Group I, II, and III into one large group and conduct checkpoint ranking experiments on it. We also add one more group of checkpoints with ResNet-101s [1]} to evaluate checkpoint ranking on deeper models. Please see more details in Appendix A.3 and A.4. Although the benchmark can be easily extended to many downstream tasks in other modalities, e.g., voice, text, and cross-modal modalities, we steer our attention into comparing several intuitive ranking measures on variants of checkpoints, covering different training strategies, source domains, and architectures at a range of early stopping stages. We initial the checkpoint ranking idea, demonstrate the existence of an effective yet lightweight measure, \(\mathcal {N}\) LEEP, and hope it can shed light on more efficient ranking methods and practical applications.
