Recent works demonstrated that using pre-trained checkpoints that have similar feature representations as the target task's representations can improve transfer learning [1]}, [2]}, [3]}. Song et al. [2]}, [3]} employed attribution maps to compare two models and then quantified transferabilities by the similarity of two models. More recently, Dwivedi et al. [6]} proposed a duality diagram similarity (DDS) based approach to select a model initialization for transfer learning. Those approaches all require a converged model on target datasets, incurring intensive computation. However, we want to design a lightweight method for ranking checkpoints, ideally without any training procedures.
