Group I: Checkpoints of mixed supervision.
The first group of checkpoints are pre-trained with mixed supervision till convergence, including supervised learning, self-supervised learning, semi-supervised learning, and the discriminators or encoders in deep generative models. It consists of 16 ResNet-50s [1]}. We borrow 14 models pre-trained on ImageNet [2]} from [3]}. Among them, four are pre-trained by self-supervised learning (Jigsaw [4]}, Relative Patch Location [5]}, Exemplar [6]}, and Rotation [7]}), six are the discriminators of generative models (WAE-UKL [8]}, WAE-GAN, WAE-MMD [9]}, Cond-BigGAN, Uncond-BigGAN [10]}, and VAE [11]}), two are based on semi-supervised learning (Semi-Rotation-10% and Semi-Exemplar-10% [12]}), one is by fully supervised learning (Sup-100%-Img [1]}), and one is trained with a hybrid supervised loss (Sup-Exemplar-100% [12]}). We also add two supervised checkpoints pre-trained on iNaturalist (Sup-100%-Inat) [15]} and Places365 (Sup-100%-Pla) [16]}, respectively. Using the evaluation procedure \(\mathbf {G}(\theta _i,t)\)  (cf. equation (REF )), we obtain their final accuracies on the downstream tasks described in Section REF .
