Kornlith et al's 12-dataset benchmark (K12) and iNaturalist-18 (iNat18).
On the K12 dataset, we follow the previous standard settings [1]} to use training set and validation set to search for the best hyper-parameters, and then merge the training and validation sets as the final training set with the searched best hyper-parameters, and evaluate the final trained models on the test set. And we adopt standard splits of train/val/test sets as in [1]}. For Aircraft, Pets, Caltech-101, Oxford 102 Flowers, the mean-per-class accuracy metric is adopted, for other datasets, the top-1 accuracy is adopted.
For K12, we follow [1]} to select the optimal learning rate, weight decay, layer decay, and drop path rate.
In pilot experiments, we find that for 1K-SUP pre-trained models, the drop path rate can be fixed as \(0.2\) , and for 1K-MIM pre-trained models, on smaller datasets like Stanford Cars, FGVC Aircraft, DTD, Caltech101, Flowers102, and Oxford Pets, drop path rate is first fixed as \(0.0\)  and fixed as \(0.2\)  for other datasets. And the weight decay can be fixed as \(0.05\) .
Then we do a grid search on learning rate and layer decay. For 1K-MIM pre-trained models, our grid consists of 5 approximately logarithmically spaced learning rates between \(1.25e\) -4 and \(2.5e\) -3 and 3 equally spaced layer decay between \(0.75\)  and \(0.95\) . For 1K-SUP pre-trained models, our grid consists of 5 approximately logarithmically spaced learning rates between \(2.5e\) -5 and \(5e\) -4 and 3 equally spaced layer decay between \(0.75\)  and \(0.95\) . Then we adjust the learning rate, layer decay, and drop path rate in the neighborhood of the best setting in the grid search to get the final results.
