Following the design principle of [1]}, we study diverse downstream tasks including Caltech101 [2]}, Flowers102 [3]}, Sun397 [4]}, and Patch Camelyon [5]}. These tasks are representative of general object recognition, fine-grained object recognition, scenery image classification, and medical image classification, respectively. Table 1 in Appendix A.1 provides more details of these tasks. A common theme is that their training sets are all medium-sized, making it especially beneficial to leverage pre-trained checkpoints to avoid overfitting.
