For tab:finalresultsrefined, we use a wide variety of experimental setups, including different model architectures, pretraining datasets and downstream datasets. We then compare the performance between linear probing, layer fine-tuning, SCL fine-tuning and full model fine-tuning. For ResNet models, we consider each block as a fine-tuning unit and fine-tune the first block of each layer (e.g., as shown in fig:ftblockdemo, ResNet18 has 4 layers where each layer has 2 blocks, we then fine-tune on the first block of each layer). For Vit-B32 model, we treat each of the 12 encoder layer as a fine-tuning unit. In terms of skip connection, for ViT-B32 model, since the feature dimension from each layer remains constant, the skip connection could be directly applied for the features of the fine-tuned layer and the penultimate layer. However, for ResNet models, the number of channels and the feature dimension change across layers. Therefore, to calculate the skip connection, we first do an adaptive average pooling on the fine-tuned layer features to make each channel has only one entry; then we further do a zero-padding to make the number of channels match with the penultimate layer features. Finally, we apply the skip connection and use the combined features to train the classifier. We note that for the sake of fair comparison, we always let the normalization layers [1]}, [2]} update the running means and variances on the downstream data for all fine-tuning methods we are comparing (linear probing, layer FT, SCL FT and full model FT).
