[leftmargin=*]
Layer fine-tuning (FT). As shown in tab:finalresultsrefined, we demonstrate that the collapse of penultimate layer features can be achieved simply by fine-tuning only one of the intermediate layers while keeping the rest of the network frozen (see fig:illustrationscl (left)). On top of that, our experiments show that fine-tuning one of the top layers near the penultimate layer often yields the best result universally.As shown in tab:appt4,tab:appt5,tab:appt6, we find fine-tuning Block 7 for ResNet18 (8 blocks in total), Block 14 for ResNet50 (16 blocks in total) and Layer 11 for ViT model (12 layers in total) always yields the best or near-optimal results. Although this is still empirical, hypothetically we believe this is because, at the position of the top layers in a network, the information from the inputs has already been extracted and distilled, and hence fine-tuning such a layer is the most effective.
As we observe from tab:finalresultsrefined, on both ResNet and ViT network architectures, this simple Layer FT approach already leads to substantial performance gain compared with vanilla linear probing (i.e., fixed feature training) universally on a variety of downstream datasets including Cifar [1]}, FGVS-Aircraft [2]}, DTD [3]} and Oxford-IIIT-Pet [4]} datasets.

Skip connection layer (SCL) fine-tuning (FT). Based upon the Layer FT, as shown in fig:illustrationscl (right), we can further improve its performance by adding a skip connection from the fine-tuned layer to the penultimate layer, and using the combined features of those two layers (i.e., adding the output features of the two layers togetherFor ResNet, if the feature dimensions are different, we can do zero-padding of the lower dimensional feature to compensate for the dimensional difference. We refer to Appendix  for more details.) as the new feature for the final linear classifier. Such a method enables the network to more effectively fine-tune the selected layer by explicitly passing the information of the data from the intermediate layer to the classifier, without suffering information loss through the cascade of intermediate layers. Additionally, instead of throwing away layers above the fine-tuned layers, the method also enjoys the benefits of depth of the deep models, which intuitively makes the features more linearly separable across layers. From tab:finalresultsrefined, for both ResNet and ViT network architectures, we observe that SCL FT nearly always outperforms Layer FT and achieves comparable or even better results compared with full model FT on a variety of datasets.

<FIGURE>