Finally, we demonstrate that the phenomenon we discovered in subsec:downstreamnc can be very useful for designing simple yet efficient fine-tuning strategies without sacrificing the performance (compared to fully fine-tune). For vision tasks, transfer learning typically adopts two strategies: (i) fixed feature training [1]}, [2]}, [3]}, use pre-trained model up to the penultimate layer as a feature extractor and only train a new linear classifier on top of the features for a downstream task; (ii) full model fine-tuning [4]}, [5]}, [6]}, use the pre-trained model as initialization and fine-tune the whole model to fit a downstream task. However, the full fine-tuning could be very expensive especially for large models, while fixed feature training, without adapting the model to the downstream data, often results in worse performance. To deal with the disadvantages of both methods, for vision problems one promising but seldom explored approach is to only fine-tune selective layers [7]}, [8]}.
Based upon the correlation between penultimate layer collapse and the transfer learning performances in subsec:downstreamnc, our conjecture is that
