[leftmargin=*]
Less collapsed features on source data leads to better transferability to certain extents. By evaluating the \(\mathcal {NC}\)  metrics on different loss functions [1]} and several widely used techniques in transfer learning (e.g., the projection head, various data augmentations [2]}, [3]}, [4]}, and adversarial training [5]}, [6]}), we find that the more diverse the features are, the better the transferability of the pre-trained model. In comparison to previous works [7]}, [8]}, however, we showed that this only holds to a certain extent as random features are not collapse but they do not generalize well.We find that there is a certain threshold, that the transferability increases with the feature diversity below the threshold but decreases or becomes uncorrelated beyond it. Increasing the feature diversity will decrease the margin upon the threshold and hence the relationship with transferability becomes more involved with too large feature diversity. Moreover, we further use the relationship for explaining the underlying mechanism of many popular heuristics for transfer learning.

More collapse of fine-tuned models leads to better test performance on downstream tasks. In contrast, when we are evaluating different pre-trained models on downstream tasks, we observe that more collapsed features on downstream data usually lead to better transfer accuracy. We demonstrate the universality across a variety of downstream data [9]}, [10]}, [11]}, [12]} as well as pre-trained models [13]}, [14]}, [15]}, [16]}. Moreover, we further showed that this phenomenon not only happens on the penultimate layer across different pre-trained models, but also across different layers of the same pre-trained model.

Pre-trained models can be more effectively fine-tuned through \(\mathcal {NC}\) . Efficient and effective transfer learning is of paramount importance for large models nowadays [17]}, [18]}, [19]}. Inspired by the above findings, we show that we can design simple but more memory-efficient fine-tuning methods by collapsing the features of the penultimate layer. By experiments on a variety of network architectures, we demonstrate that the proposed fine-tuning strategy achieves on-par or even better performances compared with full model fine-tuning and is also more robust against data scarcity.

