We begin our investigation by studying the relationship between \(\mathcal {NC}\)  metrics on pre-training dataset and transfer accuracy. Within a certain range,As random features do not collapse but not generalize well, the positive correlation above only holds to a certain extent. The reason is that random features are not discriminative. We conjecture that there could be a trade-off between feature diversity and discrimination. we show that the two are positively correlated – larger \(\mathcal {NC}\)  metrics leads to better transfer accuracy, which echoes similar discoveries in recent work [1]}, [2]}. The intuition is that if the learned representations are less collapsed on the pre-trained data, they better preserve the intrinsic structures of the input data. Moreover, compared to [1]}, [2]}, our work not only studies the effect of different training losses on feature diversity, but also (i) provides insights for several heuristics in model pre-training (e.g., projection head, data augmentation), and (ii) reveals the limitations of merely using feature diversity as the metric for evaluating representation quality for transfer learning.
