Recent work shows that \(\mathcal {NC}\)  persists across a range of canonical classification problems, on different loss functions (e.g., CE [1]}, [2]}, MSE [3]}, [4]}, [5]}, SupCon [2]}, [7]}), different neural network architectures (e.g., VGG [8]}, ResNet [9]}, and DenseNet [10]}), and a variety of standard datasets (e.g., MNIST [11]}, CIFAR [12]}, and ImageNet [13]}). As we observe from above, although the maximum between-class separation suggests the learned features are discriminative in (REF ), the intra-class variability collapsing to a single dimension in (REF ) implies that the network is memorizing the labels rather than preserving the intrinsic structures of the data. As such, the loss of information of the input data could be detrimental for the transferability of the learned deep models. Nonetheless, the \(\mathcal {NC}\)  phenomenon offers us good metrics for evaluating the transferability of pre-trained models that we discuss in the following.
