[leftmargin=*]
Understandings of the \(\mathcal {NC}\)  phenomenon. There is a line of recent works deciphering training, generalization, and transferability of deep networks in terms of \(\mathcal {NC}\) , that are related to ours (see a recent review work [1]}). For training, recent works showed that \(\mathcal {NC}\)  happens for a variety of loss functions and formulations, such as cross-entropy (CE) [2]}, [3]}, [4]}, [5]}, [6]}, [7]}, mean-squared error (MSE) [8]}, [9]}, [10]}, [11]}, [12]}, and supervised contrastive (SupCon) loss [13]}. For generalization, the work [14]} shows that \(\mathcal {NC}\)  also happens on test data drawn from the same distribution asymptotically, but not for finite samples [15]}. On the other hand, quite a few recent works [2]}, [14]}, [15]} studied the connection between \(\mathcal {NC}\)  and generalization of overparameterized deep networks. Moreover, the works [15]}, [20]}, [21]} demonstrated that the variability collapse of features is actually happening progressively from shallow to deep layers (with a linear decay rate), and [22]} showed that test performance can be improved when enforcing variability collapse on features of intermediate layers. The works [23]}, [24]}, [25]} studied problems with imbalanced training data, showing that fixing the classifier as simplex ETFs improves test performance on imbalanced training data and long-tailed classification problems. For transferability, the work [26]} implicitly showed that there is a tradeoff between variability collapse and transfer accuracy by experiments on a variety of loss functions.

Representation learning and model pre-training. There are quite a few recent works studying the factors that affect transferability of pre-trained models, but the results are largely inconclusive. For example, the work [27]} argues that models pre-trained on ImageNet with higher accuracy tend to perform better on other downstream tasks. However, such a conclusion has been challenged by more recent works [26]}, [29]}. These results showed that the training loss and diversity of the features could be more important factors of transferability than the pre-trained accuracy. However, compared to our work, they only study few aspects (e.g., training loss) of deep network architectures that affect transferability, and they only focus on the diversity aspect on the source dataset. At the same time, the work [30]} showed that models learned using contrastive type of loss functions could have better transferability, and [31]} showed that their representations are more uniform over hyperspheres. The architecture and depth of CNNs were also shown to impact transfer performance [32]}. Other work [33]} exploited the importance of layers in overparameterized networks, suggesting the shallow and deep layers are more important in fine-tuning pre-trained models for downstream tasks.

