First, we show that the choice of training losses and design of architecture substantially affects the collapse of the features on the penultimate layer, and hence the transfer accuracy.For the choices of loss, the works [1]}, [2]} have similar observations, where different choices of training losses for pre-training lead to different transfer performance. To show this, we pre-train ResNet18 models on the Cifar-100 dataset with three different choices of loss functions: CE, MSE [3]}, and SupCon [4]}). Once the model is trained, we evaluate the test accuracy on the Cifar-10 dataset, by only learning a linear classifier upon the frozen pre-trained models. In tab:initialexp, we summarize the results of \(\mathcal {NC}_1\)  of all pre-trained models and the corresponding transfer accuracy for different training scenarios. For the SupCon loss [4]}, in particular, we also follow the original setup by using a nonlinear multi-layer perception (MLP) module as a projection head after the ResNet18 encoder. More specifically, once the model has been pre-trained, the projection head will be abandoned and only the encoder network will be utilized as the pre-trained model for downstream tasks. The results are reported in the last two columns of tab:initialexp, with adding one-layer linear projection and multiple-layer MLP projection, respectively.
As we observe from tab:initialexp,
