[leftmargin=*]
Different training losses affect the feature diversity, and hence transfer accuracy, where losses with larger feature diversity (larger \(\mathcal {NC}\)\(_1\) ) lead to better transfer accuracy. For instance, the model pre-trained with the MSE loss is severely collapsed on the source dataset (smallest \(\mathcal {NC}\)\(_1\) , showing the worst transfer accuracy).

The MLP projection head plays an important role for better transferability. The model pre-trained using the SupCon with a multi-layer MLP projection head has the least collapsed features compared to the other models, and it demonstrates superior transfer performance.This observation is consistent with [1]}. When we replace the MLP with a linear projection layer, both the \(\mathcal {NC}\)\(_1\)  and transfer accuracy SupCon decrease, showing on-par performance with the model trained using the CE loss.

