To corroborate our claim, we use an ImageNet-1k [1]} pre-trained ResNet34 [2]} model
and we evaluate the \(\mathcal {NC}_1\)  metric on each residual block's output feature upon the downstream data. Additionally, we conducted similar experiments on ViT-B (vision transformer base model) [3]} by using a pre-trained checkpoint released online The Vit-B model checkpoint we used could be found here..
As we observe in fig:layerwisetransfer, layers with smaller \(\mathcal {NC}\)\(_1\)  of the output features result in better transfer accuracy.
Moreover, we can observe a near linear relationship between \(\mathcal {NC}_1\)  and transfer accuracy. Therefore, the transfer accuracy is more related to the variability collapse upon the layer rather than the depth of the layer, and such a phenomenon holds universally from ResNet to ViT models.
<FIGURE>