The usage of projection head for model pre-training is first introduced and then popularized for self-supervised learning [1]}, [2]}, but the reason why it works is not very well understood. Here, we demystify the underlying mechanism of projection head based upon progressive variability collapse. As shown in recent works [3]}, [4]}, [5]} and our experiments in fig:res18nclayerwisedynamic, the within-class variability collapse actually happens progressively from shallow to deep layers: the closer to the final layer, the severer the variability collapse is (i.e., the smaller the \(\mathcal {NC}_1\)  of the corresponding layer). Therefore, for model pre-training,
