Understanding Pre-training. There are some outstanding works [1]}, [2]}, [3]}, [4]}, [5]}, [6]}, [7]} trying to understand the pre-training procedure and inspire a lot of following works in a wide range. [1]} reveals how features of different layers are transferable in deep neural networks. [2]} performs a sufficient experimental study on different backbones and tries to answer whether better ImageNet models transfer better. Some works [6]}, [11]}, [6]} try to understand the behaviors of ViT, with CKA [13]}, loss landscape [14]} and Fourier analysis.
In NLP, after BERT [15]} pre-training came out, there is also a lot of works [16]}, [17]}, [18]}, [19]} trying to understand it. Most of them focus on the only interpretable component of Transformer, self-attention block, to give some detailed understanding.
