(3) BEiT makes use of layer scale [1]} in pre-training, while the other methods do not. During fine-tuning, the BEiT-initialized model must also be parameterized to use layer scale with the pre-trained layer scaling parameters initialized from the pre-trained model. All other models do not use layer scale in pre-training or in fine-tuning.
