Pre-training stage.
The default setting is shown in Table REF .
We use uniform distribution to initialize the positional embeddings, and we use truncated normal distribution with \(\mu =0\)  and \(\sigma =0.02\)  to initialize the mask token, similar to MAE [1]}.
We use PyTorch official implementation to implement the Transformer blocks.
We use the linear scaling rule for learning rate and batch size: lr = base_lr \(\times \)  (batch_size/8) for all datasets in the pre-training stage.
<TABLE>