We evaluate using PSSs on ViTs for ImageNet [1]} both trained from scratch and pre-trained using a reconstruction loss function. To train ViT from scratch, we modify the codebase (https://github.com/facebookresearch/deit) of DeiT [2]} to be compatible with PSSs. Additionally, we add relative position bias to each attention layer in DeiT starting from the implementation in the SimMIM [3]} codebase (https://github.com/microsoft/SimMIM). For pre-trained models, we modify the SimMIM codebase to include PSSs and start fine-tuning using their models trained using reconstructive loss. Otherwise, we follow the training regime and data augmentation used by each project for their baseline results. We train all models using a single machine with 4 A100 GPUs.
