We find that the inclusion of relative position bias [1]} in each attention layer has a significant impact on the final top-1 accuracy of ViT models trained with PSS. Table REF  shows the accuracy of DeiT-S trained with and without relative position bias. When relative position bias is added to the baseline DeiT-S, it leads to a 1% improvement in accuracy (from 80.1% to 81.1%). By comparison, the DeiT-S+PSS model without relative position bias achieves a 1.9% lower accuracy than the baseline. When relative position bias is included with PSS, it outperforms the baseline without it by 0.3%. We hypothesize that the use of relative position bias becomes more important to attention when fewer patches are observed during many training iterations.
<TABLE>