paragraph4
.5em plus1ex minus.2ex-.5emPositional Information. In the default BEiT code, the ViT is modified to use relative position biases [1]} in each transformer block instead of adding absolute position embeddings to the patch embeddings. This choice is an orthogonal enhancement that is not used by the other pre-training methods (though it could be). In an attempt to make the comparison more equal, we include these biases (and absolute position embeddings) in all fine-tuning models by default, as discussed in §REF .
