(2)
BEiT uses learned relative position biases that are added to the self-attention logits [1]} in each block, instead of the absolute position embeddings used by the other methods. To account for this, albeit imperfectly, we include both relative position biases and absolute position embeddings in all detection models regardless of their use in pre-training. For BEiT, we transfer the pre-trained biases and randomly initialize the absolute position embeddings. For all other methods, we zero-initialize the relative position biases and transfer the pre-trained absolute position embeddings. Relative position biases are shared across windowed attention blocks and (separately) shared across global attention blocks. When there is a spatial dimension mismatch between pre-training and fine-tuning, we resize the pre-trained parameters to the required fine-tuning resolution.
