Cross Attention Blocks.
We use cross-attention heads [1]} to aggregate the distributed representations into the instance representation. We ablate the number of blocks for this design in Table REF . The finetuning performance is not affected by the depth of the cross-attention blocks, while the linear probing performance is improved by 2% with two blocks and saturates for more blocks.
