Replacing dynamic convolutions with cross-attention.
A very interesting design in Sparse RCNN is the dynamic convolution [1]}, [2]} in the decoder, which acts very similar to the role of cross-attention in DETR. Specifically, they both adaptively aggregate the context from the image features to the object candidates based on their similarity. In this step, we replace the dynamic convolution with the cross-attention layer with learnable query positional embedding, and the corresponding results are shown in Table REF .
Counter-intuitively, a larger number of learnable parameters does not necessarily make the model more data-hungry. In fact, the dynamic convolutions with about 70M parameters can exhibit better data efficiency than the parameter-efficient cross-attention layer.
