The main limitation of our work is the bottleneck caused by the extensive size of the query element (multi-scale image features, there are about \(\times 20\)  more image tokens to process compared to the single-scale feature map).
Despite our proposed deformable attentions, MSTR suffers from an estimated \(10\%\)  increase in parameters and \(\sim \times 2\)  GFLOPs compared to the single-scale baseline, QPIC [1]}.
Although recent related works have tackled the efficiency problem in deformable attentions by sampling the query element as well [2]}, the research scope of this work did not cover this issue.
