Multi-scale deformable attention.
To avoid the computational and memory complexities of standard Transformers  [1]},
we adopt deformable attention from [2]}.
Given a feature map, for each query element, the deformable attention only attends to a small set \(N_{s}\)  of key sampling points around a reference point, instead of looking over all \(H_{l}W_{l}\)  spatial locations on the feature map, where \(N_{s}<<H_{l}W_{l}\) .
The multi-scale deformable attention applies deforamable attention across multi-scale feature maps and enables encoding richer context.
We use multi-scale deformable attention for the self- and cross-attention in the encoder and decoder.
