Removing FPN.
Multi-scale feature fusion has been proved effective for object detection [1]}. While the CNN-based FPN neck can achieve multi-scale feature fusion at a small computational cost, the attention mechanism has a quadratic complexity with respect to the image scale, making the modeling of multi-scale features in DETR non-trivial. Thus DETR only takes 32\(\times \)  down-sampled single-scale feature for prediction. In this stage, we remove the FPN neck and send only the 32\(\times \)  down-sampled feature to the detection head, which is consistent with DETR. As expected, without multi-scale modeling, the model performance degrades significantly by 7.3 AP under the 50E schedule, as shown in Table REF .
