Current works seem to suggest that detection transformers are superior to the CNN-based object detector, like Faster RCNN, in both simplicity and model performance. However, we find that detection transformers show superior performance only on datasets with rich training data like COCO 2017 (118K training images), while the performance of most detection transformers drops significantly when the amount of training data is small. For example, on the commonly used autonomous driving dataset Cityscapes [1]} (3K training images), the average precisions (AP) of most of the detection transformers are less than half of Faster RCNN AP performance, as shown in Fig. REF .
Moreover, although the performance gaps between different detection transformers on the COCO dataset are less than 3 AP, a significant difference of more than 15 AP exists on the small-size Cityscapes dataset.
