

Appendix
Training Classification Teachers
In this section, we provide the details of our experimental classification setup and of training classification teachers.
Experimental setup. To train and validate our classification teachers, we use the MS COCO2017 [1]} detection dataset and crop all the objects according to their ground-truth bounding boxes. The resulting classification dataset consists of 849,902 objects for training and 36,334 objects for validation. We then train the teacher models in an image-classification manner, using the same data augmentation strategy and loss function as the student detector. Specifically, Faster RCNNs and RetinaNets share the same data augmentation methods, denoted as “general”, but use the categorical cross-entropy loss (CEL) and focal loss (FL) for their classification heads, respectively; SSDs have their own data augmentation strategy and use the categorical cross-entropy loss (CEL).
In our experiments, we take ResNet50 as the teacher model. In Section REF , we conduct an ablation study with different teachers. Furthermore, we investigate the influence of different input sizes to our classification teachers because the objects in object detection have different resolutions than they typically have in image classification. Therefore, we train the classification teacher with input sizes in \([56 \times 56, 112 \times 112, 224 \times 224]\) . Because Faster RCNNs and RetinaNets share the same data augmentation, we train a teacher for both frameworks using the two losses jointly. All the teacher models are trained using ImageNet-pretrained weights for 90 epochs with an initial learning rate of \(0.0001\) , divided by 10 at epoch 50.
<TABLE>
Training Setting for Compact Students
Let us now specify the details for the training settings of the compact student models used in REF , as mentioned in the main paper. All experiments in this work are performed on Tesla V100 GPUs.
SSD300 and SSD512. For data augmentation, we first apply photometric distortion transformations on the input image, then scale up the image by a factor chosen randomly between 1\(\times \)  and 4\(\times \)  by filling the blanks with the mean values of the dataset. We then sample a patch from the image so that the minimum IoU with the objects is in \([0.1, 0.3, 0.5, 0.7, 0.9]\) , with the precise value chosen randomly. Afterwards, the sampled patch is resized to \(300 \times 300\)  or \(512 \times 512\) , normalized by subtracting the mean values of the dataset, and horizontally flipped with a probability of 0.5. We use SGD with an initial learning rate of 0.002 to train the SSDs for 24 epochs, where the dataset is repeated 5 times. The batch size is 64, and the learning rate decays by a factor of 0.1 at the 16th and 22nd epoch.
Faster RCNN with lightweight backbones. For data augmentation, the input image is first resized so that either the maximum of the longer side is 1333 pixels, or the maximum of the shorter side is 800 pixels. Then, the image is horizontally flipped with a probability of 0.5. Afterwards, it is normalized by subtracting the mean values and dividing by the standard deviation of the dataset. The Faster RCNN-MobileNetV2 is trained by SGD for 12 epochs with a batch size of 16, and an initial learning rate set to 0.02 and divided by 10 at the 8th and 11th epoch. Faster RCNN-QR50 is trained with a larger batch size of 32 and a larger initial learning rate of 0.04. Note that, in practice, increasing the batch size and the learning rate enables us to shorten the training time while keeping the same performance as with the default 1\(\times \)  training setting in MMDetetion.

Analysis of Detection Errors
As mentioned in Section REF  of the main paper, we provide the 6 types of detection errors discussed by [2]}, namely, classification (cls) error, localization (loc) error, both cls and loc error, duplicate detection error, background error, missed ground-truth error (missedGTerror).
In essence, as shown by Figure REF , localization error increases significantly as the foreground IoU increases, while all other errors decrease. The classification-related errors typically drop by using our classification distillation strategy. See, for example, the classification error for IoUs smaller than 0.65, and the error of both cls and loc for all IoUs. By contrast, our localization distillation decreases the localization-related errors, including localization error and duplicate detection errors. Specifically, with localization distillation, the localization error drops by more than 2 mAP points for IoUs larger than 0.7, albeit with a marginal increase in missedGTerror and background error. Overall, while there is a tradeoff between our classification and localization distillation strategies, they play complementary roles in improving the performance of the student detector.
<FIGURE>