While early knowledge distillation techniques [1]}, [2]}, [3]} focused on the task of image classification, several attempts have nonetheless been made for object detection. To this end, existing techniques [4]}, [5]}, [6]} typically leverage the fact that object detection frameworks consist of three main stages depicted by Figure REF (a): A backbone to extract features; a neck to fuse the extracted features; and heads to predict classes and bounding boxes. Knowledge distillation is then achieved using
a teacher with the same architecture as the student but a deeper and wider backbone, such as a Faster RCNN [7]} with ResNet152 [8]} teacher for a Faster RCNN with ResNet50 student, thus facilitating knowledge transfer at all three stages of the frameworks. To the best of our knowledge, [9]} constitutes the only exception to this strategy, demonstrating distillation across different detection frameworks, such as from a RetinaNet [10]} teacher to a RepPoints [11]} student. This method, however, requires the teacher and the student to rely on a similar detection strategy, i.e., both must be either one-stage detectors or two-stage ones,
and, more importantly, still follows a detector-to-detector approach to distillation. In other words, the study of knowledge distillation remains limited to transfer across architectures tackling the same task.
