Innovations in KD for NLP generally involve improvements in one of the following aspects: 1) the loss function for gauging the discrepancy between student and teacher predictions [1]}, 2) the method for transferring intermediate network representations between teacher and student [2]}, [3]}, [4]}, 3) the use of data augmentation during student training [5]}, and 4) multiple stages of distillation [6]}, [7]}.
Many research proposals have simultaneously introduced new variations of more than one of these components, which confounds the impact of each component on the final performance of the distillation algorithm. In addition, it is often unclear whether a proposed distillation pipeline will generalize to a new dataset or task, making automated KD challenging.
For example, MixKD [8]} has only been evaluated on classification problems and it is unclear if the method will be effective for question answering.
