Our experiments indicate that the best distillation algorithm varies among datasets/tasks (see in particular the top-5 configurations listed in Table REF  in Appendix). This inspires us to train a prediction model that recommends a good KD pipeline given a dataset. To represent the distillation performance across datasets which are evaluated on different metrics, we define distillation ratio as the fraction of the teacher's performance achieved by the student and use it as a general score for distillation performance. Then the prediction model can be trained to predict the distillation ratio based on features of the dataset/task as well as the features of each candidate distillation pipeline. Here we train our AutoDistiller performance prediction model via AutoGluon-Tabular, a simple AutoML tool for supervised learning [1]}. To the best of our knowledge, our proposed method is the first attempt towards automated KD in NLP.
