The general KD framework was popularized by [1]}, [2]}, aiming to transfer knowledge from an accurate but cumbersome teacher model to a compact student model by matching the class probabilities produced by the teacher and the student.
Focusing on AutoML settings with tabular data, [3]} proposed a general KD algorithm for different classical ML models and ensembles thereof.
Also hoping to identify good choices in the KD pipeline like our work, [4]} compared Kullback-Leibler divergence and mean squared error objectives in KD for image classification models, finding that mean squared error performs better.
