Distillation Decision Tree
Black-box machine learning models are criticized as lacking interpretability,
although they tend to have good prediction accuracy. Knowledge Distillation
(KD) is an emerging tool to interpret the black-box model by distilling its
knowledge into a transparent model. With well-known advantages in
interpretation, decision tree is a competitive candidate of the transparent
model. However, theoretical or empirical understanding for the decision tree
generated from KD process is limited. In this paper, we name this kind of
decision tree the distillation decision tree (DDT) and lay the theoretical
foundations for tree structure stability which determines the validity of DDT's
interpretation. We prove that the structure of DDT can achieve stable
(convergence) under some mild assumptions. Meanwhile, we develop algorithms for
stabilizing the induction of DDT, propose parallel strategies for improving
algorithm's computational efficiency, and introduce a marginal principal
component analysis method for overcoming the curse of dimensionality in
sampling. Simulated and real data studies justify our theoretical results,
validate the efficacy of algorithms, and demonstrate that DDT can strike a good
balance between model's prediction accuracy and interpretability.